How to use Redis for caching in a web application

In today’s digital age, providing a fast and responsive user experience is key to the success of any web application. One way to achieve this is by implementing caching. Caching is a technique that involves temporarily storing frequently-requested data so that it can be quickly retrieved and served to users without the need to perform the underlying operation again.

In this article, we will dive into how to use Redis for caching in a web application, specifically in the context of caching backend web requests. By caching these requests, you can significantly improve the responsiveness of your application and serve more users at once. Caching can also be a cost-effective method to scale your platform, especially if you have data-intensive queries that many users will want to access. Finally, using Redis as a caching system in a web application can also help improve SEO by reducing the load time and increasing the speed of the website, which are important ranking factors for search engines.

Where a cache can live

When it comes to caching in a web application, there are several different places that a cache can be implemented.

One common place to implement a cache is between the user and the frontend resources. This is often achieved with a CDN (Content Delivery Network), and rarely requires an in-memory database such as Redis, as these caches persist for hours or days, and can be accessed directly from a simple file storage. In a nutshell, CDNs are a service that allows you to store duplicates of your static content in many distributed servers around the world. When a user requests this static data, the server that is closest to them will respond with the data. We won’t be covering CDNs in this article, as it is a standard practice and very easy to find great tutorials on.

Another place you can cache data is within the frontend, using a state management system such as Redux, React Query, or manually using the local storage in the user’s browser. We’ll focus on the backend in this article, as frontend caching is a whole topic of its own.

Caching between the frontend and the backend

Moving on to the first type of cache we will cover in depth, one of the simplest places to implement a cache is between the frontend and the backend endpoints. This type of caching can be used to improve the performance of the endpoints by temporarily storing the responses from the server. This way, when a user requests the same endpoint multiple times, the cache can quickly return the stored response instead of having to generate a new one. This can greatly reduce the load on the server and improves the responsiveness of the application. Redis is an ideal candidate for this kind of cache, as we need to quickly access stored data in milliseconds, and expire old data on the scale of seconds rather than minutes or hours.

The frontend will still communicate with the webserver directly. However, if the result is in the cache, the webserver does not have to do any processing beyond grabbing the data from the cache and returning it to the frontend.

Why Redis

To begin with, let’s discuss the primary technology we will use for our cache layer, Redis. Redis is an in-memory data structure store that can be used as a database, cache, and message broker. It was designed to be lightweight, fast, and efficient, making it an ideal choice for a wide range of use cases. Redis stores all data in-memory, which means that it can perform operations much faster than traditional disk-based databases. Additionally, it supports a wide range of data structures, allowing for complex data manipulation and querying. Redis also supports a publish-subscribe messaging pattern, allowing for real-time communication between different parts of an application. All of these features make Redis a powerful and versatile tool that can be used for a variety of use cases, including caching, session management, real-time analytics, and more.

Caching GET endpoints

When using Redis for caching, the process of setting up a cache for GET endpoints is relatively straightforward. The first step is to set up a Redis server and make sure it is running. Next, you will need to install the Redis client library for your programming language of choice. Once that is done, you can start using the library to interact with the Redis server and set up your cache. In order to cache the responses from a GET endpoint, the ideal solution is to use middleware or a decorator which creates a unique key for each endpoint and parameters on those endpoints.

The next step is to check the cache before processing the request. When a user requests a GET endpoint, you can check the cache for the key that corresponds to the requested endpoint and query parameters. If the key exists in the cache, you can simply return the stored response. If the key does not exist, you would process the request, store the response in the cache using the key, and then return the response to the user. The overhead of this is relatively tiny, as Redis operations are incredibly fast.

A caching decorator in Python

An easy way to implement caching for GET endpoints in a Python web application is by using a decorator. A decorator is a special type of function that can be used to modify the behavior of other functions. In this case, the decorator can be used to cache the responses from GET endpoints, so they can be quickly returned to the user without the need to generate a new response.

Here’s an example of how you could implement a caching decorator in Python using Redis:

import redis

# Create a connection to the Redis server
r = redis.Redis(host='localhost', port=6379)

def cache(func):
    def wrapper(*args, **kwargs):
        # Create a unique key for the function call
        key = f"{func.__name__}:{args}:{kwargs}"

        # Check if the key exists in the cache
        if r.exists(key):
            # If the key exists, return the cached data
            return r.get(key)

        # If the key does not exist, call the function and store the result in the cache
        result = func(*args, **kwargs)
        r.set(key, result)
        return result
    return wrapper

# Use the decorator to cache the function
@cache
def get_data():
    # This is the function that we want to cache
    return expensive_data_operation()

With this example, the decorator cache will be used to cache the response of the function get_data(). The decorator creates a unique key for the function call based on the function name, arguments and keyword arguments and it checks if this key exists in the cache. If the key exists, the decorator returns the cached data, otherwise it calls the function, stores the result in the cache and returns the result.

It’s important to note that the decorator does not set any time-to-live for the cache, this means that the data will be stored in the cache indefinitely. If the data is expected to change often, it would be good to consider setting a short time-to-live for the cache. By using this decorator, you can easily cache GET endpoints in your web application, improving the performance and reducing the load on the server. The decorator is a simple and reusable solution that can be easily applied to multiple endpoints with minimal code changes.

Caching between the backend and the database

A cache can also be implemented between the endpoints and the database. This type of caching can be used to improve the performance of the database by temporarily storing the results of frequently-requested queries. This way, when a user requests the same data multiple times, the cache can quickly return the stored results instead of having to query the database. This can greatly reduce the load on the database and improve the responsiveness of the application.

The downside of this approach is that you will need to invest more effort into the implementation of this cache. It is less simple to create a simple solution that caches all database queries, as some applications may almost always have unique queries, or data that changes so frequently that the cache is rarely used. Since adding in this caching layer does introduce some overhead, you should estimate how likely it is that the cache will be used to ensure that it is worth the development effort and maintenance to use a cache in between the webserver and database. As the communication between the webserver and database is usually higher volume than between the frontend and the webserver, the overhead here matters a lot more, but there are huge potential benefits to your application speed.

The major bottleneck of most web applications is the database, as relational databases are very difficult to scale horizontally, with unique challenges. This is one of the reasons why there is a lot of potential performance to be gained by caching in the database layer.

Read replicas

When implementing a caching layer between the database and the queries, it’s also possible to empower this caching further by using read replicas. A read replica is a separate copy of a database that is kept in sync with the master database. This allows read queries to be executed on a read replica database, rather than on the master database. This can greatly reduce the load on the master database and improve the performance of the application.

Using read replicas in conjunction with a caching layer can provide several benefits. For example, read queries can be executed on the read replica, and the results can be stored in the cache. This way, subsequent read queries can be served from the cache, rather than from the read replica, further reducing the load on the read replica and improving the performance of the application.

It’s important to note that read replicas are read-only databases and cannot be written to directly, so any updates or inserts must be executed on the master database. This is where the master-slave architecture comes into play, any write operation must be executed on the master database, and the read replica will be updated as soon as possible, in this way we can scale the relational database and achieve a certain degree of horizontal scaling.

When you decide to introduce a read replica system, you may want to consider creating a specific caching service for accessing the database. The benefit of this is separating the concerns of the webserver from the database access, which would make it easier for this to be a drop-in replacement for existing applications. Furthermore, this architecture supports multiple different backends being able to access your database through one service, reducing code duplication. When you start needing to decide per-query which database will be connected to, you should certainly consider creating this separate service for handling your database queries.

While both options have their advantages, caching between the user and the endpoints is generally easier to implement and is a good starting point for improving the performance of an application. However, if an application has significant performance requirements, caching between the backend and the database should be considered as well.

Conclusion

In this article, we have discussed the importance of caching in web applications and the various ways in which it can be implemented. We have focused on the use of Redis as an in-memory data structure store for caching, and how it can be used to improve the performance and scalability of web applications. We have discussed different places where caching can be implemented, such as between the user and the web page, between the user and the endpoints, and between the endpoints and the database. We have also highlighted the benefits of using read replicas in conjunction with a caching layer, as well as the trade-offs to consider when choosing a caching strategy.

Most applications are using some form of frontend caching, and we would recommend considering caching between the frontend and the endpoints as a straightforward and effective solution to improve performance for frequently requested endpoints.