A Pub/Sub Model Using Redis Stream

Redis is more than an in-memory caching database. Along with its various use cases, it offers the powerful Redis Stream feature, enabling it to function as a robust Pub/Sub model. In this article, we will explore how to leverage Redis Stream as the backbone of the Pub/Sub model. By publishing messages to a channel and subscribing to that channel, we can effectively utilize the capabilities of Redis Stream.

The Pub/Sub model we will implement can be used for bulk processing of various tasks. For example, it can be used for sending bulk SMS, email, heavy calculation, and overall does not hold the API request for a long time.

Application Overview

We will have a Redis server. This Redis server provides a publishing client and an API server will publish messages to a channel using this client. On the other hand, for each worker, there will be individual subscription clients. Workers will subscribe to the channels and any time a message is published, it will read the message.

So according to the diagram, the API server will receive a request. Then according to the request, it will go to the Redis server and publish a message. The workers are always in the standby status and workers will consume the message.

Code Architecture

We will make use of 3 different containers, api-container, worker-container, and redis-container. Both the api-container and worker-container are connected with the same Redis server. The api-container uses the publisher to publish the message and the worker-container subscribed to the messages.

Code Snippet

We can simply publish a message,

const data = {
  uuid: Math.random().toString(),
  title: 'Testing redis stream',
  description: 'An article on Redis Stream from Ixora Solution Ltd.',
  timestamp: new Date().toISOString()
};
await publishClient.publish(REDIS_STREAM_CHANNEL, JSON.stringify(data));Code language: JavaScript (javascript)

To subscribe to a message,

await subscribeClient.subscribe(REDIS_STREAM_CHANNEL, message => {
  console.log(message);
});Code language: JavaScript (javascript)

Now it is important, although we are using the same Redis database, we need a different Redis client, can not use the same client for both publish and subscribe messages.

For getting a Redis client,

import { createClient } from 'redis';

const publishClient = createClient({
  socket: {
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_DOCKER_PORT as unknown as number
  },
  password: process.env.REDIS_PASSWORD
}); Code language: JavaScript (javascript)

Workflow

First, clone the code and run the app docker-compose up

From the browser, when we go to http://localhost:3000/

A message will be streamed to the Redis and our worker will consume the message as well as will be printed it in the console.

Features

This is a basic implementation of Redis. However, Redis has many advanced features like regular queue service,

• Can scale the server vertically and horizontally
• Allow acknowledgement from the consumer as it is being processing or processed
• Allow setting a TTL, so if there is no acknowledgement from the consumer, the publisher make the message visible again to be proceed by different consumer

Conclusion

A complete code base is already in Github. For any query, please comment below or contact with iXora Solution teams.