Build Job Queue using Redis Pub/Sub

Welcome to our comprehensive guide on Redis, where we will explore the world of high-performance data storage and real-time communication! In this blog, we will dive deep into Redis, an open-source, in-memory data structure store that has become an essential tool for caching, session management, real-time analytics, and more in modern web development. Refer to my previous article to know basic Redis commands

Redis Pub/Sub (Publish/Subscribe) is a powerful messaging pattern that enables real-time communication and event-driven architectures in modern applications. In this blog post, we will delve deeper into the Redis Pub/Sub system, explore its advanced concepts, and demonstrate how it can be used in real-world scenarios. We will provide code snippets in Node.js to illustrate the concepts, allowing you to grasp the power of Redis Pub/Sub for building scalable, event-driven applications.

Prerequisites

Before proceeding with this tutorial, make sure you have Redis installed and running on your local machine or server. Additionally, ensure you have Node.js and npm (Node Package Manager) installed.

Understanding Redis Pub/Sub

Redis Pub/Sub follows a straightforward model where clients can subscribe to channels and receive messages published to those channels. It offers a decoupled way of communication, where publishers and subscribers don't need to know about each other. When a message is published to a channel, all active subscribers to that channel receive the message simultaneously.

In Redis, the Pub/Sub system is entirely separate from the key-value storage system, allowing you to use it in conjunction with other Redis features.

Setting Up the Redis Pub/Sub System in Node.js

To use Redis Pub/Sub in Node.js, we will be using the popular ioredis library, which provides a comprehensive and efficient Redis client for Node.js applications.

First, install the ioredis package using npm:

npm install ioredis

Now, let's create a simple publisher and subscriber in Node.js:

// Publisher (publisher.js)

const Redis = require('ioredis');
const publisher = new Redis();

async function publishMessage(channel, message) {
  await publisher.publish(channel, message);
  console.log(`Published message to channel ${channel}: ${message}`);
}

publishMessage('news', 'Breaking news: Redis is amazing!');

// Subscriber (subscriber.js)

const Redis = require('ioredis');
const subscriber = new Redis();

async function subscribeToChannel(channel) {
  await subscriber.subscribe(channel);
  console.log(`Subscribed to channel: ${channel}`);

  subscriber.on('message', (ch, message) => {
    console.log(`Received message from channel ${ch}: ${message}`);
  });
}

subscribeToChannel('news');

Remember, run the publisher & subscriber file in different terminal windows. You will see that the subscriber receives the message published by the publisher.

Multiple Subscribers and Unsubscribing

Redis Pub/Sub allows multiple subscribers to receive messages published to a channel. Let's modify our subscriber code to handle multiple channels:

const Redis = require('ioredis');
const subscriber = new Redis();

async function subscribeToChannels(channels) {
  for (const channel of channels) {
    await subscriber.subscribe(channel);
    console.log(`Subscribed to channel: ${channel}`);
  }

  subscriber.on('message', (ch, message) => {
    console.log(`Received message from channel ${ch}: ${message}`);
  });
}

subscribeToChannels(['news', 'updates', 'events']);

With this modification, the subscriber will receive messages from all three channels: 'news', 'updates', and 'events'.

To unsubscribe from a channel, use the unsubscribe method:

async function unsubscribeFromChannel(channel) {
  await subscriber.unsubscribe(channel);
  console.log(`Unsubscribed from channel: ${channel}`);
}

unsubscribeFromChannel('news');

Pattern Subscriptions

Redis Pub/Sub supports pattern subscriptions, allowing subscribers to receive messages from channels that match a specific pattern. The pattern can include wildcards ('*') and ('?').

Let's update our subscriber code to demonstrate pattern subscriptions:

async function subscribeToPattern(pattern) {
  await subscriber.psubscribe(pattern);
  console.log(`Subscribed to pattern: ${pattern}`);

  subscriber.on('pmessage', (pattern, channel, message) => {
    console.log(`Received message from channel ${channel} matching pattern ${pattern}: ${message}`);
  });
}

subscribeToPattern('sport*');

In this example, the subscriber will receive messages from all channels that match the pattern 'sport*'.

Building a Simple Job Queue for Sending Emails with Redis

In this section, we will explore how to build a simple job queue using Redis Pub/Sub for sending out emails. 

Sending emails is a common requirement in web applications, and processing them asynchronously can significantly improve the responsiveness and performance of your application. With Redis Pub/Sub, we can efficiently manage the distribution of email jobs among multiple subscribers, ensuring that emails are sent out without delaying other critical operations.

By the end of this section, you will have a clear understanding of how to set up a job queue using Redis Pub/Sub, with one publisher publishing email jobs, and multiple subscribers processing and sending out those emails.

Installing Packages

Let us install the required package for our node js application

npm install ioredis nodemailer

// Publisher (publisher.js)

const Redis = require('ioredis');
const publisher = new Redis();
const emailQueue = 'email_queue';

async function publishEmailJob(email) {
  try {
    await publisher.publish(emailQueue, email);
    console.log('Email job published successfully!');
  } catch (err) {
    console.error('Failed to publish email job:', err);
  }
}

publishEmailJob('user@example.com');

// Subscriber.js:

const Redis = require('ioredis');
const subscriber = new Redis();
const emailQueue = 'email_queue';
const nodemailer = require('nodemailer');

async function processEmailJob(email) {
  try {
    // Create a transporter for sending emails (update with your SMTP credentials)
    const transporter = nodemailer.createTransport({
      service: 'gmail',
      auth: {
        user: 'your_email@example.com',
        pass: 'your_email_password',
      },
    });

    // Define email options (update with your email content and subject)
    const mailOptions = {
      from: 'your_email@example.com',
      to: email,
      subject: 'Test Email from Redis Job Queue',
      text: 'This is a test email sent from the Redis Job Queue.',
    };

    // Send the email
    await transporter.sendMail(mailOptions);
    console.log(`Email sent to ${email}`);
  } catch (err) {
    console.error('Failed to send email:', err);
  }
}

// Subscribe to the email queue
subscriber.subscribe(emailQueue);

// Process email jobs
subscriber.on('message', (channel, email) => {
  processEmailJob(email);
});

Explanation

The publisher publishes email jobs to the 'email_queue' channel using Redis Pub/Sub. In this example, we're publishing the email 'user@example.com'.

The subscriber subscribes to the 'email_queue' channel and listens for incoming email jobs. When a new email job arrives, the subscriber processes the job and sends out the email using Nodemailer.

Nodemailer is a popular Node.js library for sending emails. Make sure to replace the transporter and mailOptions with your own email credentials and content.

Running the Publisher and Subscriber:

In separate terminal windows, run the publisher.js and subscriber.js files:

node publisher.js

node subscriber.js

You will see that the publisher successfully publishes the email job, and the subscriber processes the job, sending out the email.

Scaling Redis Pub/Sub for High Throughput:

Redis Pub/Sub can handle a substantial number of messages, but at some point, you may need to scale the system to handle extremely high throughput.

Horizontal Scaling

To scale Redis Pub/Sub horizontally, you can run multiple Redis instances as a cluster. Each instance will manage a subset of channels and subscribers, reducing the load on a single Redis server.

Vertical Scaling

Vertical scaling involves deploying Redis on more powerful hardware, like a machine with higher memory and CPU capacity. This can increase the system's capacity to handle more subscribers and messages.

Combination of Horizontal and Vertical Scaling

For extreme scenarios, you can combine horizontal and vertical scaling to create a highly scalable Redis Pub/Sub system.

Conclusion

Redis Pub/Sub is a versatile and efficient messaging pattern that empowers developers to build real-time, event-driven architectures in their applications. In this blog post, we explored the core concepts of Redis Pub/Sub, including setting up a basic publisher and subscriber in Node.js, handling multiple subscribers, using pattern subscriptions, and discussed strategies for scaling the system for high throughput scenarios. In this blog, we discussed publishing, and subscribing using Redis, we also came to know about multiple subscribers and pattern subscriptions. We have also covered how to build a simple job queue using Redis Pub/Sub for sending out emails. Also we covered scaling and different types of scaling.

Happy Coding!

Source code available in Github