Hey there, in this article we would be discussing how you can queue tasks and run them in the background in node.js and express server.

Queuing jobs is a really easy task, we would be installing a couple of dependencies and then write some code. I would highly recommend you to code along for better understanding.

with that being said, let's get started.

Step 0: Setup

In this project, we would be using a package known as "bull" which is great for queuing jobs in the background. But in order to run the bull package, we must have Redis installed on our machine. So, head over to redis.io/download and download the stable version of Redis. After successful installation, we are good to go.

Step 1: Project Setup and installing dependencies

Now, go ahead and initialize an empty nodejs project with npm init. I would assume that you are already familiar with node.js and express and won't be going deep into the basics. After setting up the project install express and bull library by running npm install express bull.

Step 2: Basic express server

In this step, we would be creating a simple express server up and running.

// index.js
const express = require('express');

const app = express();

const PORT = process.env.PORT || 8000;

app.get('/', (req, res) => {
    res.end('Hello from the server')
})

app.listen(PORT, () => console.log(`Server started at PORT:${PORT}`))

let's start the server node index.js.

Step 3: Implementing Queue

Now, in this step, we would be implementing the queue feature. To understand the implementation queue, let's simulate a fake situation. Please read and understand the following situation carefully.

A user visits your website and sees a form. He fills up the form and uploads some high-quality photos and clicks on submit. The request is then sent to the backend along with the data and photos he uploaded. Now, on the server, you want to first compress the images and upload the image to let's say amazon s3 bucket and then update the URL of images in the database.

As you can already guess, the above scenario is a time taking process. If you do all the above-mentioned tasks on the main thread it would highly affect the performance of your server and users have to wait really long. So, to avoid that let's code the above scenario with queues.

const express = require('express');
const Queue = require('bull'); // import the bull package

const app = express();

const PORT = process.env.PORT || 8000;

// create a middleware that creates queue and add to the request object
app.use((req, res,next) => {
    const imageProcessingQueue = new Queue('imageProcessingQueue'); // Create a queue object.
    req.queue = imageProcessingQueue; // add this queue to req object so its accessible in the request handler.

    next(); // important!
});

app.get('/', (req, res) => {
    res.end('Hello from the server')
});

// Submit route, where the data is actually submitted.
app.post('/submit', (req, res) => {
    const imageFiles = req.files; // Images uploaded by the user.
    const data = insertIntoDatabase(req.body); // Rest of the form data has been saved in database;

    // Define a queue.process method
    req.queue.process((job, done) => {
        // Write all the logic to compress and upload file here.
        // I would be using setTimeout to simulate this thing.

        setTimeout(() => {
            console.log('Compression and upload done');
            // update the links in database. job.data.id contains the id of the insertedData,
            done(); // call the done method when all things are done;
        }, 5000);
    });

    req.queue.add(data); // finally add the queue and pass the required info as parameter.

    return res.json({
        message: 'successfully queued',
    });

});

app.listen(PORT, () => console.log(`Server started at PORT:${PORT}`))

let's start the server node index.js

Now, I would fire up my postman and try to send POST requests at /submit route.

Great! I got a "successfully queued" response. Now let's check our server console.

As you can see, after 5 seconds, I got a log saying "Compression and upload done" as I set a time out of 5 seconds.

I hope that now you have understood how you can queue tasks in nodejs and express server.