23.2 23.1 22.3 22.2 22.1 21.11

Use Redpanda with NodeJS

If you haven’t already, install Node by following the appropriate steps for your OS.

Redpanda is Kafka API-compatible, which means that you can leverage the countless client libraries created for Kafka (if you find something that is not supported, reach out to the Redpanda team on our Slack). In this example you will use kafkajs.

# Create and enter the project folder
mkdir redpanda-node
cd redpanda-node
# Generate package.json (use the default values)
npm init
# Install required dependencies
npm i -D typescript
npm i -D @types/node
npm i kafkajs
npm i uuid
npm i -D @types/uuid
# Generate tsconfig.json
tsc --init

Set up a Redpanda environment

Follow the Redpanda Quickstart to spin up a development environment in Docker.

If you’re running Redpanda on your laptop, or in a shared development environment, then avoid Redpanda’s optimized production settings. Running sudo rpk redpanda tune all or manually configuring Redpanda for production might affect your experience with other applications running on your machine.

Create a topic

The easiest way to create topics is using rpk:

$ rpk topic create chat-room
TOPIC      STATUS
chat-room  OK

The preceding command created a topic named chat-room, with the default number of partitions and replicas. You can list all created topics with:

$ rpk topic list
NAME       PARTITIONS  REPLICAS
chat-room  1           1

You can create topics programmatically too:

src/admin.ts
import \{Kafka} from "kafkajs"

const redpanda = new Kafka({
  brokers: ["localhost:9092"]
})
const admin = redpanda.admin()

export function createTopic(topic: string, partitions?: number, replicas?: number) {
  return admin.connect().then(() \=> {
    admin.createTopics({
      topics: [{
        topic: topic,
        numPartitions: partitions ? partitions : 1,
        replicationFactor: replicas ? replicas : 1,
      }]
    }).then(() \=> admin.disconnect())
  })
}

After you have a cluster up and running, and a topic to store your data, you can build a producer and consumer:

Producer code

src/producer.ts
import {Kafka} from "kafkajs"

const redpanda = new Kafka({
  brokers: ["localhost:9092"]
})
const producer = redpanda.producer()

export function getConnection(user: string) {
  return producer.connect().then(() => {
    return (message: string) => {
      return producer.send({
        topic: "chat-room",
        messages: [{value: JSON.stringify({message, user})},],
      })
    }
  })
}

export function disconnect() {
  return producer.disconnect()
}

You now have a working producer that sends strings entered by the user to the chat-room topic. Messages are sent as JSON encoded strings here, but keep in mind that the producer only sends buffers, so you can encode the messages however you like.

Consumer code

src/consumer.ts
import \{Kafka} from "kafkajs"
import {v4 as uuidv4} from "uuid"

const redpanda = new Kafka({
  brokers: ["localhost:9092"]
})
const consumer = redpanda.consumer({groupId: uuidv4()})

export function connect() {
  return consumer.connect().then(() \=>
    consumer.subscribe({topic: "chat-room"}).then(() \=>
      consumer.run({
        eachMessage: async ({topic, partition, message}) \=> {
          const formattedValue = JSON.parse((message.value as Buffer).toString())
          console.log(`${formattedValue.user}: ${formattedValue.message}`)
        },
      })
    )
  )
}

export function disconnect() {
  consumer.disconnect()
}

You now have a consumer that will read all messages from the chat-room topic and print them to the console. You can start as many consumer groups as you like, but bear in mind that each group will read a message only once, which is why the example is using a generated UUID for the group ID.

The following example brings all the preceding examples together:

src/index.ts
import * as readline from "node:readline"
import * as Admin from "./admin"
import * as Producer from "./producer"
import * as Consumer from "./consumer"

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout
})

function start() {
  const topic = "chat-room"
  console.log(`Creating topic: ${topic}`)
  Admin.createTopic(topic).then(() => {
    console.log("Connecting...")
    Consumer.connect().then(() => {
      rl.question("Enter user name: \n", function (username) {
        Producer.getConnection(username).then((sendMessage) => {
          console.log("Connected, press Ctrl+C to exit")
          rl.on("line", (input) => {
            readline.moveCursor(process.stdout, 0, -1)
            sendMessage(input);
          })
        })
      })
    })
  })
}
start()

process.on("SIGINT", process.exit)
process.on("exit", () => {
  Producer.disconnect();
  Consumer.disconnect();
  rl.close();
})

Running

tsc src/index.ts && node src/index.js

Run this at least twice so that you can chat between two terminals, but you can run as many as you like.

Wrapping up

Now you have the basic building blocks for working with Redpanda using Node.js.