Use Redpanda with NodeJS
If you haven’t already, install Node by following the appropriate steps for your OS.
Redpanda is Kafka API-compatible, which means that you can leverage the countless client libraries created for Kafka (if you find something that is not supported, reach out to the Redpanda team on our Slack). In this example you will use kafkajs.
# Create and enter the project folder
mkdir redpanda-node
cd redpanda-node
# Generate package.json (use the default values)
npm init
# Install required dependencies
npm i -D typescript
npm i -D @types/node
npm i kafkajs
npm i uuid
npm i -D @types/uuid
# Generate tsconfig.json
tsc --init
Set up a Redpanda environment
Follow the Redpanda Quickstart to spin up a development environment in Docker.
If you’re running Redpanda on your laptop, or in a shared development environment, then avoid Redpanda’s optimized production settings. Running sudo rpk redpanda tune all or manually configuring Redpanda for production might affect your experience with other applications running on your machine.
|
Create a topic
The easiest way to create topics is using rpk:
$ rpk topic create chat-room
TOPIC STATUS
chat-room OK
The preceding command created a topic named chat-room
, with the default number of
partitions and replicas. You can list all created topics with:
$ rpk topic list
NAME PARTITIONS REPLICAS
chat-room 1 1
You can create topics programmatically too:
src/admin.ts
import \{Kafka} from "kafkajs"
const redpanda = new Kafka({
brokers: ["localhost:9092"]
})
const admin = redpanda.admin()
export function createTopic(topic: string, partitions?: number, replicas?: number) {
return admin.connect().then(() \=> {
admin.createTopics({
topics: [{
topic: topic,
numPartitions: partitions ? partitions : 1,
replicationFactor: replicas ? replicas : 1,
}]
}).then(() \=> admin.disconnect())
})
}
After you have a cluster up and running, and a topic to store your data, you can build a producer and consumer:
Producer code
src/producer.ts
import {Kafka} from "kafkajs"
const redpanda = new Kafka({
brokers: ["localhost:9092"]
})
const producer = redpanda.producer()
export function getConnection(user: string) {
return producer.connect().then(() => {
return (message: string) => {
return producer.send({
topic: "chat-room",
messages: [{value: JSON.stringify({message, user})},],
})
}
})
}
export function disconnect() {
return producer.disconnect()
}
You now have a working producer that sends strings entered by the user to the
chat-room
topic. Messages are sent as JSON encoded strings here,
but keep in mind that the producer only sends buffers, so you can encode the
messages however you like.
Consumer code
src/consumer.ts
import \{Kafka} from "kafkajs"
import {v4 as uuidv4} from "uuid"
const redpanda = new Kafka({
brokers: ["localhost:9092"]
})
const consumer = redpanda.consumer({groupId: uuidv4()})
export function connect() {
return consumer.connect().then(() \=>
consumer.subscribe({topic: "chat-room"}).then(() \=>
consumer.run({
eachMessage: async ({topic, partition, message}) \=> {
const formattedValue = JSON.parse((message.value as Buffer).toString())
console.log(`${formattedValue.user}: ${formattedValue.message}`)
},
})
)
)
}
export function disconnect() {
consumer.disconnect()
}
You now have a consumer that will read all messages from the chat-room
topic and print them to the console. You can start as many consumer groups as
you like, but bear in mind that each group will read a message only once, which is
why the example is using a generated UUID for the group ID.
The following example brings all the preceding examples together:
src/index.ts
import * as readline from "node:readline"
import * as Admin from "./admin"
import * as Producer from "./producer"
import * as Consumer from "./consumer"
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
})
function start() {
const topic = "chat-room"
console.log(`Creating topic: ${topic}`)
Admin.createTopic(topic).then(() => {
console.log("Connecting...")
Consumer.connect().then(() => {
rl.question("Enter user name: \n", function (username) {
Producer.getConnection(username).then((sendMessage) => {
console.log("Connected, press Ctrl+C to exit")
rl.on("line", (input) => {
readline.moveCursor(process.stdout, 0, -1)
sendMessage(input);
})
})
})
})
})
}
start()
process.on("SIGINT", process.exit)
process.on("exit", () => {
Producer.disconnect();
Consumer.disconnect();
rl.close();
})
Running
tsc src/index.ts && node src/index.js
Run this at least twice so that you can chat between two terminals, but you can run as many as you like.