quinta-feira, 26 de maio de 2016

Integrating node.js and Apache Kafka

In this post I will demonstrate how we can integrate node.js and Apache Kafka, producing and consuming messages in a very simple example.

First of all, let us get Apache Kafka up and running, you can see how to do on the official kafka's site tutorial.

Once it's up and running, we can set up the project and start playing with the lib no-kafka

  • npm init
  • npm install no-kafka --save

I have used the version 2.4.2 of no-kafka. So, if you want to inform the version when installing, just run it as "npm install no-kafka@2.4.2 --save".

Here is a producer example, which will connect to kafka and produce messages in a topic.

var Kafka = require('no-kafka');
var producer = new Kafka.Producer();
return producer.init()
  return producer.send({
      topic: 'kafka-test-topic',
      partition: 0,
      message: {
          value: 'Hello!'
.then(function (result) {
  console.log('topic sent');
If you are running a local instance, it connects automatically to the local host. To connect to a external instance, you can replace "localhost" by the external host following the example below:

var Kafka = require('no-kafka');
var connString = ' kafka://localhost:9092, localhost:9092 '
var producer = new Kafka.Producer({ connectionString: connString });

Here is a consumer example, which will connect to kafka and subscribe to a topic, receiving messages and printing them to the console.

var Kafka = require('no-kafka');
var consumer = new Kafka.SimpleConsumer();
// data handler function can return a Promise 
var dataHandler = function (messageSet, topic, partition) {
    messageSet.forEach(function (m) {
        console.log('topic received: ');
            'partition': partition,
            'offset': m.offset,
            'message': m.message.value.toString('utf8')
return consumer.init()
.then(function () {
    return consumer.subscribe('kafka-test-topic', 0, dataHandler);

As you can see in the pieces of code above, all requests return promises. This is an example of the very basic features of the lib interacting with Kafka.
I put this project on my github, so then you can play with the code and evolve as needed.

Nenhum comentário:

Postar um comentário