About Kafka

Kafka

The Mail System Analogy What is Kafka? Kafka is like a postal service for data. It helps different parts of a business communicate with each other by sending and receiving messages, much like how people send and receive letters. Producers (Senders) Imagine you have many people (producers) who want to send letters (messages).…

Kafka: The Mail System Analogy

What is Kafka(Non Technical Terms)?

Kafka is like a postal service for data. It helps different parts of a business communicate with each other by sending and receiving messages, much like how people send and receive letters.

Producers (Senders)

Imagine you have many people (producers) who want to send letters (messages). These people write their letters and drop them into a mailbox.

Kafka (Post Office)

Kafka acts like the central post office. It collects all the letters from various mailboxes and sorts them.

Topics (Mailing Addresses)

Each letter has an address (topic) indicating where it should go. For instance, one topic could be “orders,” another could be “customer feedback,” and another could be “inventory updates.”

Consumers (Receivers)

There are other people (consumers) who are waiting to receive these letters. They go to the post office and pick up all the letters addressed to them.

Durability and Reliability

Kafka ensures that letters (messages) are stored safely until they are picked up, so no messages are lost even if the recipient is not ready to pick them up immediately.

Scalability

Just like a post office can handle more letters by hiring more workers and adding more mailboxes, Kafka can handle more data by adding more servers and partitions, ensuring it can scale with the growing needs.

Summary

Kafka is a robust, scalable system for handling real-time data streams, acting as a middleman that ensures messages are delivered reliably from senders to receivers.

Now, Let us understand the terms Technically

What is Kafka(Technical Terms?

Kafka is like the Pega Integration layer that facilitates communication between different applications. It allows various Pega systems or other applications to exchange information efficiently.

Producers (Senders)

Producers are like the Connectors or Data Pages that send data to Kafka. These are the sources generating the data, such as a Pega case generating an event or updating information.

Kafka (Central Data Hub)

Kafka acts as a centralized data hub, similar to the Pega Data Flows or Event Services. It collects, processes, and routes data between different parts of an enterprise system.

Topics (Data Channels)

Topics in Kafka are like data channels or queues in Pega. Each topic corresponds to a specific type of data stream, such as “Order Events” or “Customer Updates,” much like how a Pega queue or a data page handles specific types of information.

Consumers (Receivers)

– Consumers in Kafka are akin to Pega Listeners or Service Rules. These are the components that read and process the data from Kafka topics, similar to how a Pega listener processes messages from a queue.

Durability and Reliability

– Kafka ensures that messages are stored reliably until processed, similar to how Pega ensures data integrity and trans actionality in data flows and case processing. Kafka’s persistence mechanisms are akin to Pega’s use of database tables for storing work objects until they are needed.

Scalability

Kafka’s ability to handle large volumes of data by adding more servers and partitions is similar to how Pega scales its infrastructure using nodes and clustering to handle increased load and ensure high availability.

Summary

In Pega terms, Kafka serves as a highly efficient, scalable, and reliable integration layer that ensures real-time data flow between various systems and components. It acts much like Pega’s connectors, listeners, and data flows, facilitating seamless communication and data processing within a Pega application ecosystem.

How its works

Let us check by installing the Kafka apache server locally and test it out.

Step 1: Download Kafka from the official website- https://kafka.apache.org/downloads

Step 2: Update the log directory and data directory and replace the temp files.

server-log.dirs=C:/kafka/kafka-logs zookeeper – dataDir=C:/kafka/zookeeper-data Step 3: Start zookeeper and Kafka Server

zookeeper – .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties kafka: .\bin\windows\kafka-server-start.bat .\config\server.properties The following steps to test how it work in our system locally.

Create a topics by executing the following commands on kafka folder. – kafka-topics.bat –create –bootstrap-server localhost:9092 –topic test.

1. Once the topics is created. Initiate the message on the Topic by executing the following commands

kafka-console-producer.bat –broker-list localhost:9092 –topic test.

At the same time. on the consumer end. Open the command prompt again on the kafka folder and run the following command to listen to it.

Once its done. when we post any message on the producer side. in the real time- that will get posted to consumer end.

Here is the snap shot:

Since we checked how it works on the local system, let us quickly check how this can be done in Pega.

In Pega- Under Sys Admin Category- Check the Kafka rule- and create the Kafka rule and set the server information like Host and Post. Test the connectivity- Reference link : Creating-kafka-configuration-instance

1. Create a Dataset rule which is available under Data model category- configure the dataset by following the Pega article: Kafka-data-set.html