Apache Kafka – Introduction

Hi All,

I am starting a series of tutorials/learning for Apache Kafka. I will try to cover all aspects of Apache Kafka. I will start with Basic concepts and move on to advance features. Be ready for lot of theory and try to grasp the concepts.

So What is Kafka?

Apache Kafka is a fast, scalable, fault-tolerant messaging system, which enables the communication between producers and consumers using messaging-based topics. One of the features of Kafka is it’s highly available and resilient to node failures and supports automatic recovery. In short, Kafka is the most sort after messaging system as compared to ActiveMQ, RabbitMQ.

Messaging System

The main task of a Messaging System is to transfer data from one application to another so that the application can focus on data and not worry about how to transfer/share data with another system.  There are two types of messaging patterns. available:

1. Point to Point Messaging
   • In this pattern, the message produced continues to remain in the queue. The consumer/s can consume the message from the queue, but only one consumer can consume a particular message.
2. Publish-Subscribe Messaging
   • In this pattern, the message produced continues to remain in a topic. The consumer/s can take more than one topic and consume every message in that topic. In Kafka, message producers are called Kafka producers ( publishers) and Message consumers are Kafka consumers (subscribers).

History ( Source: Apache Kafka – Wikipedia)

Kafka was originally developed by LinkedIn and was subsequently open-sourced in early 2011. Graduation from the Apache Incubator occurred on 23 October 2012. Jay Kreps chose to name the software after the author Franz Kafka because it is “a system optimized for writing”, and he liked Kafka’s work.