080 41714080 info@consultkpi.com

Apache Kafka & Streaming Distributed Messaging Training

Apache Kafka & Streaming Distributed Messaging training covers Basic, Intermediate and Advanced level Kafka , its implementation and Application Development.

Advanced 0 (0 Rating) 0 Students enrolled
Created by KPI Consulting Last updated Mon, 08-Jun-2020 English
What will i learn?
  • This Instructor-led Live training will provide complete knowledge on Kafka from basic to intermediate to advance
  • 80:20 Hands-on sessions | Strong usecases along with hands-on sessios

Curriculum for this course
0 Lessons 00:00:00 Hours
+ View more


Apache Kafka & Streaming Distributed Messaging Training


Confluence Kafka Enterprise 5.x

Kafka 2.x Version


Intermediate and Advanced


Java 1.8


3 days or 24 hours 

Virtual Machine

Linux OS [Mint/Ubuntu OS Based]

Host Machine


Host Machine RAM

16 GB if developer wants to use Virtual Machine


8 GB if developer can use Linux/Mac machine directly without Host Machine

Setup Requirements

for Docker setup


40-50 GB GB Free harddrive

OS: Windows 10/Ubuntu Desktop/Mac


Virtualization to be enabled for VirtualBox

Training Setup

Trainer shall provide pre-installed VirtualBox Linux (GUI) image which has Java JDK 1.8, Scala, SBT, Maven,  IntelliJ, Confluent Kafka, PostgreSQL, Apache Spark, Dockers pre-installed.


The Lab should have facility to copy 20GB of images to every test machines through network or USB

Cloud Machine

Trainer provides cloud machine for each participant with 160 GB SSD Harddrive, 4 vCPU and 16 GB RAM for participants, cost about 5 Euros per student. The VM shall be available for 4 Days total during training period.

Participants Pre-requites


Good Working knowledge on Java or Scala Languages

Basic/Exposure to Reactive Programming for Streaming


Good Knowledge on Using JDK, Compiler Setup, either Java/Scala languages

Maven & POM XML

Good Knowledge on Maven & POM XML files handling

Scala SBT if Scala is preferred

Good knowledge on Scala SBT

Highspeed Internet

Access to Highspeed Internet


Good expertise in using IntelliJ/Eclipse Editor

Preferred, but not mandatory

High level understanding of AMQP

Understanding Producer and Consumer Design Patterns

Message Queuing

Distributed Systems

Stream vs Batch Processing


Gitbash for windows

SSH client for Linux/Mac/Windows

Kafka Basic to Intermediate


Introduction to Big Data

Big Data and Streaming

Kafka Introduction

Kafka Features

Kafka vs AMQP

Kafka Features

Kafka Use Case 1

Understanding Big Data and Data Streaming Problem Use Cases

Introduction to Data Ingestion

Messaging System

Data Streaming

Batch Processing

Live Data Processing

Data Forwarding, Queuing problems

Publish/Subscribe patterns

Workflow Application

Hadoop and Map Reduce Integration

Connected, Disconnected Architecture

Persistence issues with large data processing

AMQP/MQTT to Kafka Bridge

Kafka Architecture

Elements of Kafka [Consumer, Producer, Broker, ZooKeeper, Cluster]

Kafka Architectures

Kafka Overall Messaging Architecture

Kafka Streaming Architecture

Kafka Consumer and Producer Architecture

Kafka Persistence Architecture

Kafka Clusters

Kafka Consumer Workers Group

Kafka Distribution

Apache Kafka

Confluence Kafka Open Source

Confluence Kafka Enterprise

Kafka Setup



Java JDK

Kafka Setup

Zoo Keeper Configuration

Single Broker Configuration

Understanding build-in tools, Kafka server, test publisher, test consumer

Configuration Kafka Single Node Broker

Start ZooKeeper

Start Kafka Server

Using Example console consumer and publisher

Broker, Consumer, Producer Configuration

Zoo Keeper connection

Broker ID



Consumer Configuration

Producer Configuration

Kafka Administrator Commands



List all topics

List all brokers

Create Topics

Change Topics

Delete Topics

Topic  Description

Topics & Partitions

Understanding Topics

Understanding Partitions

Understanding Replications




Kafka Producer Introduction

Kafka Producer API

Writing First Producer

Kafka Producer Class

Build a Kafka Producer

Configure Producers

Kafka Topics, Partitions Configuration

Send messages to Kafka

Send messages Synchronously

Send Message Asynchronously and process the async acknowledgement

Producing Keyed and Non-Keyed Messages

Serializers, Key/Value Serializer, ProducerRecord

Serialize Using Apache Avro, JSON

Working with Multiple Brokers



Kafka Consumer Introduction

Consumers and Consumer Groups

Subscribe for Topics

Consumer Records

Achieving higher performance with Consumer group

De-serialize key and values

Consumer Groups and Partition Rebalance

Creating a Kafka Consumer

The Poll Loop

Configuring Consumers

Commits and Offsets

Rebalance Listeners

Consuming Records with Specific Offsets

Kafka Internal Architecture, Storages, Cluster, Partitions in Depth

Cluster Membership

The Controller

Replication workflow, Replication Internals

Request Processing

Physical Storage


Partition allocation and number of partitions

Broker Configuration

Using Producers in a Reliable System

Using Consumers in a Reliable System

Validating System Reliability

Performance Tuning in Kafka

Intermediate to Advanced

Cluster Architecture and Workflow



Kafka Use Case

Revising Cluster, Producer, Consumer, Broker, ZooKeeper

Data Flow Between all actors in Kafka Systems

Creating Clusters with Multiple Brokers

Broker IDs and Master/Slave Brokers

Fail over handling in clusters

Understanding Leader Broker

Balancing leadership

Cluster Administration and Configuration


Configure Cluster


High Availability

Multi-Cluster Architectures

Apache Kafka’s MirrorMaker

Topic Operations

Partition in Depth


Topics and Partitions

Understanding partition numbers

Default partitions

Explicit Partitions

Under the hood partitions


Understanding Replications

Replication Factors and High Availability

Messages in Details



Message Models

Message Serialize

JSON handling

Message IDs



Persisting Messages on File System

Offline Consumer and Subscription handling

Durability of Messages

Failover and Persistence

Kafka Streaming


4 Hours

Stream Processing

Stream-Processing Concepts

Stream-Processing Design Patterns

Kafka Streams by Example

Kafka Streams: Architecture Overview




KStream to KTable

KTable to KSTream


Kafka Monitoring

Considerations When Building Data Pipelines

 Metric Basics

 Kafka Broker Metrics

 Client Monitoring

 Lag Monitoring

 End-to-End Monitoring

Kafka Connect APIs



Kafka Connect API overview

Kafka Connect

When to Use Kafka Connect?

Kafka Connect Properties

Perform File source and sink using Kafka Connect

Kafka Connect vs Consumer/Producer model

Kafka Connect with JDBC [Requires MySQL]

Cassandra Connectivity [Optional]

Working with HDFS Connectors

Working with Elastic Search Connectors

REST Proxy



Managing REST Proxy Configuration

Kafka Connect Configuration & Administration



Managing Kafka Connect

Managing Sink/Sources

Configuration and settings

Zoo Keeper Administration



Managing Zoo Keeper Cluster

Zoo keeper lead, configurations

Confluence Control Centre



Introduction to Confluent Control Center

Managing Control Center and Dashboard

Schema Registry Configuration & Administration



Managing Schema Registry services

Kafka Integrate with Apache Spark [Demo only] [Optional]

Integrate Apache Spark with Kafka

Spark Streaming Jobs and Kafka Stream

Mirror Maker and Confluent Replicator for Data Centre Backup and Replication

Mirror Maker Replication

Confluent Replicator

Setup and demo



Configuration, cleaner installation

+ View more
Other related courses
About the instructor
  • 0 Reviews
  • 28 Students
  • 54 Courses
+ View more
This workshop is delivered by one of top most industry-leading faculty with at least 10 to 15+ years of Industry as well as training experience

KPI Consulting is one of the fastest growing (with 1000+ tech workshops) e-learning & consulting Firm which provides objective-based innovative & effective learning solutions for the entire spectrum of technical & domain skills

Student feedback
Average rating
  • 0%
  • 0%
  • 0%
  • 0%
  • 0%
₹9999 ₹15999
Buy now
  • 00:00:00 Hours On demand videos
  • 0 Lessons
  • Full lifetime access
  • Access on mobile and tv
Developed By: Monnet Digital India Pvt Ltd