Apache Kafka & Streaming Distributed Messaging Training

Topic

Apache Kafka & Streaming Distributed Messaging Training

Scope

Confluence Kafka Enterprise 5.x

Kafka 2.x Version

Level

Intermediate and Advanced

Languages

Java 1.8

Days

3 days or 24 hours 

Virtual Machine

Linux OS [Mint/Ubuntu OS Based]

Host Machine

Windows/Linux/Mac

Host Machine RAM

16 GB if developer wants to use Virtual Machine

RAM

8 GB if developer can use Linux/Mac machine directly without Host Machine

Setup Requirements

for Docker setup

16 GB RAM

40-50 GB GB Free harddrive

OS: Windows 10/Ubuntu Desktop/Mac

Virtualization

Virtualization to be enabled for VirtualBox

Training Setup

Trainer shall provide pre-installed VirtualBox Linux (GUI) image which has Java JDK 1.8, Scala, SBT, Maven,  IntelliJ, Confluent Kafka, PostgreSQL, Apache Spark, Dockers pre-installed.

The Lab should have facility to copy 20GB of images to every test machines through network or USB

Cloud Machine

Trainer provides cloud machine for each participant with 160 GB SSD Harddrive, 4 vCPU and 16 GB RAM for participants, cost about 5 Euros per student. The VM shall be available for 4 Days total during training period.

Participants Pre-requites

Languages

Good Working knowledge on Java or Scala Languages

Basic/Exposure to Reactive Programming for Streaming

JDK

Good Knowledge on Using JDK, Compiler Setup, either Java/Scala languages

Maven & POM XML

Good Knowledge on Maven & POM XML files handling

Scala SBT if Scala is preferred

Good knowledge on Scala SBT

Highspeed Internet

Access to Highspeed Internet

Editor

Good expertise in using IntelliJ/Eclipse Editor

Preferred, but not mandatory

High level understanding of AMQP

Understanding Producer and Consumer Design Patterns

Message Queuing

Distributed Systems

Stream vs Batch Processing

SSH

Gitbash for windows

SSH client for Linux/Mac/Windows

Kafka Basic to Intermediate

Introduction

Introduction to Big Data

Big Data and Streaming

Kafka Introduction

Kafka Features

Kafka vs AMQP

Kafka Features

Kafka Use Case 1

Understanding Big Data and Data Streaming Problem Use Cases

Introduction to Data Ingestion

Messaging System

Data Streaming

Batch Processing

Live Data Processing

Data Forwarding, Queuing problems

Publish/Subscribe patterns

Workflow Application

Hadoop and Map Reduce Integration

Connected, Disconnected Architecture

Persistence issues with large data processing

AMQP/MQTT to Kafka Bridge

Kafka Architecture

Elements of Kafka [Consumer, Producer, Broker, ZooKeeper, Cluster]

Kafka Architectures

Kafka Overall Messaging Architecture

Kafka Streaming Architecture

Kafka Consumer and Producer Architecture

Kafka Persistence Architecture

Kafka Clusters

Kafka Consumer Workers Group

Kafka Distribution

Apache Kafka

Confluence Kafka Open Source

Confluence Kafka Enterprise

Kafka Setup

[Hands-on]

Java JDK

Kafka Setup

Zoo Keeper Configuration

Single Broker Configuration

Understanding build-in tools, Kafka server, test publisher, test consumer

Configuration Kafka Single Node Broker

Start ZooKeeper

Start Kafka Server

Using Example console consumer and publisher

Broker, Consumer, Producer Configuration

Zoo Keeper connection

Broker ID

Logs

Port

Consumer Configuration

Producer Configuration

Kafka Administrator Commands

[Hands-on]

List all topics

List all brokers

Create Topics

Change Topics

Delete Topics

Topic  Description

Topics & Partitions

Understanding Topics

Understanding Partitions

Understanding Replications

Producer

[Hands-on]

Kafka Producer Introduction

Kafka Producer API

Writing First Producer

Kafka Producer Class

Build a Kafka Producer

Configure Producers

Kafka Topics, Partitions Configuration

Send messages to Kafka

Send messages Synchronously

Send Message Asynchronously and process the async acknowledgement

Producing Keyed and Non-Keyed Messages

Serializers, Key/Value Serializer, ProducerRecord

Serialize Using Apache Avro, JSON

Working with Multiple Brokers

Consumer

[Hands-on]

Kafka Consumer Introduction

Consumers and Consumer Groups

Subscribe for Topics

Consumer Records

Achieving higher performance with Consumer group

De-serialize key and values

Consumer Groups and Partition Rebalance

Creating a Kafka Consumer

The Poll Loop

Configuring Consumers

Commits and Offsets

Rebalance Listeners

Consuming Records with Specific Offsets

Kafka Internal Architecture, Storages, Cluster, Partitions in Depth

Cluster Membership

The Controller

Replication workflow, Replication Internals

Request Processing

Physical Storage

Reliability 

Partition allocation and number of partitions

Broker Configuration

Using Producers in a Reliable System

Using Consumers in a Reliable System

Validating System Reliability

Performance Tuning in Kafka

Intermediate to Advanced

Cluster Architecture and Workflow

Kafka Use Case

Revising Cluster, Producer, Consumer, Broker, ZooKeeper

Data Flow Between all actors in Kafka Systems

Creating Clusters with Multiple Brokers

Broker IDs and Master/Slave Brokers

Fail over handling in clusters

Understanding Leader Broker

Balancing leadership

Cluster Administration and Configuration

[Hands-on]

Configure Cluster

Replication

High Availability

Multi-Cluster Architectures

Apache Kafka’s MirrorMaker

Topic Operations

Partition in Depth

[Hands-on]

Topics and Partitions

Understanding partition numbers

Default partitions

Explicit Partitions

Under the hood partitions

Replications

Understanding Replications

Replication Factors and High Availability

Messages in Details

[Hands-on]

Message Models

Message Serialize

JSON handling

Message IDs

Persistence

[Hands-on]

Persisting Messages on File System

Offline Consumer and Subscription handling

Durability of Messages

Failover and Persistence

Kafka Streaming

4 Hours

Stream Processing

Stream-Processing Concepts

Stream-Processing Design Patterns

Kafka Streams by Example

Kafka Streams: Architecture Overview

KStream

KTable

GlobalKTable

KStream to KTable

KTable to KSTream

Advanced

Kafka Monitoring

Considerations When Building Data Pipelines

 Metric Basics

 Kafka Broker Metrics

 Client Monitoring

 Lag Monitoring

 End-to-End Monitoring

Kafka Connect APIs

[Hands-on]

Kafka Connect API overview

Kafka Connect

When to Use Kafka Connect?

Kafka Connect Properties

Perform File source and sink using Kafka Connect

Kafka Connect vs Consumer/Producer model

Kafka Connect with JDBC [Requires MySQL]

Cassandra Connectivity [Optional]

Working with HDFS Connectors

Working with Elastic Search Connectors

REST Proxy

[Hands-on]

Managing REST Proxy Configuration

Kafka Connect Configuration & Administration

[Hands-on]

Managing Kafka Connect

Managing Sink/Sources

Configuration and settings

Zoo Keeper Administration

[Hands-on]

Managing Zoo Keeper Cluster

Zoo keeper lead, configurations

Confluence Control Centre

[Hands-on]

Introduction to Confluent Control Center

Managing Control Center and Dashboard

Schema Registry Configuration & Administration

[Hands-on]

Managing Schema Registry services

Kafka Integrate with Apache Spark [Demo only] [Optional]

Integrate Apache Spark with Kafka

Spark Streaming Jobs and Kafka Stream

Mirror Maker and Confluent Replicator for Data Centre Backup and Replication

Mirror Maker Replication

Confluent Replicator

Setup and demo

Production

Deployment

Configuration, cleaner installation