Apache Kafka & Streaming Data Engineering Training Workshop

Course overview

Apache Kafka & Streaming Data Engineering Training Workshop is a practical, technology-focused programme designed to equip data engineers, developers, architects and technology professionals with the skills required to design, build, operate and optimise modern real-time data streaming platforms.

The workshop provides hands-on coverage of Apache Kafka, event streaming, producers, consumers, topics, partitions, consumer groups, replication, Kafka Connect, Kafka Streams and real-time data pipelines. Participants learn how organisations capture continuous streams of data from applications, databases, IoT devices, transactions, APIs and enterprise systems and make that data available for real-time processing and analytics.

Course objectives

• Design and implement Apache Kafka architectures using topics, partitions, producers, consumers, brokers, replication and consumer groups for scalable event-streaming workloads.
• Develop stream-processing applications using Kafka Streams and complementary technologies for filtering, transformation, aggregation, joins, windowing and stateful processing.
• Build real-time data pipelines that ingest, process, transform and distribute continuously generated data from applications, databases, APIs, IoT devices and enterprise systems.
• Implement reliable production streaming platforms using appropriate approaches to fault tolerance, data delivery, schema management, monitoring, security, performance optimisation and recovery.
• Integrate Kafka with modern data platforms and AI/analytics workflows, including data lakes, lakehouses, Spark, cloud platforms, machine learning pipelines and real-time business intelligence.

Who should attend

Course content

• 1. Real-Time Data Engineering & Event Streaming Foundations Batch versus streaming data, event-driven architectures, real-time data engineering, data-in-motion, event producers and consumers, streaming use cases, streaming architecture patterns, Kafka ec
• 2. Apache Kafka Architecture, KRaft & Cluster Fundamentals Kafka architecture, brokers, clusters, topics, partitions, records, offsets, replication, KRaft architecture, cluster configuration, fault tolerance, scalability, Kafka administration and distrib
• 3. Kafka Topics, Partitions, Producers & Data Publishing Topic design, partitioning strategies, partition keys, replication factors, retention policies, producer architecture, message keys, acknowledgements, batching, retries, idempotent producers, deliv
• 4. Kafka Consumers, Consumer Groups & Data Consumption Consumer architecture, consumer groups, partition assignment, offset management, rebalancing, parallel processing, consumer scaling, consumer lag, message ordering, error handling and reliable data c
• 5. Event Serialization, Schemas & Data Contracts JSON, Avro, Protobuf, serialization and deserialization, schema design, schema evolution, compatibility, Schema Registry concepts, event contracts, data validation and schema governance
• 6. Kafka Connect & Enterprise Data Integration Kafka Connect architecture, source and sink connectors, JDBC integration, databases, files, APIs, cloud storage, connector configuration, distributed Connect workers, connector management and streaming data
• 7. Change Data Capture & Database Streaming with Debezium CDC concepts, transaction logs, database change events, inserts, updates and deletes, Debezium architecture, PostgreSQL and SQL Server integration, database-to-Kafka pipelines, CDC event processin
• 8. Kafka Streams & Real-Time Stream Processing Kafka Streams architecture, KStream, KTable, stream transformations, filtering, mapping, grouping, aggregation, joins, stateful processing, state stores, stream topologies and real-time application developme
• 9. Event Time, Windowing & Advanced Stream Analytics Event time versus processing time, timestamps, tumbling windows, hopping windows, sliding windows, session windows, late-arriving events, stateful aggregations, stream joins, real-time metrics and cont
• 10. Python, Java & Spark for Kafka Streaming Kafka clients with Python and Java, producer and consumer applications, asynchronous processing, Kafka integration with Apache Spark and PySpark, Spark Structured Streaming, Kafka source and sink, checkpointin
• 11. Real-Time ETL, Data Quality & Streaming Pipelines Streaming ETL, data transformation, filtering, enrichment, validation, deduplication, routing, error handling, data-quality checks, stream-to-database pipelines, stream-to-data-lake pipelines and real
• 12. Event-Driven Architecture, Microservices & IoT Streaming Event-driven applications, microservices integration, asynchronous communication, domain events, event choreography, application integration, IoT data streams, sensor data, transaction events,
• 13. Kafka Reliability, Security & Governance At-most-once, at-least-once and exactly-once processing, replication, fault tolerance, recovery, authentication, authorisation, TLS, ACLs, access control, data protection, schema governance, event governance a
• 14. Kafka Performance, Monitoring & Production Operations Throughput and latency optimisation, partition sizing, producer and consumer optimisation, broker performance, consumer lag, monitoring, logging, metrics, alerting, troubleshooting, capacity plann
• 15. Enterprise Real-Time Streaming Engineering Capstone End-to-end implementation of a real-time streaming platform incorporating Kafka topics, producers, consumers, consumer groups, Kafka Connect, CDC, stream processing, data transformation, data qualit

General notes

• The programme progresses from Kafka fundamentals into advanced streaming engineering, including event-driven architecture, stream processing, event time, windowing, stateful processing, data serialization, schema management, fault tolerance, exactly-once processing, stream monitoring, security and production deployment. Kafka Streams supports operations such as transformations, joins and aggregations over continuously updating streams.
• The workshop also explores integration with modern data engineering technologies such as Python, Java, Apache Spark, PySpark, Kafka Connect, Debezium, PostgreSQL, SQL Server, cloud platforms, data lakes, lakehouses, Microsoft Fabric, Databricks and Power BI.
• The course is particularly relevant to organisations implementing real-time analytics, fraud detection, financial transaction processing, IoT monitoring, logistics tracking, customer-event processing, microservices, operational intelligence and AI-ready streaming data platforms. Apache Kafka's event-streaming architecture is designed for high-throughput, distributed and fault-tolerant real-time data processing.
• Recommended Technology Stack Apache Kafka, Kafka Connect, Kafka Streams, KRaft, Python, Java, PySpark, Apache Spark Structured Streaming, Debezium, Confluent ecosystem concepts, PostgreSQL, SQL Server, REST APIs, JSON, Avro, Protobuf, Schema Registry, Docker, Kubernetes concepts, Git/GitHub, Microsoft Fabric, Azure, Databricks, Delta Lake, cloud object storage, Power BI, Prometheus/Grafana concepts, and modern data-lake/lakehouse platforms.

Frequently asked questions

Schedule & fees