Modern Data Engineering & Cloud Data Platforms Training

Course overview

Modern Data Engineering & Cloud Data Platforms Training Workshop is a practical, technology-focused programme designed to equip professionals with the skills required to build, integrate, operate and optimise modern enterprise data platforms. The workshop explores the complete data engineering lifecycle, from data ingestion and integration to cloud storage, data transformation, pipeline orchestration, lakehouse architecture, distributed processing, real-time data streaming, analytics and AI-ready data platforms. Participants gain practical exposure to modern technologies including Python, SQL, Apache Spark and PySpark, cloud data platforms, data lakes, lakehouses, Delta Lake, data pipelines and modern orchestration tools.

The workshop also examines contemporary approaches to building scalable and reliable data infrastructure, including batch and streaming pipelines, ETL/ELT, Change Data Capture (CDC), medallion architectures, data quality, observability, metadata, security, governance, performance optimisation and cloud cost management. Practical exercises can be delivered using platforms such as Microsoft Fabric, Azure, Databricks, AWS and Google Cloud, enabling participants to understand both platform-specific capabilities and transferable data engineering principles. Modern Fabric learning pathways, for example, combine lakehouse development, Spark, Delta tables, ingestion, orchestration and real-time intelligence within an end-to-end data engineering environment.

Course objectives

Design modern, scalable data engineering architectures using cloud data platforms, data lakes, lakehouses and appropriate storage and processing technologies.
Build and manage reliable data pipelines using modern ETL/ELT approaches for batch, incremental and real-time data ingestion, transformation and integration.
Apply modern data engineering technologies including Python, SQL, Apache Spark/PySpark, cloud-native data services and contemporary orchestration and pipeline tools.
Implement production-ready data platforms using data quality, metadata, observability, security, governance, monitoring, performance optimisation and cloud cost-management practices.
Develop data platforms that support analytics, AI and machine learning by creating trusted, scalable and analytics-ready data products and pipelines for business intelligence, advanced analytics and AI workloads.

Who should attend

### Who Should Attend? **Data Engineers, Junior and Senior Data Engineers, Data Architects, Cloud Data Architects, Data Platform Engineers, Cloud Engineers, Big Data Engineers, Data Analysts, Analytics Engineers, Business Intelligence Professionals, ETL/ELT Developers, Database Administrators, Database Developers, Data Scientists, Machine Learning Engineers, MLOps Professionals, Software Engineers, DevOps Engineers, Solutions Architects, Enterprise Architects, Data Integration Specialists, Data Warehouse Developers, Data Lake Administrators, Data Platform Administrators, IT Managers, ICT Managers, Technology Managers, Digital Transformation Professionals, Business Analysts, Research Data Professionals, Data Governance Professionals, Data Quality Professionals, and IT professionals transitioning into data engineering and cloud data roles.**

Course content

1. Modern Data Engineering Foundations & Data Platform Architecture: evolution of data engineering, modern data engineering lifecycle, role of the data engineer, operational databases, analytical databases, data warehouses, data lakes, lakehouses, cloud d
2. Python & SQL for Modern Data Engineering: Python programming for data engineering, Python data structures, Pandas, PyArrow, file processing, database connectivity, API integration, SQL fundamentals, advanced SQL, joins, aggregations, window functions,
3. Data Sources, APIs & Modern Data Ingestion: relational databases, NoSQL databases, CSV, JSON, XML, Parquet, application data, enterprise systems, REST APIs, SaaS data sources, cloud applications, IoT data, sensor data, file ingestion, database ingestio
4. ETL, ELT & Data Transformation Engineering: ETL architecture, ELT architecture, extraction, data cleansing, data standardisation, data enrichment, deduplication, record matching, missing-data handling, data type conversion, business rules, incremental
5. Cloud Data Storage & Data Lake Engineering: cloud object storage, Azure Data Lake Storage, Amazon S3, Google Cloud Storage, Microsoft OneLake, data lake architecture, landing zones, raw data zones, staging zones, curated data zones, structured data, se
6. Lakehouse Architecture & Modern Data Engineering: data lakes versus data warehouses, data lakehouses, lakehouse architecture, medallion architecture, Bronze layer, Silver layer, Gold layer, Delta Lake, Apache Iceberg, ACID transactions, schema enforcem
7. Apache Spark & PySpark for Large-Scale Data Engineering: distributed computing, Apache Spark architecture, Spark DataFrames, Spark SQL, PySpark, transformations, actions, joins, aggregations, window functions, partitioning, shuffling, lazy evaluation,
8. Data Pipelines, Orchestration & Workflow Automation: data pipeline architecture, pipeline dependencies, scheduling, triggers, parameterised pipelines, reusable pipeline templates, conditional workflows, parallel processing, error handling, retry mechan
9. Data Modelling & Analytics Engineering: data modelling concepts, conceptual data models, logical data models, physical data models, dimensional modelling, fact tables, dimension tables, star schemas, snowflake schemas, normalisation, denormalisation, a
10. Real-Time Data Engineering & Streaming: batch versus streaming, event-driven architectures, real-time data pipelines, Apache Kafka, Kafka producers, Kafka consumers, Kafka topics, Kafka partitions, event streaming, stream processing, Spark Structured
11. Data Quality, Data Contracts & Data Observability: data quality dimensions, data profiling, automated data validation, quality rules, schema validation, data contracts, producer-consumer agreements, data freshness, pipeline health, data lineage, data
12. Data Governance, Security, Metadata & Lineage: data governance, data ownership, data stewardship, technical metadata, business metadata, metadata cataloguing, data discovery, data lineage, data provenance, data classification, role-based access contro
13. DataOps, DevOps & Production Data Engineering: DataOps principles, development environments, testing environments, production environments, Git, GitHub, version control, CI/CD, automated data testing, infrastructure as code, deployment pipelines, pipe
14. AI-Assisted Data Engineering & Intelligent Development: generative AI for data engineering, AI-assisted SQL development, AI-assisted Python development, AI-assisted PySpark development, AI-generated pipeline logic, AI-assisted documentation, AI-powere
15. Data Engineering for Machine Learning & AI: AI-ready data pipelines, machine learning datasets, training datasets, validation datasets, testing datasets, feature engineering, feature pipelines, feature stores, data and model lineage, ML data versionin
16. Generative AI Data Engineering & RAG Data Pipelines: enterprise AI data preparation, document ingestion, PDF processing, Word document processing, HTML processing, unstructured data pipelines, text extraction, text cleaning, document chunking, metadat
17. Modern Cloud Data Engineering Platforms: Microsoft Fabric, OneLake, Azure Data Lake Storage, Azure Data Factory, Azure Databricks, Amazon S3, AWS Glue, Amazon Redshift, Google Cloud Storage, BigQuery, Databricks Lakehouse, Snowflake, cloud-native Spar
18. Performance Engineering, Scalability & Cloud Cost Optimisation: data pipeline performance, Spark optimisation, partitioning, file-size optimisation, data clustering, query optimisation, caching, compute optimisation, storage optimisation, workload man
19. Data Platform Monitoring, Reliability & Disaster Recovery: data platform observability, pipeline monitoring, infrastructure monitoring, logging, metrics, alerts, service-level objectives, failure detection, retry strategies, recovery procedures, backu
20. Enterprise Data Engineering Capstone & AI-Ready Cloud Data Platform: requirements analysis, source-system assessment, enterprise data architecture, ingestion architecture, cloud storage, lakehouse implementation, batch pipelines, streaming pipelines,

Frequently asked questions

1. 1. Who should attend the Modern Data Engineering & Cloud Data Platforms Training Workshop?

The workshop is designed for data engineers, data architects, cloud engineers, database administrators, data analysts, BI professionals, data scientists, software engineers, ETL/ELT developers, DevOps professionals, IT managers, data platform administrators and technology professionals seeking practical skills in modern data engineering and cloud data platforms

2. 2. What will participants learn during the workshop?

Participants will learn how to design and implement modern data platforms, build ETL and ELT pipelines, ingest data from databases and APIs, work with cloud storage and data lakes, develop lakehouse architectures, process large datasets using Apache Spark and PySpark, automate data workflows, implement data quality and governance, and prepare reliable data platforms for analytics, machine learning and AI.

3. 3. Which technologies and cloud platforms are covered?

The workshop can include technologies such as Python, SQL, Apache Spark, PySpark, Delta Lake, Microsoft Fabric, OneLake, Azure Data Factory, Azure Databricks, Apache Airflow, dbt, Apache Kafka and cloud data storage platforms. Microsoft Fabric, for example, currently provides lakehouse, Spark, notebooks and data pipelines as core data-engineering capabilities.

4. 4. Does the workshop cover data lakes and lakehouse architecture?

Yes. Participants explore modern data lake and lakehouse architectures, including cloud object storage, medallion architecture, Bronze/Silver/Gold data layers, Delta Lake, data transformation and analytics-ready datasets. Modern lakehouses combine scalable data-lake storage with analytical processing capabilities and can support BI, machine learning and AI workloads.

5. 5. Does the training cover AI and machine learning data engineering?

Yes. The workshop incorporates AI-ready data engineering, including preparation of machine-learning datasets, feature pipelines, data quality for AI, ML data pipelines, generative AI data preparation, document-processing pipelines, embeddings, vector databases and Retrieval-Augmented Generation (RAG) data pipelines. The emphasis is on building reliable data foundations that can support modern analytics, machine learning and generative AI applications.

Schedule & fees

Dates	Location	Duration	Fee (USD)
04/01/2027 - 15/01/2027	Nairobi,kenya or Virtual	10 Days	$3,000
08/02/2027 - 19/02/2027	Nairobi,Kenya	10 Days	$3,000
08/06/2027 - 18/06/2027	Rwanda,Kigali	10 Days	$3,800