+254 727 693887 info@nexoriuminstitute.com
Nairobi, Kenya
Home/All Courses/Data Engineering and Architecture/Data Pipeline Engineering, ETL & ELT Training Workshop,
Data Engineering and Architecture · Physical, Online

Data Pipeline Engineering, ETL & ELT Training Workshop,

10 Days 4 sessions available Certificate on completion

Data Pipeline Engineering, ETL & ELT Training Workshop is a practical programme designed to equip participants with the skills required to design, build, automate, monitor and optimise modern data pipelines. The workshop covers the complete movement of data from source systems through ingestion, transformation, validation, storage and delivery to analytical and operational destinations.

Participants explore traditional ETL (Extract, Transform, Load) and modern ELT (Extract, Load, Transform) architectures, including batch processing, incremental data loading, Change Data Capture (CDC), API integration, database integration, cloud data pipelines and automated workflow orchestration.

Course objectives

  • Design modern ETL and ELT architectures for integrating data from databases, files, APIs, applications, cloud services and enterprise systems
  • Build reliable data pipelines for batch, incremental and near-real-time data ingestion, transformation and loading.
  • Develop data transformations using SQL, Python and modern data-processing tools, including reusable and scalable transformation workflows.
  • Automate and orchestrate data workflows using technologies such as Microsoft Fabric Data Factory, Azure Data Factory, Apache Airflow and Databricks Workflows.
  • Implement production-ready data pipelines with data-quality validation, error handling, monitoring, logging, security, testing and performance optimisation.

Who should attend

Data Engineers, Data Pipeline Engineers, ETL Developers, ELT Developers, Data Integration Specialists, Data Architects, Cloud Data Engineers, Data Platform Engineers, Database Administrators, Database Developers, SQL Developers, Business Intelligence Developers, Analytics Engineers, Data Analysts, Data Scientists, Machine Learning Engineers, Software Engineers, Cloud Engineers, DevOps Engineers, Data Warehouse Developers, Data Lake Engineers, IT Managers, ICT Managers, Technology Managers, Data Governance Professionals, Data Quality Professionals, Enterprise Architects, Solutions Architects, Digital Transformation Professionals, and IT professionals transitioning into modern data engineering and cloud data roles

Course content

  • Module 1: Foundations of Modern Data Pipeline Engineering Data pipeline concepts, data integration fundamentals, data pipeline lifecycle, ETL architecture, ELT architecture, traditional versus modern data pipelines, operational data sources, analytical d
  • Module 2: Data Sources & Data Connectivity Relational databases, SQL Server, PostgreSQL, MySQL, Oracle, NoSQL databases, CSV files, Excel files, JSON, XML, Parquet, cloud storage, SaaS applications, enterprise applications, REST APIs, web services, IoT d
  • Module 3: Data Extraction & Ingestion Engineering Data extraction strategies, full extraction, incremental extraction, scheduled ingestion, batch ingestion, file ingestion, database ingestion, API ingestion, cloud ingestion, source authentication, connec
  • Module 4: ETL Architecture & Implementation Extract processes, transformation stages, staging areas, transformation engines, loading strategies, ETL workflows, ETL architecture patterns, source-to-target mapping, transformation rules, data cleansing, dat
  • Module 5: ELT Architecture & Cloud-Native Data Pipelines ELT concepts, ETL versus ELT, cloud-scale processing, loading raw data first, transformation after ingestion, warehouse-based transformations, lakehouse transformations, SQL transformations, Spark
  • Module 6: Data Transformation with SQL SQL for data pipelines, filtering, joins, aggregations, subqueries, Common Table Expressions, window functions, CASE expressions, data type conversion, string manipulation, date transformations, deduplication, condi
  • Module 7: Python for Data Pipeline Development Python for data engineering, Pandas, PyArrow, file processing, database connectivity, API consumption, JSON processing, data transformation, automated scripts, reusable functions, exception handling, logging
  • Module 8: Data Cleaning, Preparation & Transformation Missing-value handling, duplicate records, invalid values, inconsistent formats, data type conversion, standardisation, outlier handling, reference-data matching, record linkage, data enrichment, busi
  • Module 9: Incremental Loading & Change Data Capture Full versus incremental loading, watermarking, high-water marks, timestamp-based extraction, key-based extraction, Change Data Capture, insert/update/delete detection, source-system changes, incremental
  • Module 10: Data Pipeline Orchestration & Workflow Automation Pipeline orchestration, workflow dependencies, scheduling, triggers, parameterised pipelines, reusable pipelines, conditional execution, parallel activities, sequential workflows, pipeline vari
  • Module 11: Microsoft Fabric & Azure Data Pipeline Engineering Microsoft Fabric Data Factory, Azure Data Factory, connectors, Copy jobs, Dataflow Gen2, pipelines, notebooks, OneLake, Lakehouse ingestion, data movement, transformation activities, pipeline
  • Module 12: Apache Airflow & Advanced Workflow Orchestration Apache Airflow fundamentals, Directed Acyclic Graphs, DAG design, tasks, operators, dependencies, scheduling, task parameters, retries, failure handling, sensors, workflow monitoring, Airflow wi
  • Module 13: Databricks, Spark & Distributed Data Pipelines Apache Spark fundamentals, Spark DataFrames, Spark SQL, PySpark, distributed processing, Spark transformations, Spark jobs, partitioning, shuffling, caching, large-scale transformations, Databrick
  • Module 14: Analytics Engineering & dbt-Based Transformation Analytics engineering concepts, dbt architecture, dbt projects, sources, models, staging models, intermediate models, marts, SQL transformations, incremental models, dbt tests, documentation, de
  • Module 15: Data Quality & Pipeline Validation Data-quality dimensions, completeness, accuracy, consistency, validity, uniqueness, data profiling, validation rules, schema validation, automated testing, source-to-target reconciliation, record counts, dupl
  • Module 16: Pipeline Error Handling, Monitoring & Observability Pipeline logging, execution monitoring, job monitoring, performance metrics, error detection, retry mechanisms, failure recovery, alerting, pipeline health, data freshness, latency monitoring
  • Module 20: AI-Assisted Data Pipeline Engineering Generative AI for pipeline development, AI-assisted SQL, AI-assisted Python, AI-generated transformation logic, natural-language pipeline design, AI-assisted debugging, pipeline documentation, automated da
  • Module 18: Data Pipelines for Analytics, Machine Learning & AI Analytics-ready datasets, BI data pipelines, semantic models, machine-learning datasets, feature engineering pipelines, model-training data, AI-ready data, document-processing pipelines, unst

General notes

  • The programme combines Python, SQL, cloud data integration, data transformation, pipeline orchestration and modern analytics engineering. Practical implementation can include Microsoft Fabric Data Factory, Azure Data Factory, Databricks, Apache Airflow, dbt, Apache Spark/PySpark and cloud storage platforms. Microsoft Fabric, for example, currently supports data movement, Dataflow Gen2 transformations, notebooks, pipelines and dbt-based transformations within integrated data workflows.
  • Module 22: Enterprise Data Pipeline Capstone Project Business requirements analysis, source-system assessment, pipeline architecture, source-to-target mapping, ingestion implementation, ETL/ELT transformation, data-quality framework, orchestration, monitoring, error handling, security, performance optimisation, BI integration, AI-ready data preparation, technical documentation, deployment and presentation of an end-to-end production-style data pipeline.

Frequently asked questions

Yes. Data-quality checks, schema validation, duplicate detection, missing-data checks, source-to-target reconciliation, pipeline logging, error handling, alerts, monitoring and data observability are covered to help participants develop reliable production pipelines.
The workshop is suitable for data engineers, ETL/ELT developers, data integration specialists, database administrators, data architects, cloud engineers, analytics engineers, BI professionals, data analysts, software developers, data scientists, DevOps professionals, IT managers and professionals seeking to develop practical data pipeline engineering skills.
Participants will learn how to design, develop, automate, monitor and optimise modern data pipelines. The workshop covers data extraction, ingestion, ETL, ELT, data transformation, incremental loading, Change Data Capture, workflow orchestration, data-quality validation, error handling, pipeline monitoring and production deployment.
ETL (Extract, Transform, Load) transforms data before it is loaded into the target system, while ELT (Extract, Load, Transform) loads data into a scalable target platform before transformation takes place. The workshop covers both approaches and helps participants understand when each architecture is appropriate.
Depending on the training configuration, participants can work with Python, SQL, Pandas, Apache Spark, PySpark, Microsoft Fabric Data Factory, Azure Data Factory, Azure Databricks, Apache Airflow, dbt, SQL Server, PostgreSQL, REST APIs, cloud storage, Delta Lake and Git/GitHub.
From —
View sessions
Chat with us Join Inner Circle