Apache Spark & PySpark for Big Data Engineering Training Workshop is a practical, hands-on programme designed to equip participants with the skills required to process, transform, analyse and engineer large-scale datasets using Apache Spark and PySpark.
The workshop introduces the architecture of Apache Spark and progressively develops practical skills in Spark DataFrames, Spark SQL, PySpark transformations, joins, aggregations, window functions, distributed processing, partitioning, caching, performance optimisation and large-scale ETL/ELT. PySpark provides a Python interface to Spark, while DataFrames are a central abstraction for working with distributed datasets.
Participants also explore Spark's role in modern data platforms, including data lakes, lakehouses, Delta Lake, cloud data platforms, Databricks, Microsoft Fabric and large-scale data pipelines. The workshop can include practical processing of CSV, JSON, Parquet and database data and demonstrate how Spark connects to external data sources.
- Understand and apply Apache Spark architecture and distributed-computing concepts for processing large datasets.
- Build large-scale ETL and ELT pipelines that integrate files, databases, APIs, data lakes and cloud data platforms.
- Develop scalable data-processing applications using PySpark, Spark DataFrames and Spark SQL.
- Optimise Spark workloads using partitioning, caching, broadcast joins, query optimisation, appropriate file formats and efficient resource utilisation.
- Develop production-oriented Spark solutions including batch processing, Structured Streaming, monitoring, data-quality controls and integration with modern lakehouse and AI data platforms.
Data Engineers, Big Data Engineers, Data Scientists, Data Analysts, Machine Learning Engineers, ETL/ELT Developers, Data Architects, Cloud Data Engineers, Database Administrators, Database Developers, Analytics Engineers, BI Professionals, Software Developers, Python Developers, Cloud Engineers, DevOps Engineers, Data Platform Engineers, Data Warehouse Developers, Data Lake Engineers, MLOps Professionals, IT Managers, ICT Managers, Technology Managers, Research Data Professionals, and IT professionals seeking practical Apache Spark and PySpark skills.
- Module 1: Big Data Engineering & Apache Spark Foundations Big data concepts, characteristics of big data, limitations of traditional processing, distributed computing, Hadoop versus Spark, Apache Spark ecosystem, Spark use cases, Spark workloads, Spark a
- Module 2: Setting Up the Spark & PySpark Environment Apache Spark installation, PySpark installation, Python environment configuration, Jupyter notebooks, Spark shell, local Spark mode, cluster environments, Databricks notebooks, Microsoft Fabric noteboo
- Module 3: PySpark Programming Fundamentals Python for PySpark, SparkSession, DataFrame creation, schema definition, data types, columns, expressions, selecting columns, filtering records, creating calculated columns, renaming columns, dropping columns, s
- Module 4: Spark DataFrames & Data Manipulation DataFrame architecture, DataFrame operations, column transformations, conditional expressions, string functions, date functions, mathematical functions, null handling, duplicate records, data profiling, Data
- Module 5: Spark SQL & Advanced Querying Spark SQL fundamentals, temporary views, SQL queries, filtering, joins, aggregations, subqueries, Common Table Expressions, window functions, analytical functions, SQL and PySpark integration, query execution, SQL
- Module 6: Reading & Writing Big Data Formats CSV processing, JSON processing, Parquet, ORC, Avro concepts, text files, compressed files, schema inference, explicit schemas, partitioned datasets, reading from directories, writing datasets, append and over
- Module 7: Data Cleaning & Transformation with PySpark Missing-value treatment, duplicate removal, data-type conversion, standardisation, data validation, outlier handling, record matching, data enrichment, conditional transformations, complex data types,
- Module 8: Spark Joins, Aggregations & Window Functions Inner joins, left joins, right joins, full joins, semi joins, anti joins, cross joins, join optimisation, groupBy operations, aggregations, multiple aggregations, window specifications, ranking, runn
- Module 9: Spark Architecture & Distributed Data Processing Spark execution model, lazy evaluation, transformations and actions, jobs, stages and tasks, DAGs, driver and executor processes, cluster managers, task scheduling, serialization, shuffle operati
- Module 10: Partitioning, Shuffling & Data Distribution Spark partitions, partition sizing, repartitioning, coalescing, shuffle operations, narrow transformations, wide transformations, data distribution, partition pruning, skewed data, shuffle optimisati
- Module 11: Advanced Spark Performance Optimisation Caching, persistence, broadcast variables, broadcast joins, predicate pushdown, column pruning, partition pruning, query optimisation, Adaptive Query Execution, memory management, serialization, file opt
- Module 12: Apache Spark for ETL & ELT Engineering Spark-based ETL, Spark-based ELT, data ingestion, staging datasets, transformation pipelines, data cleansing, enrichment, aggregation, loading analytical datasets, incremental processing, reusable Spark p
- Module 13: Spark with Data Lakes & Lakehouse Platforms Spark and data lakes, cloud object storage, Azure Data Lake Storage, Amazon S3, Google Cloud Storage, OneLake, Delta Lake, Apache Iceberg, Bronze/Silver/Gold architecture, transactional data processi
- Module 14: Spark Data Integration & External Data Sources JDBC connectivity, relational databases, SQL Server, PostgreSQL, MySQL, cloud databases, APIs, external data sources, database ingestion, reading and writing tables, schema mapping, query pushdown
- Module 15: Apache Spark Structured Streaming Batch versus streaming, streaming architecture, Structured Streaming, streaming DataFrames, streaming sources, streaming sinks, checkpoints, output modes, event-time processing, windows, watermarks, late-arriv
- Module 16: Spark with Kafka & Real-Time Data Engineering Apache Kafka fundamentals, topics, partitions, producers and consumers, Spark-Kafka integration, consuming Kafka streams, transforming streaming data, writing streaming outputs, checkpointing, faul
- Module 17: Spark for Machine Learning & AI Data Engineering Spark ML concepts, MLlib, feature engineering, large-scale dataset preparation, training-data pipelines, feature transformation, distributed machine learning workflows, model-data preparation, M
- Module 18: Spark on Databricks & Modern Cloud Platforms Databricks architecture, Spark clusters, notebooks, compute resources, Spark jobs, workflows, Delta Lake, cloud data storage, Databricks data engineering, job scheduling, monitoring, collaborative d
- Module 19: Spark Production Engineering, Monitoring & Troubleshooting Spark application monitoring, Spark UI, logs, stages, tasks, executor metrics, failed jobs, memory errors, slow jobs, skewed partitions, shuffle problems, serialization issues, perform
- Module 20: Enterprise Big Data Engineering Capstone Big-data requirements analysis, source-system assessment, Spark architecture design, large-scale ingestion, PySpark transformation, Spark SQL analytics, data-quality validation, partitioning, performanc
- Recommended Technology & Tools Apache Spark, PySpark, Spark SQL, Python, Jupyter Notebook, Databricks, Microsoft Fabric, Delta Lake, Apache Iceberg, Apache Kafka, Hadoop/HDFS concepts, SQL Server, PostgreSQL, MySQL, Parquet, JSON, CSV, ORC, JDBC, Azure Data Lake Storage, Amazon S3, Google Cloud Storage, OneLake, Power BI, Git/GitHub and MLlib.
- PySpark data-processing applications, Spark SQL analytical queries, large-scale ETL/ELT pipelines, database-to-Spark ingestion pipelines, CSV/JSON/Parquet processing workflows, data-lake processing pipelines, Delta Lake transformation workflows, Spark performance optimisation plans, Structured Streaming pipelines, Kafka-to-Spark streaming workflows, machine-learning data preparation pipelines, Databricks Spark jobs, production-oriented Spark monitoring framework, End-to-End Big Data Engineering Capstone Project