Data Engineering and Architecture · Physical, Online
Enterprise Data Lake Architecture & Management Training Workshop
10 Days4 sessions available Certificate on completion
Data Lake Architecture, Management & Governance Training is a practical professional development programme designed to equip data professionals, IT teams, analysts and technology managers with the knowledge and skills required to design, implement, manage and govern modern enterprise data lakes. The course covers data lake architecture, data ingestion, data storage, metadata and data cataloguing, data quality, data governance, security and access control, data integration, data processing, analytics and data lakehouse architectures. Participants will explore how organisations can build scalable and well-governed data platforms capable of supporting business intelligence, advanced analytics, artificial intelligence and machine learning. The training also addresses data lifecycle management, performance optimisation, cost management, lineage, monitoring and responsible data use, providing a practical foundation for organisations seeking to establish secure, scalable and analytics-ready data lake and modern data engineering environments.
Course objectives
Design and evaluate modern data lake architectures for collecting, storing, processing and managing large volumes of structured, semi-structured and unstructured data.
Implement effective data ingestion, integration and storage strategies, including batch and streaming pipelines, data organisation, partitioning and scalable storage architectures.
Apply data governance, quality, metadata, lineage and security practices to improve the reliability, discoverability, privacy and controlled access of data within enterprise data lakes.
Manage and optimise data lake environments by applying appropriate processing technologies, monitoring approaches, performance optimisation, lifecycle management and cost-control techniques.
Develop analytics- and AI-ready data platforms by integrating data lakes with business intelligence, advanced analytics, machine learning and modern data lakehouse architectures to support evidence-based organisational decision-making.
Who should attend
Data Engineers, Data Architects, Data Platform Engineers, Data Management Professionals, Data Governance Officers, Data Stewards, Database Administrators (DBAs), Big Data Engineers, Big Data Analysts, Data Analysts, Business Intelligence (BI) Professionals, Data Scientists, Machine Learning Engineers, Cloud Data Engineers, Cloud Architects, DevOps Engineers, Enterprise Architects, Solution Architects, Data Quality Professionals, Metadata and Data Catalog Managers, IT Managers, ICT Managers, Chief Data Officers (CDOs), Digital Transformation Managers, Analytics Managers, IT Project Managers, Information Security Professionals, Data Security Specialists, Compliance and Risk Professionals, Software Developers working with data platforms, ETL/ELT Developers, Database Developers, Business Analysts, Research and Monitoring & Evaluation Professionals, Government Data Managers, NGO and Development-sector Data Professionals, Consultants, Technology Managers, and professionals responsible for designing, implementing, managing, securing or governing enterprise data platforms.
Course content
1. Foundations of Modern Data Lakes and Data Platforms: Data lake concepts and evolution, data lakes versus data warehouses and lakehouses, structured semi-structured and unstructured data, enterprise data lake use cases, data platform components, modern
2. Enterprise Data Lake Architecture and Design: Data lake architecture components, landing and ingestion zones, raw data layers, Bronze Silver and Gold architecture, medallion architecture, data domains, centralized and federated architectures, data prod
3. Data Ingestion and Modern Data Integration: Batch ingestion, real-time and streaming ingestion, API-based ingestion, database ingestion, file-based ingestion, Change Data Capture, incremental loading, event-driven ingestion, data replication, schema ha
4. Cloud Data Lake Storage and Data Organisation: Cloud object storage, Amazon S3, Azure Data Lake Storage, Google Cloud Storage, data containers and storage zones, CSV and JSON, Avro and Parquet, columnar storage, compression, partitioning, file organisa
5. Data Lakehouse Architecture and Open Table Formats: Data lakehouse concepts, lakehouse architecture, Delta Lake, Apache Iceberg, Apache Hudi, ACID transactions, schema enforcement, schema evolution, time travel, table versioning, table optimisation, in
6. Metadata Management Data Cataloguing and Data Discovery: Technical metadata, business metadata, operational metadata, metadata repositories, data catalogues, data classification, business glossaries, dataset ownership, data stewardship, data discovery,
7. Data Quality Validation and Data Observability: Data profiling, completeness, accuracy, consistency and validity, schema validation, duplicate detection, missing-data management, data quality rules, data quality thresholds, anomaly detection, data fres
10. Data Pipelines Orchestration and Automation: ETL and ELT pipelines, workflow orchestration, pipeline scheduling, dependency management, parameterised pipelines, reusable workflows, event-driven workflows, retries and recovery, pipeline monitoring, aut
11. Data Lake Performance and Cost Optimisation: Partitioning strategies, file-size optimisation, compaction, clustering, query optimisation, indexing approaches, caching, materialised datasets, compute and storage separation, workload management, storage
12. Data Lake Integration with Business Intelligence Analytics and Reporting: Analytics-ready datasets, curated data layers, semantic models, SQL analytics, Power BI integration, business intelligence workflows, self-service analytics, dashboards, reporti
13. Data Lakes for Artificial Intelligence and Machine Learning: AI-ready data architecture, machine learning datasets, feature engineering, feature stores, training and validation datasets, ML data pipelines, model-data lineage, MLOps foundations, genera
14. Cloud Data Lake Platforms and Modern Data Ecosystems: Microsoft Fabric and OneLake, Azure Data Lake Storage, Amazon S3 data lake architectures, Google Cloud data lake concepts, Databricks lakehouse, cloud data integration, cloud-native processing, clo
15. Enterprise Data Lake Operations Monitoring and Implementation: Data lake administration, platform monitoring, pipeline monitoring, logging and auditing, capacity management, backup and disaster recovery, business continuity, data lifecycle management,
11. Data Lake Performance and Cost Optimisation: Partitioning strategies, file-size optimisation, compaction, clustering, query optimisation, indexing approaches, caching, materialised datasets, compute and storage separation, workload management, storage
12. Data Lake Integration with Business Intelligence Analytics and Reporting: Analytics-ready datasets, curated data layers, semantic models, SQL analytics, Power BI integration, business intelligence workflows, self-service analytics, dashboards, reporti
13. Data Lakes for Artificial Intelligence and Machine Learning: AI-ready data architecture, machine learning datasets, feature engineering, feature stores, training and validation datasets, ML data pipelines, model-data lineage, MLOps foundations, genera
14. Cloud Data Lake Platforms and Modern Data Ecosystems: Microsoft Fabric and OneLake, Azure Data Lake Storage, Amazon S3 data lake architectures, Google Cloud data lake concepts, Databricks lakehouse, cloud data integration, cloud-native processing, clo
15. Enterprise Data Lake Operations Monitoring and Implementation: Data lake administration, platform monitoring, pipeline monitoring, logging and auditing, capacity management, backup and disaster recovery, business continuity, data lifecycle management,
General notes
Key Technologies Participants Should Gain Exposure To Data Lake Architecture, Cloud Object Storage, Microsoft Fabric, OneLake, Databricks, Apache Spark, PySpark, SQL, Delta Lake, Apache Iceberg, Apache Parquet, ETL/ELT, Data Pipelines, Data Factory, Apache Kafka, Data Governance, Data Cataloguing, Data Lineage, Data Quality, Data Security, Data Observability, Power BI, Machine Learning, MLflow, Lakehouse Architecture, Medallion Architecture, Cloud Data Engineering
Core Hands-On Stack For practical delivery, the strongest combination would be Microsoft Fabric + OneLake + Data Factory + Spark/PySpark + SQL + Delta Lake + Power BI, supplemented with Databricks, Apache Iceberg, Kafka, Microsoft Purview and MLflow for advanced and cross-platform scenarios. Fabric currently provides integrated Data Engineering, Data Factory, Data Science, Real-Time Intelligence, Data Warehouse and Power BI capabilities around OneLake, making it particularly suitable for demonstrating an end-to-end modern data platform.
Alternative Enterprise Stack AWS: Amazon S3, AWS Glue, Amazon Kinesis, Amazon Athena, Amazon Redshift, AWS Lake Formation, Amazon EMR, Amazon SageMaker Azure: Azure Data Lake Storage Gen2, Azure Data Factory, Azure Databricks, Microsoft Fabric, Microsoft Purview, Azure Event Hubs, Power BI
Frequently asked questions
The course is suitable for data engineers, data architects, data scientists, database administrators, data analysts, BI professionals, cloud professionals, data governance officers, IT managers, enterprise architects, technology managers and professionals involved in designing, managing, securing or governing enterprise data platforms.
Participants will learn how to design and manage modern data lakes and lakehouse platforms, covering data ingestion, cloud storage, data processing, metadata and cataloguing, data quality, lineage, governance, security, data pipelines, analytics, AI/ML data management, performance optimisation and operational management.
Yes. The training provides exposure to technologies and architectures including Microsoft Fabric and OneLake, Azure Data Lake Storage, Databricks, Apache Spark, PySpark, Delta Lake, Apache Iceberg, Amazon S3, cloud data pipelines and modern lakehouse architectures. The exact technology mix can be adapted to organisational requirements and participant experience.
Yes. The course demonstrates how properly governed data lakes and lakehouses can provide foundations for business intelligence, advanced analytics, machine learning and AI applications. Participants also explore analytics-ready datasets, ML data pipelines, feature engineering, generative AI data requirements and Retrieval-Augmented Generation (RAG) data foundations.