Engineering Platforms, Not Just Pipelines.
Key projects from my work as a Senior Lead Data Engineer and Data Platform Architect. Each one built around generalisation, reusability, and long-term operability on Microsoft Azure.
2023
Generalised ADF Pipeline Architecture
Architected a metadata-driven data workflow on Azure Data Factory that eliminated per-source pipeline development. A hierarchy of single-purpose pipelines — orchestrated by system- and object-level top-level pipelines — supports ingestion, delta processing (Snowflake & Databricks), data quality testing, auditing, and error handling. Adding a new data source requires only a metadata record, no pipeline changes.
2023
Delta Lake Table Processing Framework
Designed and implemented a metadata-driven Delta Lake processing framework on Databricks using a modular OOP Python architecture. Four core classes — MetadataClass, FileLoaderClass, DeltaProcessorClass, and ProcessingClass — handle all pipeline concerns independently. Supports full load, incremental load, and CDC merge. Onboarding a new data object requires only a metadata entry, not code changes.
2024
Metadata-Driven Automated Testing Framework
Built a data quality testing module integrated directly into the ADF ingestion workflow. Reuses the same generalised Copy pipeline and Metadata database — no separate test infrastructure. A Databricks Python test class executes row count, nullability, freshness, uniqueness, referential, and custom SQL checks after every ingestion run. All results are recorded against the object run ID for full traceability. Tests are configured via a self-service Metadata UI without code deployment.
2024
Data Governance & Script Generation Framework
Designed and delivered an end-to-end framework for generating and executing Snowflake and Databricks management scripts from a centralised template repository. Implemented as stored procedures (Snowflake) and an OOP TemplateGenerator class (Databricks/Python). Uses owner's-rights execution and Service Principal privilege escalation so no human user holds GRANT or CREATE USER rights on any platform. A self-documenting help system is built directly into the interface.
