[{"data":1,"prerenderedAt":296},["ShallowReactive",2],{"navigation":3,"\u002Fprojects\u002Fdlt-processing":50,"\u002Fprojects\u002Fdlt-processing-meta":271,"\u002Fprojects\u002Fdlt-processing-surround":285},[4],{"title":5,"path":6,"stem":7,"children":8,"page":49},"Blog","\u002Fblog","blog",[9,13,17,21,25,29,33,37,41,45],{"title":10,"path":11,"stem":12},"Data Catalog 3.0: Rise of the Active Metadata Platform","\u002Fblog\u002Fdata-catalog-3","blog\u002Fdata-catalog-3",{"title":14,"path":15,"stem":16},"dbt on Databricks: Data Transformation Pipelines","\u002Fblog\u002Fdbt-on-databricks","blog\u002Fdbt-on-databricks",{"title":18,"path":19,"stem":20},"Deploying Azure Resources with VS Code","\u002Fblog\u002Fdeploying-azure-resource-with-vs-code","blog\u002Fdeploying-azure-resource-with-vs-code",{"title":22,"path":23,"stem":24},"Dynamic Management Views (DMVs)","\u002Fblog\u002Fdynamic-management-views","blog\u002Fdynamic-management-views",{"title":26,"path":27,"stem":28},"General Delta Table Processing","\u002Fblog\u002Fgeneral-delta-table","blog\u002Fgeneral-delta-table",{"title":30,"path":31,"stem":32},"Microsoft Fabric as an All-in-One Analytics Solution","\u002Fblog\u002Fmicrosoft-fabric","blog\u002Fmicrosoft-fabric",{"title":34,"path":35,"stem":36},"Orchard Core Shapes: Demystifying the View Data Model","\u002Fblog\u002Forchard-core-shapes","blog\u002Forchard-core-shapes",{"title":38,"path":39,"stem":40},"Power BI DAX Masterclass","\u002Fblog\u002Fpower-bi-dax-masterclass","blog\u002Fpower-bi-dax-masterclass",{"title":42,"path":43,"stem":44},"Power BI Incremental Refresh","\u002Fblog\u002Fpower-bi-incremental-refresh","blog\u002Fpower-bi-incremental-refresh",{"title":46,"path":47,"stem":48},"Set Power BI Row-Level Security to SAP Cost Center","\u002Fblog\u002Fpower-bi-row-level-security","blog\u002Fpower-bi-row-level-security",false,{"id":51,"title":52,"body":53,"date":262,"description":263,"extension":264,"image":262,"meta":265,"navigation":266,"path":267,"seo":268,"stem":269,"__hash__":270},"projectsContent\u002Fprojects\u002Fdlt-processing.md","Delta Lake Table Processing Framework",{"type":54,"value":55,"toc":248},"minimark",[56,60,65,100,104,107,112,115,126,129,133,136,147,150,154,157,171,174,178,181,202,205,209,241,245],[57,58,59],"p",{},"Designed and implemented a metadata-driven Delta Lake processing framework on Databricks, enabling automated ingestion of Parquet files from a data lake into Delta tables. The framework is built around a modular, object-oriented Python architecture where each class has a clearly defined responsibility, and the entire pipeline is governed by configuration stored in a centralised Metadata database.",[61,62,64],"h2",{"id":63},"technologies-stack","Technologies & Stack",[66,67,68,76,82,88,94],"ul",{},[69,70,71,75],"li",{},[72,73,74],"strong",{},"Platform:"," Apache Spark \u002F Databricks (PySpark)",[69,77,78,81],{},[72,79,80],{},"Storage Layer:"," Delta Lake (Delta format tables, ACID transactions)",[69,83,84,87],{},[72,85,86],{},"Language:"," Python (OOP, class inheritance, polymorphism)",[69,89,90,93],{},[72,91,92],{},"Metadata Store:"," Relational Metadata database (system config, object registry, audit logging)",[69,95,96,99],{},[72,97,98],{},"Source:"," Data Lake storage (Parquet files as ingestion source)",[61,101,103],{"id":102},"architecture-class-design","Architecture & Class Design",[57,105,106],{},"The framework follows the Single Responsibility Principle, with four core Python classes working in concert. Each class encapsulates a distinct concern of the data pipeline, and all are composed together within the top-level Processing class. The design enables straightforward customisation by inheriting from the base Processing class and overriding specific processing methods.",[108,109,111],"h3",{"id":110},"metadataclass-metadata-audit","MetadataClass — Metadata & Audit",[57,113,114],{},"The MetadataClass is responsible for all communication with the Metadata database. On initialisation it queries three categories of information:",[66,116,117,120,123],{},[69,118,119],{},"System-level configuration: environment settings, connection parameters, storage paths",[69,121,122],{},"Object-level metadata: table definitions, schema expectations, processing rules per entity",[69,124,125],{},"File-level metadata: newly ingested Parquet files awaiting processing, including location and load timestamps",[57,127,128],{},"During pipeline execution, this class is also used by other components to write audit records back to the Metadata database, capturing processing status, row counts, error messages, and timestamps. This creates a full audit trail for every ingestion run.",[108,130,132],{"id":131},"fileloaderclass-data-ingestion","FileLoaderClass — Data Ingestion",[57,134,135],{},"The FileLoaderClass reads the Parquet files identified by the MetadataClass into a Spark DataFrame. It abstracts file access from the rest of the pipeline, handling:",[66,137,138,141,144],{},[69,139,140],{},"Locating the correct file path from metadata-provided information",[69,142,143],{},"Reading Parquet data with appropriate Spark read options and schema enforcement",[69,145,146],{},"Exposing the loaded DataFrame to downstream processing components",[57,148,149],{},"This separation ensures that changes to the source file format or storage layout require only isolated changes in this class, without affecting processing logic.",[108,151,153],{"id":152},"deltaprocessorclass-delta-lake-writer","DeltaProcessorClass — Delta Lake Writer",[57,155,156],{},"The DeltaProcessorClass contains the core data processing logic. It selects and executes the appropriate write strategy for each object based on metadata configuration. Supported processing modes include:",[66,158,159,162,165,168],{},[69,160,161],{},"Full Load — truncates and reloads the target Delta table entirely from the source file",[69,163,164],{},"Incremental Load — appends only new records based on a watermark or partition key",[69,166,167],{},"CDC (Change Data Capture) — merges changes (inserts, updates, deletes) using Delta Lake MERGE INTO operations",[69,169,170],{},"Custom modes — additional strategies can be added by extending the class",[57,172,173],{},"All processing outcomes, including affected row counts and any exceptions, are reported back to the Metadata database via the MetadataClass upon completion.",[108,175,177],{"id":176},"processingclass-orchestrator-extension-point","ProcessingClass — Orchestrator & Extension Point",[57,179,180],{},"The ProcessingClass serves as the top-level orchestrator and the single entry point for pipeline execution. It holds instances of all other classes as member attributes:",[66,182,183,190,196],{},[69,184,185,189],{},[186,187,188],"code",{},"self.metadata"," — MetadataClass instance for configuration and audit logging",[69,191,192,195],{},[186,193,194],{},"self.loader"," — FileLoaderClass instance for data ingestion",[69,197,198,201],{},[186,199,200],{},"self.processor"," — DeltaProcessorClass instance for Delta Lake writes",[57,203,204],{},"This composition pattern means the full pipeline is encapsulated in one object, making it trivial to invoke, test, and extend. Custom processing scenarios are handled by subclassing ProcessingClass and overriding specific methods (e.g., pre-processing hooks, custom transformation logic, or alternative write strategies) without modifying the base framework.",[61,206,208],{"id":207},"design-principles-benefits","Design Principles & Benefits",[66,210,211,217,223,229,235],{},[69,212,213,216],{},[72,214,215],{},"Metadata-driven:"," no hardcoded configuration; all pipeline behaviour is controlled by the Metadata database, enabling onboarding of new objects without code changes",[69,218,219,222],{},[72,220,221],{},"Separation of concerns:"," each class has one clear responsibility, reducing coupling and simplifying unit testing",[69,224,225,228],{},[72,226,227],{},"Extensibility:"," base class inheritance allows new processing types to be added cleanly",[69,230,231,234],{},[72,232,233],{},"Full auditability:"," every pipeline run produces a complete audit trail in the Metadata database",[69,236,237,240],{},[72,238,239],{},"Consistency:"," a unified entry point and logging pattern across all processed objects simplifies monitoring and operations",[61,242,244],{"id":243},"outcome-impact","Outcome & Impact",[57,246,247],{},"The framework enabled standardised, repeatable Delta Lake ingestion across multiple data objects, reducing the effort to onboard new data sources from days of custom development to a metadata configuration task. The extensible class design has been reused across subsequent projects as the base processing pattern for all Databricks Delta table pipelines.",{"title":249,"searchDepth":250,"depth":250,"links":251},"",2,[252,253,260,261],{"id":63,"depth":250,"text":64},{"id":102,"depth":250,"text":103,"children":254},[255,257,258,259],{"id":110,"depth":256,"text":111},3,{"id":131,"depth":256,"text":132},{"id":152,"depth":256,"text":153},{"id":176,"depth":256,"text":177},{"id":207,"depth":250,"text":208},{"id":243,"depth":250,"text":244},null,"Designed and implemented a metadata-driven Delta Lake processing framework on Databricks using a modular OOP Python architecture. Four core classes — MetadataClass, FileLoaderClass, DeltaProcessorClass, and ProcessingClass — handle all pipeline concerns independently. Supports full load, incremental load, and CDC merge. Onboarding a new data object requires only a metadata entry, not code changes.","md",{},true,"\u002Fprojects\u002Fdlt-processing",{"title":52,"description":263},"projects\u002Fdlt-processing","rTnUxnSU5v8-ihnF3G3epjw2jtfsf_RwrjCOtsSh_gU",{"id":272,"title":52,"date":273,"description":263,"extension":274,"image":275,"meta":276,"stem":269,"tags":277,"url":283,"__hash__":284},"projects\u002Fprojects\u002Fdlt-processing.yml","2023-01-01","yml","\u002Fblog\u002Fdelta-processing.png",{},[278,279,280,281,282],"Databricks","Delta Lake","PySpark","Python","Unity Catalog","#","MBnBetollUqHKW4fn7Xzk58-uySPZVtq8i6EthcqspA",[286,291],{"title":287,"path":288,"stem":289,"description":290,"children":-1},"Data Governance & Script Generation Framework","\u002Fprojects\u002Fdata-governance","projects\u002Fdata-governance","Designed and delivered an end-to-end framework for generating and executing Snowflake and Databricks management scripts from a centralised template repository. Implemented as stored procedures (Snowflake) and an OOP TemplateGenerator class (Databricks\u002FPython). Uses owner's-rights execution and Service Principal privilege escalation so no human user holds GRANT or CREATE USER rights on any platform. A self-documenting help system is built directly into the interface.",{"title":292,"path":293,"stem":294,"description":295,"children":-1},"Generalised ADF Pipeline Architecture","\u002Fprojects\u002Fgen-adf-pipeline","projects\u002Fgen-adf-pipeline","Architected a metadata-driven data workflow on Azure Data Factory that eliminated per-source pipeline development. A hierarchy of single-purpose pipelines — orchestrated by system- and object-level top-level pipelines — supports ingestion, delta processing (Snowflake & Databricks), data quality testing, auditing, and error handling. Adding a new data source requires only a metadata record, no pipeline changes.",1783165025598]