[{"data":1,"prerenderedAt":454},["ShallowReactive",2],{"navigation":3,"\u002Fprojects\u002Fgen-adf-pipeline":50,"\u002Fprojects\u002Fgen-adf-pipeline-meta":436,"\u002Fprojects\u002Fgen-adf-pipeline-surround":449},[4],{"title":5,"path":6,"stem":7,"children":8,"page":49},"Blog","\u002Fblog","blog",[9,13,17,21,25,29,33,37,41,45],{"title":10,"path":11,"stem":12},"Data Catalog 3.0: Rise of the Active Metadata Platform","\u002Fblog\u002Fdata-catalog-3","blog\u002Fdata-catalog-3",{"title":14,"path":15,"stem":16},"dbt on Databricks: Data Transformation Pipelines","\u002Fblog\u002Fdbt-on-databricks","blog\u002Fdbt-on-databricks",{"title":18,"path":19,"stem":20},"Deploying Azure Resources with VS Code","\u002Fblog\u002Fdeploying-azure-resource-with-vs-code","blog\u002Fdeploying-azure-resource-with-vs-code",{"title":22,"path":23,"stem":24},"Dynamic Management Views (DMVs)","\u002Fblog\u002Fdynamic-management-views","blog\u002Fdynamic-management-views",{"title":26,"path":27,"stem":28},"General Delta Table Processing","\u002Fblog\u002Fgeneral-delta-table","blog\u002Fgeneral-delta-table",{"title":30,"path":31,"stem":32},"Microsoft Fabric as an All-in-One Analytics Solution","\u002Fblog\u002Fmicrosoft-fabric","blog\u002Fmicrosoft-fabric",{"title":34,"path":35,"stem":36},"Orchard Core Shapes: Demystifying the View Data Model","\u002Fblog\u002Forchard-core-shapes","blog\u002Forchard-core-shapes",{"title":38,"path":39,"stem":40},"Power BI DAX Masterclass","\u002Fblog\u002Fpower-bi-dax-masterclass","blog\u002Fpower-bi-dax-masterclass",{"title":42,"path":43,"stem":44},"Power BI Incremental Refresh","\u002Fblog\u002Fpower-bi-incremental-refresh","blog\u002Fpower-bi-incremental-refresh",{"title":46,"path":47,"stem":48},"Set Power BI Row-Level Security to SAP Cost Center","\u002Fblog\u002Fpower-bi-row-level-security","blog\u002Fpower-bi-row-level-security",false,{"id":51,"title":52,"body":53,"date":427,"description":428,"extension":429,"image":427,"meta":430,"navigation":431,"path":432,"seo":433,"stem":434,"__hash__":435},"projectsContent\u002Fprojects\u002Fgen-adf-pipeline.md","Generalised ADF Pipeline Architecture",{"type":54,"value":55,"toc":404},"minimark",[56,60,63,68,71,75,116,120,125,128,132,135,173,176,180,183,187,190,208,211,215,218,238,241,245,248,274,278,281,313,317,320,324,327,331,334,338,341,345,389,393],[57,58,59],"p",{},"Architected and delivered a generalised, metadata-driven data workflow built on Azure Data Factory (ADF), designed to support data ingestion, transformation, quality testing, auditing, and error handling across multiple source systems and target warehouses. The framework is composed of a set of independent, single-purpose pipelines that are orchestrated by two top-level processing pipelines — one at system level and one at object level — driven entirely by configuration held in the Metadata database.",[57,61,62],{},"The architecture eliminates the need to build or modify pipelines when new data sources, objects, or processing parameters are introduced. Reusable Copy pipelines handle ingestion, data quality, and schema evolution scenarios, while custom pipelines are available as drop-in replacements for non-standard cases. The result is a scalable, maintainable platform that supports multiple source systems and both Snowflake and Databricks target warehouses from a single shared codebase.",[64,65,67],"h2",{"id":66},"overall-architecture","Overall Architecture",[57,69,70],{},"The workflow is structured as a hierarchy of independent, composable ADF pipelines. Two orchestration pipelines sit at the top of the hierarchy — one for system-level processing and one for object-level processing. Beneath them, a set of specialised pipelines each handle a single concern: ingestion, raw processing, quality testing, auditing, or error handling. This layered design means each pipeline can be developed, tested, versioned, and maintained independently.",[64,72,74],{"id":73},"technologies-stack","Technologies & Stack",[76,77,78,86,92,98,104,110],"ul",{},[79,80,81,85],"li",{},[82,83,84],"strong",{},"Azure Data Factory (ADF)"," — orchestration engine; all pipelines deployed and managed in ADF",[79,87,88,91],{},[82,89,90],{},"Metadata Database"," — single source of truth for system config, object definitions, pipeline version assignments, and audit records",[79,93,94,97],{},[82,95,96],{},"Snowflake"," — one of two target warehouse platforms with a dedicated raw processing pipeline",[79,99,100,103],{},[82,101,102],{},"Databricks \u002F Delta Lake"," — second target warehouse platform; raw Parquet processed via the Databricks pipeline",[79,105,106,109],{},[82,107,108],{},"Azure Data Lake Storage"," — staging area for raw Parquet files between ingestion and processing",[79,111,112,115],{},[82,113,114],{},"Metadata UI"," — user interface for registering systems, objects, test templates, and pipeline version assignments",[64,117,119],{"id":118},"orchestration-layer","Orchestration Layer",[121,122,124],"h3",{"id":123},"systemprocessing-pipeline","SystemProcessing Pipeline",[57,126,127],{},"The SystemProcessing pipeline is the outermost entry point for the workflow. It queries the Metadata database for all active systems, then iterates over them and invokes the ObjectProcessing pipeline for each. System-level configuration — such as source connection parameters, schedule settings, and global flags — is resolved here and passed downstream as pipeline parameters, keeping individual pipelines stateless with respect to system context.",[121,129,131],{"id":130},"objectprocessing-pipeline","ObjectProcessing Pipeline",[57,133,134],{},"The ObjectProcessing pipeline is the core orchestrator for a single system's data objects. For each active object it:",[76,136,137,143,149,155,161,167],{},[79,138,139,142],{},[82,140,141],{},"Loads object metadata"," — schema, processing mode, pipeline version, test templates — from the Metadata DB",[79,144,145,148],{},[82,146,147],{},"Resolves which ingestion pipeline to invoke"," — general or custom — based on the object's metadata registration",[79,150,151,154],{},[82,152,153],{},"Calls the appropriate Copy pipeline"," to ingest source data into the staging data lake",[79,156,157,160],{},[82,158,159],{},"Triggers the correct delta processing pipeline"," (Snowflake or Databricks) based on the object's target warehouse",[79,162,163,166],{},[82,164,165],{},"Invokes data quality testing"," if test templates are configured for the object",[79,168,169,172],{},[82,170,171],{},"Calls the AuditLogging pipeline"," on success, or the ErrorHandling pipeline on failure",[57,174,175],{},"Because all pipeline selection logic is driven by metadata object properties, adding a new object to the workflow requires only a metadata record — no pipeline modifications.",[64,177,179],{"id":178},"ingestion-layer-general-custom-copy-pipelines","Ingestion Layer — General & Custom Copy Pipelines",[57,181,182],{},"Data ingestion is handled by Copy Activity pipelines that read from source systems and write raw Parquet files to the data lake staging area. Two variants exist — a general pipeline covering the majority of objects, and custom pipelines for edge cases — both adhering to the same output contract so downstream processing pipelines are unaffected by the choice of ingestion variant.",[64,184,186],{"id":185},"delta-processing-layer-snowflake-databricks","Delta Processing Layer — Snowflake & Databricks",[57,188,189],{},"Two independent Delta processing pipelines consume the staged Parquet files and load data into the respective target warehouse. Both pipelines are invoked with the same interface (object metadata + staging path) and support the same set of processing modes — full load, incremental load, and CDC merge — ensuring consistent behaviour regardless of the target platform.",[76,191,192,198],{},[79,193,194,197],{},[82,195,196],{},"DeltaProcessingSnowflake"," — reads staged Parquet via ADF and executes the appropriate load strategy against Snowflake tables using SQL COPY INTO, INSERT, or MERGE commands",[79,199,200,203,204,207],{},[82,201,202],{},"DeltaProcessingDatabricks"," — delegates to the ",[82,205,206],{},"Delta Processing Framework"," via a Databricks notebook activity; handles full load, incremental, and CDC processing into Delta Lake tables",[57,209,210],{},"Separating the two processing pipelines ensures that Snowflake-specific and Databricks-specific implementation details are fully isolated. Changes to one warehouse's loading behaviour have zero impact on the other.",[64,212,214],{"id":213},"copy-pipeline-reuse-across-concerns","Copy Pipeline Reuse Across Concerns",[57,216,217],{},"A deliberate architectural choice was to reuse the Copy Activity pipeline pattern across three distinct workflow concerns rather than building bespoke pipelines for each:",[76,219,220,226,232],{},[79,221,222,225],{},[82,223,224],{},"Data Ingestion"," — CopyIngestion loads source data into the staging data lake",[79,227,228,231],{},[82,229,230],{},"Data Quality"," — CopyQuality loads a source snapshot for quality test comparison",[79,233,234,237],{},[82,235,236],{},"Schema Evolution"," — CopySchemaEvolution loads a source sample to detect schema drift against the registered object definition",[57,239,240],{},"This reuse means the same tested, monitored pipeline infrastructure serves multiple purposes. Operational knowledge, alerting rules, and performance baselines apply uniformly, reducing the overall surface area of the platform.",[64,242,244],{"id":243},"metadata-driven-pipeline-selection","Metadata-Driven Pipeline Selection",[57,246,247],{},"All pipeline behaviour is governed by object and system records in the Metadata database. Pipeline parameters are passed as ADF pipeline objects — structured metadata payloads — rather than individual scalar parameters. This design means:",[76,249,250,256,262,268],{},[79,251,252,255],{},[82,253,254],{},"New processing parameters"," can be added to the metadata schema and consumed by pipelines without modifying pipeline definitions or adding new parameters to the ADF pipeline interface",[79,257,258,261],{},[82,259,260],{},"Pipeline selection"," (general vs. custom ingestion, Snowflake vs. Databricks processing) is resolved at runtime by reading the object's metadata record",[79,263,264,267],{},[82,265,266],{},"Version assignments"," are stored per object in the Metadata DB, allowing each object to be mapped to a specific pipeline version independently",[79,269,270,273],{},[82,271,272],{},"Zero-touch onboarding"," — adding a new data source or object to the platform requires only a metadata registration, not a pipeline deployment",[64,275,277],{"id":276},"single-purpose-pipeline-design","Single-Purpose Pipeline Design",[57,279,280],{},"Each pipeline in the framework is designed to do exactly one thing. Ingestion pipelines ingest. Processing pipelines process. Audit pipelines audit. This strict single-responsibility approach delivers practical advantages across the entire development lifecycle:",[76,282,283,289,295,301,307],{},[79,284,285,288],{},[82,286,287],{},"Testability"," — each pipeline can be unit-tested and integration-tested in isolation, with well-defined inputs and outputs and no implicit dependencies on sibling pipelines",[79,290,291,294],{},[82,292,293],{},"Maintainability"," — a bug or change in the error handling pipeline has no risk of affecting the ingestion or processing pipelines",[79,296,297,300],{},[82,298,299],{},"Reusability"," — the same Copy pipeline is invoked for ingestion, quality, and schema evolution without code duplication",[79,302,303,306],{},[82,304,305],{},"Clarity"," — developers and operators can identify the purpose of any pipeline immediately from its name and scope",[79,308,309,312],{},[82,310,311],{},"Independent deployment"," — a new version of one pipeline can be deployed and tested without touching the rest of the framework",[64,314,316],{"id":315},"auditing-error-handling","Auditing & Error Handling",[57,318,319],{},"Dedicated pipelines handle auditing and error management as first-class concerns rather than afterthoughts embedded in processing logic.",[121,321,323],{"id":322},"auditlogging-pipeline","AuditLogging Pipeline",[57,325,326],{},"Called by ObjectProcessing on every successful pipeline run, the AuditLogging pipeline writes a structured audit record to the Metadata database containing the object run ID, pipeline name and version, execution timestamps, row counts, processing mode, and source\u002Ftarget details. This creates a complete, queryable history of every data movement event across the platform.",[121,328,330],{"id":329},"errorhandling-pipeline","ErrorHandling Pipeline",[57,332,333],{},"On any pipeline failure, the ObjectProcessing pipeline's error branch invokes the ErrorHandling pipeline. It captures structured error details — pipeline name, activity, error code, message, and object run ID — and writes them to the Metadata database. Configurable per-object actions include alert notification, retry scheduling, and run status flagging. Because error handling is a separate pipeline, its logic can be updated independently and applied consistently across all failure scenarios in the framework.",[64,335,337],{"id":336},"pipeline-versioning-backward-compatibility","Pipeline Versioning & Backward Compatibility",[57,339,340],{},"Larger changes to pipeline logic are managed through explicit versioning rather than in-place modification, ensuring that existing objects are never unexpectedly affected by updates. The versioning strategy is designed for gradual, controlled migration.",[64,342,344],{"id":343},"design-principles-benefits","Design Principles & Benefits",[76,346,347,353,359,365,371,377,383],{},[79,348,349,352],{},[82,350,351],{},"Metadata-driven:"," all pipeline behaviour, selection, and versioning is governed by the Metadata DB — no pipeline changes are required to onboard new objects or parameters",[79,354,355,358],{},[82,356,357],{},"Single-purpose pipelines:"," each pipeline has one clear responsibility, making the entire framework easier to test, maintain, and extend",[79,360,361,364],{},[82,362,363],{},"Infrastructure reuse:"," Copy pipelines serve ingestion, data quality, and schema evolution from the same codebase",[79,366,367,370],{},[82,368,369],{},"Multi-platform support:"," Snowflake and Databricks are supported through dedicated but interface-compatible processing pipelines",[79,372,373,376],{},[82,374,375],{},"Full auditability:"," every successful run and every failure is recorded in the Metadata DB with a unique object run ID",[79,378,379,382],{},[82,380,381],{},"Controlled evolution:"," versioned pipelines with backward compatibility allow the framework to evolve without disrupting live objects",[79,384,385,388],{},[82,386,387],{},"Extensibility:"," custom ingestion pipelines and custom test SQL provide escape hatches for non-standard cases without coupling them to the core framework",[64,390,392],{"id":391},"outcome-impact","Outcome & Impact",[57,394,395,396,399,400,403],{},"The generalised ADF workflow became the standard data movement platform across all registered source systems and both warehouse targets. Onboarding a new data source was reduced from a pipeline development effort to a metadata configuration task. The single-purpose pipeline design significantly reduced the time required to diagnose and fix issues in production, and the versioning strategy allowed the framework to evolve iteratively without service disruption. The Copy pipeline reuse pattern, shared with the ",[82,397,398],{},"Automated Testing Framework"," and ",[82,401,402],{},"Delta Lake Table Processing Framework",", further reduced the total number of pipelines under management while increasing coverage.",{"title":405,"searchDepth":406,"depth":406,"links":407},"",2,[408,409,410,415,416,417,418,419,420,424,425,426],{"id":66,"depth":406,"text":67},{"id":73,"depth":406,"text":74},{"id":118,"depth":406,"text":119,"children":411},[412,414],{"id":123,"depth":413,"text":124},3,{"id":130,"depth":413,"text":131},{"id":178,"depth":406,"text":179},{"id":185,"depth":406,"text":186},{"id":213,"depth":406,"text":214},{"id":243,"depth":406,"text":244},{"id":276,"depth":406,"text":277},{"id":315,"depth":406,"text":316,"children":421},[422,423],{"id":322,"depth":413,"text":323},{"id":329,"depth":413,"text":330},{"id":336,"depth":406,"text":337},{"id":343,"depth":406,"text":344},{"id":391,"depth":406,"text":392},null,"Architected a metadata-driven data workflow on Azure Data Factory that eliminated per-source pipeline development. A hierarchy of single-purpose pipelines — orchestrated by system- and object-level top-level pipelines — supports ingestion, delta processing (Snowflake & Databricks), data quality testing, auditing, and error handling. Adding a new data source requires only a metadata record, no pipeline changes.","md",{},true,"\u002Fprojects\u002Fgen-adf-pipeline",{"title":52,"description":428},"projects\u002Fgen-adf-pipeline","f1S67yPNZduXHwGMwQvu3_B9dBfIYTTDFeFc6Iblaz0",{"id":437,"title":52,"date":438,"description":428,"extension":439,"image":440,"meta":441,"stem":434,"tags":442,"url":447,"__hash__":448},"projects\u002Fprojects\u002Fgen-adf-pipeline.yml","2023-01-01","yml","blog\u002Fgeneralized-adf.png",{},[443,96,444,445,446],"Azure Data Factory","Databricks","Metadata-Driven","Azure","#","6IZahWNUcqsrkUnywVGX1zrWzq_UUekN7QEn1YkmOLQ",[450,427],{"title":402,"path":451,"stem":452,"description":453,"children":-1},"\u002Fprojects\u002Fdlt-processing","projects\u002Fdlt-processing","Designed and implemented a metadata-driven Delta Lake processing framework on Databricks using a modular OOP Python architecture. Four core classes — MetadataClass, FileLoaderClass, DeltaProcessorClass, and ProcessingClass — handle all pipeline concerns independently. Supports full load, incremental load, and CDC merge. Onboarding a new data object requires only a metadata entry, not code changes.",1783165025638]