Azure Data Factory and Fabric Data Pipeline Development

Fabric Data Factory if you have or plan a Fabric capacity and want your data landing in OneLake for Power BI and AI use — it is where Microsoft's new capabilities such as Copy job and mirroring ship. Azure Data Factory if your target is Azure-native (Azure SQL, Synapse, Data Lake Storage) and you are not adopting Fabric, or you already run a factory that works. We recommend one, or a deliberate combination, in the design phase and write down why.

Not as far as Microsoft has said. At the time of writing there is no announced end date for Azure Data Factory; it is supported, and existing factories can be mounted into a Fabric workspace and run alongside Fabric pipelines. Microsoft's new investment is clearly going into Fabric Data Factory, which is why we recommend Fabric for new builds when you have a capacity — but nothing you own has to be rebuilt to start.

Not by default. Mounting your existing factory in Fabric lets you trigger, run and monitor those pipelines from the Fabric workspace while new work is built natively. A full conversion is a separate, inventoried piece of work that we quote when there is a reason for it — retiring the Azure subscription, say — rather than for its own sake.

Bronze holds data as it landed, silver holds it cleaned and conformed, gold holds it shaped for its consumers — a star schema for Power BI, or curated tables for an AI agent. It is a discipline, not a product: it keeps raw data replayable, puts every cleaning rule in one place, and stops reports and agents reading half-transformed tables. Small estates use a lighter version of it; we do not build three layers where one will do.

SQL Server and Azure SQL, Oracle and other relational databases, Dataverse and Dynamics 365, SharePoint and files, and SaaS systems through Microsoft's connectors — Salesforce, HubSpot and similar — plus REST APIs. Coverage differs between Azure Data Factory and Fabric and changes with Microsoft's releases, so we verify every source against your platform at design time and say plainly when one needs an export or a small custom step.

Yes. Fabric reaches on-premises sources through the on-premises data gateway; Azure Data Factory uses the self-hosted integration runtime. Either runs on a server inside your network that you provide or we help provision. For SQL Server specifically, mirroring can replicate tables into OneLake near real time through the gateway, which often removes the need for a nightly copy pipeline altogether.

Only where the source is small enough that it does not matter. Otherwise loads are incremental — watermarks on modified-date columns, change tracking or change data capture where the database supports it, or mirroring — so runs are short, capacity consumption is low, and a rerun after a failure is safe.

Mirroring keeps a near-real-time replica of a database in OneLake without a pipeline to maintain — Microsoft supports it for SQL Server, Azure SQL and a growing list of other sources. We use it when the source is supported, you want current data, and you do not need transformation on the way in; pipelines still handle the silver and gold layers on top. Where a source is not supported or you need shaping at ingest, Copy job or a pipeline does the work.

An alert reaches the owner you named, by Teams or email, with the run and the failing step. The runbook documents how to rerun — and because loads are built to be idempotent, a rerun does not duplicate data. Long-running runs alert too, so a stuck job is caught before the morning reports refresh against it.

Time and materials at $175 per hour, estimated in writing before work begins. Cost drivers are the number of sources, how well each is documented, whether history handling is needed, the transformation complexity in the silver and gold layers, and the environments you want. Fabric capacity or Azure consumption is billed by Microsoft to you, separately, and we instrument it from the first run so you know what it costs.

About four weeks for a handful of sources when access, capacity and data owners are in place from day one. Larger estates are delivered in sprints — usually source by source, so useful data lands early rather than everything arriving at the end.

For pipelines alone, the smallest paid capacities are often enough for a modest estate; what pushes the size up is heavy transformation, large volumes on tight schedules, and the Power BI consumers on the same capacity. Capacity is Microsoft's list price per capacity unit, pay-as-you-go or reserved. Sizing is properly a platform decision under our Fabric modernization service; for this build we make a minimal, documented recommendation and monitor consumption so you can right-size on real numbers.

It is the prerequisite. An agent grounded on a gold table that is refreshed, reconciled and governed gives better answers than one reading a spreadsheet someone exported last month. The gold layer is shaped with the downstream consumers in mind — a Power BI semantic model, a retrieval index, or a Fabric data agent — and our Microsoft Foundry and document-processing services build on exactly this plumbing.

Under a separate service — Power BI Dashboard and Report Development — usually scoped together with this one so the gold layer and the semantic model are designed as a pair. Keeping them separate on paper means you can see what the pipelines cost and what the reports cost.

You do. Everything is built in your Fabric workspace or Azure subscription, in a Git repository you own, with documentation that lets your team add a source or change a schedule without us. There is no proprietary layer and no obligation to keep using us.

Report building, Fabric platform and capacity design, Microsoft consumption and licensing, changes to source systems, data-cleansing programs, wholesale conversion of an existing Azure Data Factory estate, streaming and machine-learning work, and running the pipelines after handover. Each has a named home in our catalog and is cross-referenced rather than blurred into this scope.