Access & separation
Least-privilege roles, managed identities and environment separation ensure consumers never require direct source-system access.
Architecture blueprint 02
Hybrid Data Movement
A controlled route from on-premise operational systems into a governed Azure analytics platform.
- Model
- Logical hybrid data flow
- Scope
- On-premise / Azure / Power BI
- Status
- Illustrative target pattern
Designed outcome
Reliable insight without exposing source platforms or creating fragile point-to-point reporting connections.
Data architecture / hybrid flow
Move data without exposing the source estate.
A one-way analytical flow from operational systems into a governed Azure data product. Validation happens at every boundary; raw history makes each movement observable and replayable.
Read left to right
Operational estate: On-prem ERP, Transactions, product, customer and financial records.; Warehouse systems, Inventory, pick, pack, dispatch and carrier events.; SQL platforms, Line-of-business databases and controlled reporting extracts.. Secure extraction: Self-hosted gateway, Outbound-only managed bridge between local and cloud services.; CDC & scheduled loads, Incremental movement with explicit windows and source impact limits.; Schema validation, Expected shape, volume and freshness checked before transfer.. Integration boundary: API & integration gateway, Managed connections, policy, retries and workload isolation.; Encrypted private routes, TLS in transit with restricted endpoints and network rules.; Managed identity & keys, No embedded credentials; rotation and access are centrally governed.. Azure landing zone: Governed subscription, Resource policy, budgets, roles, diagnostics and private endpoints.; Data lake staging, Immutable source-shaped history partitioned by time and domain.; Quarantine & replay, Invalid loads are isolated, diagnosed and safely reprocessed.. Transform & model: Transformation pipelines, Dependency-led processing with idempotent execution and retries.; Data quality rules, Completeness, validity, uniqueness and freshness thresholds.; Curated data model, Owned entities, conformed measures and documented semantics.. Insight & action: Governed measures, Reusable definitions, row-level security and certified datasets.; Power BI Embedded, Contextual insight presented inside secure business workflows.; Self-service reporting, Curated discovery for approved analysts and decision makers.
01 / SOURCE Operational estate
Systems of record stay protected and continue to serve operations.
- Finance & ordersOn-prem ERP
Transactions, product, customer and financial records.
- FulfilmentWarehouse systems
Inventory, pick, pack, dispatch and carrier events.
- Operational dataSQL platforms
Line-of-business databases and controlled reporting extracts.
- Finance & ordersOn-prem ERP
02 / CAPTURE Secure extraction
Change is collected without opening source platforms to consumers.
- RuntimeSelf-hosted gateway
Outbound-only managed bridge between local and cloud services.
- PatternCDC & scheduled loads
Incremental movement with explicit windows and source impact limits.
- ContractSchema validation
Expected shape, volume and freshness checked before transfer.
- RuntimeSelf-hosted gateway
03 / TRANSFER Integration boundary
A governed route protects credentials, transport and contracts.
- Control pointAPI & integration gateway
Managed connections, policy, retries and workload isolation.
- TransportEncrypted private routes
TLS in transit with restricted endpoints and network rules.
- SecretsManaged identity & keys
No embedded credentials; rotation and access are centrally governed.
- Control pointAPI & integration gateway
04 / LAND Azure landing zone
Policy and subscription boundaries contain the data workload.
- FoundationGoverned subscription
Resource policy, budgets, roles, diagnostics and private endpoints.
- Raw zoneData lake staging
Immutable source-shaped history partitioned by time and domain.
- Exception pathQuarantine & replay
Invalid loads are isolated, diagnosed and safely reprocessed.
- FoundationGoverned subscription
05 / CONFORM Transform & model
Quality and business meaning become explicit and testable.
- OrchestrationTransformation pipelines
Dependency-led processing with idempotent execution and retries.
- TrustData quality rules
Completeness, validity, uniqueness and freshness thresholds.
- ProductCurated data model
Owned entities, conformed measures and documented semantics.
- OrchestrationTransformation pipelines
06 / CONSUME Insight & action
Users access trusted models rather than operational sources.
- Semantic layerGoverned measures
Reusable definitions, row-level security and certified datasets.
- ExperiencePower BI Embedded
Contextual insight presented inside secure business workflows.
- AccessSelf-service reporting
Curated discovery for approved analysts and decision makers.
- Semantic layerGoverned measures
Lineage & audit
Run identifiers, source-to-model lineage, schema history and access logs provide evidence for every data movement and decision product.
Observe & replay
Freshness, volume, quality and pipeline health are measured against service expectations, with failed loads recoverable from raw history.
The preferred path is incremental and outbound-only. Refresh frequency, retention, residency, classification and recovery requirements must be agreed per data domain.
Sources stay operational
Analytics never connects directly to ERP, WMS or operational SQL. Every extraction has a measured source-impact budget and approved window.
Movement is one-way
The integration bridge initiates outbound encrypted connections. Consumer access cannot become a return path into the operational estate.
Raw means recoverable
Source-shaped history, watermarks, manifests and checksums make a load replayable. Quality failure enters quarantine rather than trusted data.
Trust is measurable
Freshness, volume, validity, completeness, lineage and access are service measures with named owners and retained evidence.
01 / Source-to-insight path
Ten controlled hand-offs. No invisible leap.
Each stage has an accountable owner, an explicit exit condition and a recovery position. The service only becomes trustworthy when every hand-off can be observed and replayed.
Interactive control path
Select a stage to inspect its mechanism, decisions, controls, evidence and recovery stance.
Keyboard: arrow keys · Home · End
Stage 1 of 10On-premise operational trust zone
01 / Discover / On-premise operational trust zone
Source estate
ERP, WMS, line-of-business SQL and controlled file exports remain systems of record. Every feed starts with a named business owner, classification, extraction contract and source-impact budget.
- Freshness
- Per domain
- RTO
- 4–24h
- RPO
- 15m–24h
How this stage works
Read from supported APIs, reporting replicas, change tables or approved views. Inventory primary keys, change indicators, retention, peak windows and data sensitivity before a pipeline is designed.
Choose deliberately
- Prefer a reporting replica or vendor-supported API when production reads could affect transactions.
- Use a full snapshot only for small, slow-changing entities or a controlled initial load.
- Agree ownership for deletions, corrections, late records and source schema changes.
What protects the hand-off
- Source register and data classification
- Least-privilege read account
- Query timeout and concurrency limits
- Approved extraction window
What proves it
- Signed source contract
- Baseline row counts and timings
- Data classification record
Primary failure mode
An extraction query competes with operational workload or an undocumented source release changes the shape of the feed.
Contain & recover
Stop at the source boundary, preserve the last trusted watermark, compare the contract with the new schema, and resume from the last confirmed position after owner approval.
02 / Capture / On-premise data integration enclave
Secure extraction
A dedicated self-hosted integration runtime (SHIR) executes cloud-orchestrated copy work close to private sources without allowing inbound analytics access.
- Freshness
- ≤15m–24h
- RTO
- ≤4h
- RPO
- Last watermark
How this stage works
Run SHIR on hardened Windows hosts with outbound connectivity, monitored service health and sufficient disk, CPU and network capacity. Use an active-active node set for critical feeds and keep it separate from the Power BI on-premises data gateway.
Choose deliberately
- One node is acceptable for a non-critical pilot; resilient workloads use two or more nodes across failure domains.
- Scale up for compression and parallel copy, then scale out when one host becomes a shared bottleneck.
- Keep gateway patterns separate: SHIR moves pipeline data; the Power BI gateway serves supported on-premises refresh or live-query cases.
What protects the hand-off
- Dedicated hardened hosts
- Outbound-only firewall path
- Patched runtime and OS
- Service health and capacity alerts
What proves it
- Node inventory and version
- Firewall egress record
- Failover exercise result
Primary failure mode
A runtime node, local network route or service identity becomes unavailable midway through an extraction.
Contain & recover
Fail work to a healthy node, retry only idempotent copy activity, and reconcile run manifests before advancing the watermark.
03 / Orchestrate / Cloud control plane / private execution boundary
Integration boundary
Azure Data Factory or Fabric Data Factory owns schedules, dependency state and run metadata after an explicit platform decision, while execution remains isolated from consumers and source credentials.
- Freshness
- By workload
- RTO
- ≤4h
- RPO
- No confirmed run loss
How this stage works
Use metadata-driven pipelines, parameterised datasets and explicit linked services. Credentials resolve from Key Vault or managed identity; retry, timeout, concurrency and dependency rules are set by workload class.
Choose deliberately
- Assess ADF and Fabric Data Factory against connectors, networking, SHIR, deployment, monitoring and lineage; preserve portable contracts and run evidence.
- Use a managed API boundary when source APIs require throttling, transformation or partner-facing policy.
- Separate dev, test and production factories, identities, endpoints and release approvals.
What protects the hand-off
- Managed identity and secret references
- Environment separation
- Run IDs and correlation IDs
- Retry and concurrency policy
What proves it
- Pipeline release record
- Linked-service identity map
- Run and dependency log
Primary failure mode
A retry duplicates data, a dependency runs out of order, or a secret rotation breaks the linked service.
Contain & recover
Pause downstream promotion, re-establish identity, replay the failed run with its original correlation ID, and prove idempotence through manifests and target counts.
04 / Transfer / Corporate egress / Azure private endpoint boundary
Encrypted transfer
Data moves over encrypted, explicitly allowed routes. The route is designed as a one-way analytical path, not a new inbound doorway into the operational estate.
- Freshness
- No added tier breach
- RTO
- ≤4h
- RPO
- Last durable chunk
How this stage works
Use TLS with restricted egress, private endpoints and private DNS for Azure data services. Add site-to-site VPN or ExpressRoute where network policy or throughput requires it; allow only named destinations and ports.
Choose deliberately
- Private endpoints reduce public exposure, but public network access must still be disabled or restricted explicitly.
- Provision and resolve both blob and dfs endpoints when ADLS Gen2 operations require both service surfaces.
- Size bandwidth from compressed peak volume, recovery backlog and competing business traffic—not average daily volume.
What protects the hand-off
- TLS in transit
- Private DNS validation
- Named firewall allow-list
- Network flow logging
What proves it
- Data-flow diagram
- Approved firewall rules
- Private endpoint and DNS tests
Primary failure mode
A DNS change resolves a service publicly, a firewall rule expires, or available bandwidth cannot clear the ingestion backlog.
Contain & recover
Block promotion, validate name resolution from each runtime node, restore the approved route, then drain the backlog in priority order while watching source load.
05 / Contain / Governed Azure data subscription
Azure landing zone
Subscription, resource-group and policy boundaries provide the workload with known ownership, residency, cost, diagnostic and network controls before data is admitted.
- Freshness
- N/A
- RTO
- ≤4h critical
- RPO
- Config from code
How this stage works
Deploy policy-compliant storage, orchestration, observability, Key Vault, private endpoints and identities through versioned infrastructure. Send platform diagnostics to a protected central workspace.
Choose deliberately
- Separate environments and duties; do not use a production data account as a development convenience.
- Select region, redundancy and encryption options from classification, recovery and residency decisions.
- Use customer-managed keys only where the control benefit justifies rotation and availability dependencies.
What protects the hand-off
- Azure Policy guardrails
- Resource locks and tags
- Budget and anomaly alerts
- Central diagnostics
What proves it
- Infrastructure deployment record
- Policy compliance export
- Identity and network inventory
Primary failure mode
Configuration drift re-opens a public surface, removes diagnostics or deploys a resource outside the approved region.
Contain & recover
Quarantine the change, redeploy the approved infrastructure baseline, verify policy and network posture, and document any data exposure assessment.
06 / Land / Bronze / raw data zone
Lake & staging
Source-shaped data lands immutably with the context needed to reproduce, inspect and replay every load. Failed validation is isolated rather than hidden.
- Freshness
- Tier target minus processing
- RTO
- ≤4h
- RPO
- Durable raw history
How this stage works
Partition by stable domain and ingestion time, retain source timestamps and keys, and write a manifest containing run ID, watermark, file list, counts and checksums. Publish completion atomically after all parts arrive.
Choose deliberately
- Preserve raw history long enough to meet replay, audit, privacy and cost requirements.
- Quarantine malformed files or contract failures with reason codes; never silently coerce trusted fields.
- Avoid over-partitioning small data and avoid mutable folder conventions that erase load history.
What protects the hand-off
- ADLS RBAC and ACLs
- Immutable run folders
- Manifest and checksum
- Retention and deletion policy
What proves it
- Raw run manifest
- Source-to-land reconciliation
- Quarantine register
Primary failure mode
A partial write looks complete, a duplicate batch is landed, or a malformed record bypasses schema validation.
Contain & recover
Keep the run unpublished, compare manifest to actual objects, remove only the incomplete run scope, and replay from the source watermark or durable extract.
07 / Conform / Silver / validated data zone
Transformation
Raw records become typed, deduplicated and conformed data with explicit treatment for bad, late and changing information. Quality gates decide whether data can progress.
- Freshness
- Within tier budget
- RTO
- 4–8h
- RPO
- Replay from bronze
How this stage works
Apply schema enforcement, reference-data validation, deterministic deduplication, standardised keys and timestamps, and late-arriving rules. Make processing idempotent by run or business key.
Choose deliberately
- Fail the load when a critical invariant breaks; warn and measure when a non-critical threshold degrades.
- Use contract tests for structure and domain tests for business validity, completeness and uniqueness.
- Version transformations and test representative history before promoting a rule change.
What protects the hand-off
- Schema and contract tests
- Quality thresholds
- Idempotent merge logic
- Code review and release gates
What proves it
- Test result by run
- Rejected-record sample
- Transformation version
Primary failure mode
Schema drift, an incorrect deduplication rule or a late-arriving fact corrupts a conformed entity.
Contain & recover
Hold the silver-to-gold gate, restore the last trusted rule version, replay affected bronze partitions, and reconcile corrected outputs against the incident scope.
08 / Model / Gold / governed data product zone
Curated model
Conformed entities become business-ready facts, dimensions and measures with a named product owner, steward and contract for consumers.
- Freshness
- 15m / 2h / daily
- RTO
- 4–24h
- RPO
- By product tier
How this stage works
Publish stable dimensional or domain models, conformed dimensions, declared grain, slowly changing history and certified calculation definitions. Version breaking contract changes and communicate deprecation windows.
Choose deliberately
- Model for decision use, not as a mirror of source tables.
- Declare grain, currency, timezone, status logic and inclusion rules beside every critical measure.
- Keep experimental fields separate from certified products until ownership and quality are proven.
What protects the hand-off
- Data product contract
- Named owner and steward
- Measure certification
- Backward-compatibility tests
What proves it
- Model and measure dictionary
- Source-to-product lineage
- Owner acceptance record
Primary failure mode
A semantic change preserves schema but alters a KPI, breaking management confidence without a technical pipeline error.
Contain & recover
Withdraw certification, restore the prior product version, quantify affected reports and decisions, then release the corrected definition with owner sign-off.
09 / Serve / Semantic and insight consumption zone
Power BI
Reports, embedded experiences and approved self-service analysis consume certified cloud data products—not operational ERP or WMS connections.
- Freshness
- Per product tier
- RTO
- ≤4–8h
- RPO
- Last certified refresh
How this stage works
Choose Import, DirectQuery or composite models from freshness, scale and functionality needs. Use incremental refresh where appropriate, reusable semantic models, certified measures, row-level security and controlled workspaces.
Choose deliberately
- Prefer Import for predictable interactive performance when the freshness target allows it.
- Use DirectQuery deliberately for low-latency needs and test source concurrency, query folding and user experience.
- Use an on-premises data gateway only for justified on-premises sources; deploy a standard-mode gateway cluster for high availability and do not co-host it with SHIR.
What protects the hand-off
- Workspace deployment pipeline
- RLS and object-level controls
- Certified semantic models
- Refresh and usage monitoring
What proves it
- Model refresh history
- RLS role test
- Report-to-model lineage
Primary failure mode
A refresh fails, an RLS role is mis-scoped, or a report bypasses the certified model and recreates a critical measure.
Contain & recover
Disable or roll back the affected artifact, restore the last trusted model, validate roles with representative users, and reconcile impacted decisions or exports.
10 / Assure / Cross-cutting governance and operations plane
Access & audit
Identity, lineage, quality, service health and decision evidence span every stage. Access is group-based, time-bounded where appropriate and reviewed against real use.
- Freshness
- Measured end-to-end
- RTO
- Per incident tier
- RPO
- Audit evidence retained
How this stage works
Use Entra groups, Azure RBAC and storage ACLs at the correct scope; catalogue data products and lineage; centralise run, access and security logs; publish freshness and quality service dashboards.
Choose deliberately
- Separate platform administration, data engineering, stewardship and consumer duties.
- Automate lineage where supported and record manual lineage for unsupported transformations and business decisions.
- Review privileged and sensitive-data access on a defined cadence, including dormant consumers and service principals.
What protects the hand-off
- Group-based least privilege
- Purview catalogue and lineage
- Access reviews and alerting
- Runbooks and service dashboard
What proves it
- Access review export
- End-to-end lineage record
- SLO and incident history
Primary failure mode
A privileged identity persists after need ends, lineage has a blind spot, or a stale dataset appears healthy to consumers.
Contain & recover
Revoke access, contain affected products, reconstruct lineage from run IDs and release records, notify owners, and complete an evidence-backed incident review.
02 / Network & security
Cross boundaries through named control points.
The route starts inside the operational estate, crosses a dedicated execution enclave, lands in a private Azure data zone and ends at governed semantic access.
ZONE 01 Operational trust zone
ERP, WMS, SQL, files and vendor APIs. Read-only contracts, source-safe windows and no analytical users.
Source ACLs · query budgets · classificationZONE 02 Integration enclave
Dedicated Windows hosts run SHIR close to private sources. Critical feeds use active-active nodes and monitored outbound HTTPS.
Hardened hosts · HA · egress allow-listZONE 03 Private Azure data zone
Orchestration, lake and processing services use managed identities, private endpoints, private DNS and central diagnostics.
Policy · RBAC/ACL · Key Vault · logsZONE 04 Governed consumption zone
Certified data products and semantic models serve embedded and self-service insight through controlled workspaces and roles.
Certification · RLS/OLS · access review
SHIR is the pipeline bridge
Run it on dedicated, hardened Windows hosts. Critical feeds use multiple active-active nodes so maintenance or a host fault does not become a single point of failure.
Do not co-host gateway roles
SHIR and the Power BI on-premises data gateway have different operational jobs and should not share a machine. Prefer Power BI over curated cloud data; use a gateway only for a justified on-premises refresh or live-query path.
Private is an end-to-end posture
A private endpoint alone does not remove public access. Test DNS from each runtime host, restrict the public surface and retain the approved firewall, endpoint and flow evidence.
03 / Ingestion decisions
Choose capture from change behaviour, not fashion.
A domain can mix patterns. The important decisions are how a change is identified, how far back it can be replayed and what load the source can safely tolerate.
| Pattern | Use when | Primary trade-off | Replay stance | Source impact |
|---|---|---|---|---|
| Full snapshot | Initial loads or small, slow-changing reference entities. | Simple, but high source, network and reconciliation cost at scale. | Replace a versioned snapshot; never overwrite the only raw copy. | High and predictable |
| Watermark incremental | A reliable increasing timestamp or key identifies changed rows. | Portable, but misses in-place change when the watermark is unreliable. | Store low/high watermarks and overlap safely to catch late commits. | Low to medium |
| Change tracking / CDC | The engine exposes a supported, retained change stream. | Efficient deletes and updates, with retention and ordering dependencies. | Persist offsets and raw change history; rebuild before retention expires. | Low after setup |
| API or event feed | A vendor contract or low-latency event path is authoritative. | Clear contract, but throttling, paging, ordering and expiry need design. | Retain payload plus cursor or event offset; design dead-letter handling. | Contract-controlled |
| File drop / micro-batch | Partners or legacy systems publish complete extracts on a cadence. | Decoupled, but partial files and duplicate delivery must be detected. | Manifest, checksum and immutable receipt folder by delivery ID. | Low at ingestion time |
Platform decision
ADF or Fabric Data Factory?
Treat orchestration as a current architecture decision. Keep Azure Data Factory where it is the established, supported estate; assess Fabric Data Factory for new integration work when its capacity, networking, governance and operating model fit.
- Compare
- Connectors · private networking · SHIR support · deployment · monitoring · Purview lineage
- Preserve
- Data contracts · run IDs · manifests · idempotence · quality gates · tiered recovery
- Prove
- A representative thin slice and failure drill before platform commitment
04 / Quality & governance
Promote data only when its evidence advances too.
Bronze, silver and gold are responsibilities rather than product names. A simpler implementation is valid if it preserves the same separation between receipt, validation and certified meaning.
Immutable & source-shaped
Retain original payload, source time, ingestion time, run ID, watermark, file manifest and checksum. Invalid delivery is visible in quarantine.
Exit gateComplete run + source reconciliation + published manifestValidated & interoperable
Enforce schema, standardise keys and types, deduplicate deterministically, resolve reference data and apply explicit late-arriving rules.
Exit gateContract tests + quality thresholds + rejected-record evidenceBusiness-ready & owned
Publish declared-grain facts, conformed dimensions, stable domain products and certified measures with versioned consumer contracts.
Exit gateOwner approval + lineage + semantic regression resultsCross-cutting assurance
- Data contracts & schema history
- Owner, steward & classification
- Quality, freshness & volume SLOs
- Automated + declared lineage
- RBAC, ACLs & access reviews
- Retention, privacy & deletion
Lineage needs a complete story
Collect automated pipeline lineage where the platform supports it, then add the parts automation cannot infer: business rules, semantic measures, manual approvals, exports and the decision products that consume them. Run IDs connect operational proof to that logical story.
05 / Reliability model
Define failure before production defines it for you.
Freshness, RPO and RTO belong to each data product. The targets below are starting points for discovery—not Microsoft service commitments or universal promises.
Illustrative service targetsValidate business impact, source retention, platform capability, support hours and cost for every domain before adoption.
≤15 minutes target; alert by 30 minutes
- RPO
- 15 minutes
- RTO
- 4 hours
Inventory position, dispatch exceptions, urgent order flow
≤2 hours
- RPO
- 1 hour
- RTO
- 8 hours
Trading, service, margin and warehouse performance
Complete by 06:30 local time
- RPO
- 24 hours
- RTO
- 24 hours
Finance close support, daily executive packs, slower master data
Proposed service objectives
Measure the whole hand-off.
- ≥99.5% of scheduled runs complete inside their service window, excluding owner-approved source maintenance.
- 100% of promoted loads carry a run ID, manifest and source-to-target reconciliation result.
- ≥99.8% completeness for required records and fields, or the domain-specific stricter threshold.
- Critical pipeline, freshness and access failures alert the named responder within five minutes of detection.
Failure-mode register
Detect. Contain. Recover. Prove.
Recovery is complete only when data is reconciled and the next hand-off has not consumed a partial or untrusted state.
| Failure mode | Detection signal | Containment | Recovery & evidence |
|---|---|---|---|
| Source load or lock contention | Query duration, blocking or ERP/WMS latency crosses the agreed budget. | Cancel extraction; keep the last confirmed watermark. | Move to the approved window, replica or lower-concurrency plan; re-baseline. |
| SHIR node or route unavailable | Runtime heartbeat, copy queue or connection test fails. | Fail to a healthy node; prevent downstream completion. | Restore node or egress, replay the run, reconcile manifests and counts. |
| Private DNS or firewall drift | Name resolution, TLS handshake or network-flow alert changes. | Block transfer and public fallback; preserve queued work. | Redeploy approved DNS/rules, test from each node, drain by priority. |
| Schema drift or quality breach | Contract, null, uniqueness, volume or reference-data gate fails. | Quarantine the run; do not promote bronze to trusted layers. | Approve a compatible contract or fix the source/rule, then replay raw data. |
| Partial, late or duplicate batch | Manifest mismatch, missing completion marker or repeated delivery ID. | Keep the batch unpublished and isolate its run scope. | Complete or replay idempotently; deduplicate by run and business key. |
| Identity, key or permission failure | Authentication, authorisation or key-health alert fails a known path. | Do not broaden access as a workaround; halt the affected stage. | Restore the approved identity/key, verify least privilege, record rotation evidence. |
| Bad model or semantic release | Regression test, KPI reconciliation or consumer incident finds changed meaning. | Withdraw certification or route users to the last trusted version. | Roll back, replay affected products and publish an impact statement. |
| Power BI refresh or RLS failure | Refresh SLA, role test or access anomaly alert fails. | Disable the affected artifact or distribution route. | Restore last certified model, re-test representative identities and republish. |
06 / Rollout & evidence
Scale the proven control path, not just the pipeline.
Begin with one consequential but bounded domain. Each wave leaves behind reusable assets and passes an evidence gate before scope expands.
- Wave 00
Decide & found
One domain, tier, owner, classification and target architecture agreed.
Evidence gateSigned data contract, threat model, service tier and cost envelope. - Wave 01
Prove the thin slice
One representative entity flows source → bronze → silver → gold → Power BI.
Evidence gateReconciled counts, measured source impact and a repeatable deployment. - Wave 02
Harden movement
Resilient runtime, private route, incremental capture and replay are operational.
Evidence gateNode failover, network recovery and backlog tests meet the chosen tier. - Wave 03
Engineer trust
Medallion layers, contracts, quality thresholds and lineage are automated.
Evidence gateBad-data drill proves quarantine, rollback and deterministic replay. - Wave 04
Serve & adopt
Certified semantic model, RLS, embedded or self-service experience reaches users.
Evidence gateOwner signs off measures, role tests, performance and adoption support. - Wave 05
Scale & operate
More domains onboard through templates, service dashboards and runbooks.
Evidence gateRecovery exercise, access review and SLO trend accepted into service.
Minimum operating evidence
The handover is a proof pack.
Store evidence with clear retention and ownership. A live dashboard is useful, but release records, manifests, reviews and recovery exercises must remain auditable after the screen changes.
- 01Source register, owner, classification and extraction contract
- 02End-to-end data-flow diagram and threat model
- 03Runtime inventory, firewall rules, private endpoint and DNS tests
- 04Identity map, linked-service permissions and key-rotation record
- 05Run manifest, watermark, checksum and source-to-land reconciliation
- 06Schema, quality, deduplication and transformation test results
- 07Source-to-report lineage and certified measure dictionary
- 08RLS tests, privileged-access review and workspace release record
- 09Failure drill, restore evidence and measured RTO/RPO
- 10Freshness, quality, pipeline success and incident service dashboard
07 / Design references
Grounded in current Microsoft guidance.
These are design inputs, not substitutes for workload testing, threat modelling or an agreed service model. Links open Microsoft Learn in a new tab.
- 01
Self-hosted integration runtime
Deployment, outbound connectivity, host guidance and multi-node scale-out.
Microsoft Learn - 02
Integration runtime concepts
Execution placement, network boundaries and runtime selection.
Microsoft Learn - 03
Fabric Data Factory
Current Microsoft data-integration capabilities for new platform decisions.
Microsoft Learn - 04
Storage private endpoints
Private routing, public-access posture and blob/dfs endpoint considerations.
Microsoft Learn - 05
ADLS access-control model
Azure RBAC, ABAC and POSIX-style ACL design for lake data.
Microsoft Learn - 06
Medallion architecture
Bronze, silver and gold responsibilities in a layered lakehouse pattern.
Microsoft Learn - 07
Microsoft Purview lineage from ADF
Supported source, activity and output lineage plus current limitations.
Microsoft Learn - 08
Power BI on-premises gateway
When a gateway is required and how standard-mode clusters support resilience.
Microsoft Learn - 09
Power BI row-level security
Semantic-model row filtering and role behaviour in the service.
Microsoft Learn
Architecture principles
The rules that survive product change.
Technology choices can evolve. These decision rules keep the architecture coherent as delivery moves from target state to transition and operation.
Sources remain operational systems of record
Every movement is observable and replayable
Data quality is explicit and measurable
Access is granted through governed models