Migration / Modernization

Legacy Pipeline Migration to Snowflake for National Public Health Data Provider

The Situation

A national shared services organization providing data infrastructure to a network of public health bodies across an entire country was running approximately 100 data pipelines on a legacy on-premise platform. Their Snowflake environment was already in place but underused, consuming credits primarily on data replication rather than productive workloads. Key challenges included:

  • No modern transformation or orchestration layer in place
  • Legacy pipelines built without version control, CI/CD, or peer review practices
  • Internal team technically capable but without exposure to modern data engineering standards
  • Pressure to demonstrate value from an existing Snowflake contract before renegotiation

What We Did

Designed a full cloud-native architecture: dbt Core for transformations, Apache Airflow (AWS MWAA) for orchestration, and DLT for ingestion

  • Ran a proof-of-concept comparing Apache Airflow against the client's preferred option (AWS Step Functions) and recommended Airflow based on fit
  • Implemented dbt Core within the Snowflake environment with comprehensive blueprints, training, and documentation
  • Built a reusable DAG factory in Airflow, reducing future per-pipeline migration effort to approximately 5 lines of configuration per pipeline
  • Introduced CI/CD pipelines, branch policies, pre-commit checks, and peer review practices across the codebase
  • Onboarded external graduate consultants to the new architecture
  • Provided independent advisory on Snowflake credit consumption to help the client right-size ahead of contract renegotiation

Results

  • Full modern cloud architecture designed, built, and operational
  • DAG factory established, enabling rapid scale-out across the remaining pipeline backlog
  • Engineering team trained and working to modern software development standards for the first time
  • Architecture blueprints embedded into the client's internal wiki for self-sustaining adoption
  • Engagement extended three times over approximately nine months
  • Adjacent organizations within the same public health network now in scoping for replication of the same approach

What This Unlocks

For public sector organizations or any technically capable team running legacy on-premise pipelines with an underused Snowflake investment, this engagement shows a proven path to modernization using an open-source stack. The DAG factory pattern, dbt Core blueprints, and migration architecture are reusable and have already been validated for scale.

Team

Led by a senior data engineer serving as on-site delivery lead, supported by a public sector account lead and a senior solutions architect providing strategic advisory across the engagement.

Back to Snowflake Use Cases
Snowflakedbt CoreApache Airflowpipeline migrationpublic sectorhealthcarelegacy modernizationopen-source stackEMEA

Read more Snowflake Use Cases

Need help with Snowflake
planning, implementation or optimization?

Services Contact Us

InterWorks uses cookies to allow us to better understand how the site is used. By continuing to use this site, you consent to this policy. Review Policy OK

×

Interworks GmbH
Ratinger Straße 9
40213 Düsseldorf
Germany
Geschäftsführer: Mel Stephenson

Kontaktaufnahme: markus@interworks.eu
Telefon: +49 (0)211 5408 5301

Amtsgericht Düsseldorf HRB 79752
UstldNr: DE 313 353 072

×

Love our blog? You should see our emails. Sign up for our newsletter!