Platform Build / Rapid Start
Snowflake-Native Data Stack Implementation and Matillion Migration for Major K-12 EdTech Platform
The Situation
A leading K-12 curriculum and assessment platform serving more than 10 million students across all 50 US states had made a strategic decision to rebuild its data infrastructure around a modern, Snowflake-native, code-first stack. Their existing Matillion-based pipeline estate was the primary obstacle — it had accumulated significant technical debt, rising licensing costs, and was incompatible with the company's Terraform-first infrastructure model where all cloud resources are managed as code. Key challenges included:
- A large, complex Matillion pipeline estate with underdocumented logic and hidden technical debt across individual jobs
- Unreliable connectors causing recurring data issues
- A requirement to rebuild all ingestion and transformation logic in Fivetran and dbt with Snowflake as the destination, while maintaining full data parity at every step
- All Snowflake objects, schemas, and roles needing to be built and managed within a Terraform infrastructure-as-code framework
- A documentation standard requiring complete YAML files, doc blocks, and Markdown column and table-level documentation for every migrated model
What We Did
Conducted a full audit of all Matillion jobs, documenting schedules, logic, pipeline dependencies, and business use case mappings
- Assessed each pipeline to determine whether native Fivetran connectors could replace Matillion ingestion, or whether custom AWS Lambda functions were required
- Built all Fivetran connectors, AWS Lambda functions, and Snowflake objects — including tables, schemas, and roles — within Terraform, aligning the full data platform with the client's infrastructure-as-code standard
- Transposed Matillion SQL logic into dbt models targeting Snowflake, pipeline by pipeline, with full parity validation comparing original Matillion output against new dbt model outputs
- Produced comprehensive documentation for every migrated model, using AI tooling to accelerate what would otherwise be weeks of manual documentation work — reducing a one-week documentation task to under one day
- Deprecated Matillion jobs incrementally as each Snowflake-native pipeline was certified, avoiding a big-bang cutover
- Maintained burndown reporting and weekly standups throughout, operating as an embedded part of the client's internal data team
Results
Major pipelines including submission, onboarding, and retail fully migrated to the Snowflake-native stack and certified
- Snowflake objects, schemas, and roles now fully managed within Terraform alongside all other infrastructure
- Every migrated pipeline accompanied by thorough documentation — a material upgrade over the prior state
- Matillion licensing costs reducing incrementally as jobs are deprecated and replaced by the Snowflake-native architecture
- Documentation efficiency improved significantly through AI-assisted YAML and doc block generation
What This Unlocks
For organizations running Matillion that want to move to a governed, Snowflake-native, code-first data stack, this engagement provides a proven migration playbook covering audit, classification, parity validation, documentation, and incremental deprecation. The Snowflake plus Fivetran plus dbt plus Terraform architecture pattern is directly reusable, and the AI-assisted documentation approach significantly reduces the overhead of large-scale migration work.
Team
Led by a senior director of strategy solutions for initial scoping, with a delivery lead overseeing burndown and staffing, a senior data engineer serving as primary technical lead on pipeline migration and Terraform builds, and a data architect handling SQL-to-dbt conversion and parity analysis across all migrated pipelines.
Read more Snowflake Use Cases
Migration / Modernization
Tableau-to-Sigma Migration and Ongoing Developer Enablement for an AI Platform Company
Migration / Modernization
Tableau-to-Sigma Migration and Custom App Development for a Global Consumer Products Company
A global consumer products manufacturer with a portfolio of widely recognized licensed and proprietary brands had reached the limits of its existing analytics setup. Rising Tableau licensing costs had made the platform difficult to justify, and an entire class of critical business processes was still being managed through hundreds of individual spreadsheets with no scalable path forward.