Databricks Data+AI Summit just wrapped and it was a big one. Genie One launched as a full agentic coworker, Lakeflow got a lot of attention as Databricks’ answer to a unified ingestion and transformation layer, and Agent Bricks expanded into a full agent platform with governance, memory, and document intelligence built in. A lot of ground covered in a few days.
A few themes kept coming up across sessions and conversations: Consolidation of the data stack, automation through Genie, and simplification of what used to require multiple tools and teams.
One question had me thinking about the blurred lines between data engineering and analytics engineering. A client asked me, “With Spark Declarative Pipelines, do I still need dbt?”
Why dbt Exists
To answer whether dbt is still needed, let’s first understand why it exists.
Before dbt, analytics teams with strong SQL skills still lacked the software engineering practices that made code reliable, maintainable and safe to hand off to someone else. After dbt filled that gap, analytics teams started using version control, working across separate dev, stage and prod environments, and writing automated tests against their data. The learning curve for dbt is low because if you know SQL you are already 95% of the way there, the rest is just Jinja and curly brackets.
I still love dbt for how it up-leveled analytics teams and lowered the barrier to entry, but that does not mean it is always the right tool for transformations.
The Real Question
It was never dbt vs. Spark Declarative Pipelines (SDP). The real question: Who is on your team and who owns data transformations?
If you have a small team with mostly strong SQL skills, even if you are on Databricks, using dbt can pass the ownership from your one data engineer to your three analysts.
But if you have more data engineers and more than just strong SQL skills, dbt might be redundant. Most data engineers I know would prefer PySpark or Python to dbt when they have a choice.
Spark Declarative Pipelines gives you everything dbt gives you, plus automatic compute scaling, built-in data quality, streaming and batch in the same pipeline, and deep Unity Catalog integration. No extra setup, no extra connections, no extra bill. The trade-off is a steeper learning curve. More power, more responsibility.
Data Engineering Is Taking Its Ground Back
If you are an analytics engineer running SDP pipelines, congratulations, you and the data engineer on your team are doing more or less the same job now. Buy them a coffee.
The disciplines are converging because the tooling is converging. Analytics engineering emerged partly because data engineering tools had a high barrier to entry. dbt lowered that barrier for SQL-first teams. Now SDP is raising the ceiling while keeping things approachable for teams that already have engineering talent.
The goal has not changed. Take messy raw data and transform it so it is easier and faster to get insights. Version control, separate environments, DRY code, modular design. These principles belong to good data work, not to any specific tool.
Do You Still Need dbt on Databricks?
I’m going to be a stickler on the word “need,” because words matter. So the answer is no, you do not need dbt if you are on Daatbricks. Databricks can be the only tool in your data stack. Lakeflow handles ingestion (Lakeflow Connect), transformation (Spark Declarative Pipelines) and orchestration (Lakeflow Jobs) natively.
If you want dbt, use it. There are no wrong answers when best practices are being followed and the outcomes are good.
If you are starting fresh on Databricks or already moving toward SDP, use the native tooling. More features, tighter integration, one less thing to maintain.
📝 Note: The same logic applies beyond Databricks. If you are on GCP, Dataform is native and more capable than dbt out of the box. Whatever platform you are on, reach for the native tooling first and add dbt only if you have a specific reason.
The Bigger Picture from Summit
The main theme this year that I latched onto was consolidation. Databricks is not trying to be a restaurant with a massive menu where everything is mediocre. They are betting they can do ingestion, transformation, orchestration, governance, analytics, AI, and app deployment at a high level in one platform.
After a week of sessions, that bet looks like it’s landing. The fragmented data stack stitched together with five tools is not a requirement anymore.
That does not mean every third-party tool is dead. It means the bar for adding something to your stack just got higher. If Databricks does it well and it’s already there, the burden of proof is on the external tool. Pick the one that works best for you and your team.
