This blog post is Human-Centered Content: Written by humans for humans.

One of our biggest projects this year was a white paper collaboration between InterWorks and dbt, one of the most important players in the data space at the moment, titled “Semantic Layers in Action: Real-World Use Cases and Business Impact.” Nor surprisingly, for white paper co-author Jack Hulbert, it was his favorite project of the year.

Making sense of the rapidly evolving AI/ML landscape is a dizzying proposition. Even if you’re trying to stay up to date on surface level knowledge, it seems like that can slip away in the moment. With his unique expertise, Jack wanted to work on one aspect of the field in depth, tag teaming with some of the leaders in the arena of semantic layers.

From the Top

So, first off, I had to ask Jack, “How did this white paper project start?”

Earlier this year, when we started launching our partnership with DBT and working to grow it, one of our goals at InterWorks was to validate the value proposition for the semantic layer. We wanted to figure out the right patterns for implementing it, how to tell that story to our clients, and the best options in this category. Of those options, we wanted to identify the patterns we’d recommend as first steps for our client base.

With dbt, we already had an existing partnership. They’ve been investing heavily in their built-in semantic layer integration over the last couple of years, which aligns strongly with our focus on validating use cases for the semantic layer.

Speaking of use cases for the semantic layer (which are outlined in depth in the white paper above),  I wanted to get a brief look at what his philosophy was behind putting together a white paper on this topic:

One of the core themes for the business intelligence practice at InterWorks has been the portability of development data. Rather than being locked into or limited by one or two tools, the goal is to create data products that can be funneled into any tooling, reporting platform, spreadsheet or front-end application. The semantic layer helps enable that. That’s the alignment we wanted to explore — showing how clients can take their work in dbt and push it out to more places than just a dashboard.

The semantic layer is, today, the best way for developers to achieve that. Ultimately, we want to show our clients that there are more ways to consume data and maximize its value than just using a dashboard.

And why go to dbt in the first place for collaboration?

We’ve been growing our working relationship with dbt for a while now. One of our main contacts, Roxi Dahlke, is the product manager for the semantic layer at dbt. We connected with her and her team, and we pitched the idea. We asked if they’d be interested in collaborating to talk about the semantic layer — how it could be used to grow this mission in a way that’s mutually beneficial. For dbt, it’s about telling the story of this new technology category and showing their customers, “Hey, you already have dbt. Here’s how can you use this additional feature to maximize the value of your data.” For our clients, it’s about the same thing.

It was a mutually beneficial story to tell.

Writing the White Paper Itself

So, with the green light to work together on this white paper, Jack and Roxi set to work:

With [the semantic layer] being new [to InterWorks], we came in with a lot of hypotheses. While we had customers who were interested, none were necessarily using it in a production state — most were tinkering and asking questions about it.

The key research phase for us was partnering with dbt to understand how their customers were using it in production. What gaps did we have in our understanding? That partnership was critical because they provided valuable insights into real-world design and implementation patterns for the semantic layer. They showed us how their customers were using it, the types of tooling they were integrating it with and the workflows involved.

It was really about filling in the gap — seeing what the semantic layer looks like in the wild right now.

On the InterWorks side of things, earlier in the year, Brooks Barth and I worked on evaluating different semantic layer tools. We used an internal dataset to build an example project from start to finish. The goal was to understand how it works and connect it to a few different BI platforms to see how it integrates.

During the technical evaluation and research phase, we quickly identified the current limitations, where it works well and areas that could be improved. That research directly informed the content we’ve put out, like blogs and the white paper, which discuss the current state of the semantic layer — where we recommend it, where it’s emerging and where it needs further development before being production-ready.

That technical evaluation phase was fundamental to shaping our stance and positioning on semantic layers, not just with dbt but in general.

And how was it working with Roxi and the dbt team?

Roxi brought clear intentions about the story we wanted to tell. She was very aware that this is an emerging category. We worked together to be pragmatic and honest about the current limitations, what’s aspirational and what’s still in development. I thought that was a really healthy approach because, from our customers’ perspective, this is such a new area.

The Final Product

Of course, the white paper is finished and has been out in the world for some time (available for download here). But what’re the takeaways Jack wanted the readers to have with this project?

There are a few key takeaways here. First, building on our core theme of data portability over the last couple of years, the semantic layer offers a strong value proposition. It allows teams to deploy their work across data platforms like dbt and Snowflake, extending beyond just business intelligence.

With the semantic layer, we’re thinking about clients who want to productize or monetize their data, or build internal tools for developers through an API. These are emerging use cases that have become more prominent in the last few years. There’s growing importance in having multimodality — more ways to use data than just putting it into a dashboard.

The key takeaway is that with dbt, you don’t necessarily need to buy a completely separate tool to achieve this.  There’s a pattern built into dbt that you can make your data multimodal.

And what’s next for Jack and the semantic layer, now that his white paper project is out in the world?

Next, I’d like to work on a similar project — whether it’s a white paper, a series of webinars, blogs or in-depth articles — focused on real-world design patterns. Specifically, how do you push the semantic layer into production? What does that work look like? What’s the architecture of a good semantic layer project?

Another emerging topic is the importance of the semantic layer for generative AI, particularly when training large language models. We hypothesized early on that a semantic layer would significantly improve accuracy when using tools like ChatGPT or Anthropic LLMs in internal applications. A research paper we cited in the white paper highlights that having a semantic layer on top of your data provides metadata about the data stored in your database, how it’s related and how it should be aggregated, which boosts the accuracy of these AI models.

I’m hearing more and more interest in building applications like that internally now that we have better tooling. We’re moving past the early adopter phase of generative AI applications, and it’s becoming as common as dashboards.

I’d like to explore the semantic layer’s role in this in future pieces, with a focus on practical implementation rather than just its importance.

So, keep an eye out for future semantic layer projects coming from Jack and the rest of the InterWorks team!