Have you ever wanted to make it big as a musician? Then you might know just how hard it is to share your music with the world, let alone create it. Once upon a time, you had to sign to a label to get anywhere close, but that changed in 2006 when Brooklyn-based TuneCore provided an audacious alternative path to success: The ability for artists to distribute their music online to leading digital music stores like Spotify and Apple Music while keeping 100% of the profits. The idea caught on, and TuneCore is now the world’s leading digital music aggregator. Distribution is just the start. TuneCore also provides numerous artist services, including music publishing administration and artist education.

A New Groove with Snowflake, Matillion and Tableau

TuneCore is no stranger to data. With countless plays on countless platforms from countless people across the world, the sheer volume of data generated by TuneCore artists is staggering. The advent of music streaming has contributed mightily to this exponential explosion of data. Their existing MySQL reporting environment and its accompanying data pipeline kept up for a time, but as more and more data made its way through the pipeline, bottlenecks began appearing. Query load times began taking longer, as did nightly data refreshes. Trying to troubleshoot and optimize these issues took significant time and effort from the TuneCore Data Analytics Team. Challenges with data polymorphism and tables not running efficiently began emerging more frequently, which made it difficult to turn reports around in a timely manner for other departments.

Knowing that their existing data architecture and reporting framework wouldn’t last for much longer, TuneCore Director of Data Analytics, Raj Sarma, began researching solutions for a new data architecture that could better meet their growing needs. Raj eventually fell in favor of a stack that included Snowflake as a scalable cloud data platform and Tableau for insightful reporting, but he was also looking for an ETL solution that would help them transform data and orchestrate workflows with ease. Matillion stood out because of its ability to handle data prevalent at TuneCore, as well as its easy-to-use GUI for setting up workflows.

With the technology stack selected, Raj began conversations with those vendors. During these conversations, Tableau recommended InterWorks as the ideal partner to help TuneCore implement their new stack. After a few conversations, Raj felt that a partnership with InterWorks would be a great fit and decided to engage InterWorks for a proof of concept – think of it like the data equivalent of a demo tape. This proof of concept would demonstrate whether the stack of Tableau, Snowflake and Matillion could indeed handle TuneCore’s needs before they made an all-in investment.

“When you’re on an airplane and they go through emergency protocols, they tell you to put on your oxygen mask first and then you’ll be able to better help others. It’s the same with data. We knew that we wouldn’t be able to empower our customers with better insights if we didn’t first enable ourselves, which is why we started with an internal proof of concept.”

– Raj Sarma, Director of Data Analytics, TuneCore

Jamming out with an InterWorks Proof of Concept

After some initial planning, TuneCore and InterWorks decided that the proposed proof of concept would be split 50/50 between data architecture and Tableau dashboarding work. On the InterWorks side, Data Lead Michael Treadwell, Data Architect Justin Lemmon and Analytics Consultant Matthew Albacete got to work on the data architecture portion and began gathering requirements alongside the TuneCore Data Analytics Team to tailor Snowflake and Matillion to their specific needs.

As the InterWorks team began standing up Snowflake as a cloud data platform, we found that its ability to perform “zero copy cloning” proved immensely valuable for TuneCore’s Dev > Prod > Deployment environments. It allowed them to make rapid changes to their different environments without having to copy everything over and duplicate storage with traditional data warehousing paradigms.

For the data transformation side of things with Matillion, InterWorks utilized Matillion’s CDC feature – a process that also allows for incremental data refreshes using source system logs – which reduces query load on the source system.

The InterWorks team also helped with TuneCore’s polymorphism data modeling challenges and created a workflow that would get TuneCore’s data in a manageable format for their analytics layer. While this might seem like a small component of the overall project, it was one of the key differentiators that made TuneCore’s challenge so unique, and InterWorks’ ability to work through it with them was a huge benefit to making the overall stack work well.

Once the data architecture phase was complete, nightly refresh times had been reduced from nine hours to 30 minutes.

“Every company should understand its limitations. We could have grown our team to achieve this in house, but that would have cost more and taken more time. In the interest of getting things done the right way and in a timely manner, engaging a consulting company like InterWorks was the right choice because they’re laser-focused on delivering that project.”

– Raj Sarma, Director of Data Analytics, TuneCore

Once all the foundational data architecture work was done, InterWorks Solutions Lead Karl Riddett came in to assist with Tableau dashboard planning. Karl immediately brought in InterWorks Analytics Consultant Danny Steinmetz to build the dashboards because of Danny’s unique experience as a consultant and as a musician. The process for Danny was easy and straightforward thanks to data architecture work done beforehand. Without having to worry about data prep, he could focus more on building dashboards and determining designs alongside Raj and his team. After gathering requirements with various stakeholders, Danny was able to build two baseline dashboards in a matter of days.

One dashboard focused on content review of music recordings, allowing TuneCore staff to easily check for copyright violations and other matters with far greater speed and efficiency. This dashboard allows them to see genres, languages and countries in a much more streamlined view. The second dashboard was created for the TuneCore Marketing Team to give them a better view of new client inflow and how that compares across different regions. Compared to their old QlikView dashboards, these Tableau dashboards gave them deeper insights while also being more user friendly.

“The biggest value from InterWorks was their ability to come in and hit the ground running with their business and technical expertise. They were able to understand our business very quickly and uncover a lot of hidden gems within our data. We feel really confident in what they’ve delivered to us.”

– Raj Sarma, Director of Data Analytics, TuneCore

The Sweet Sound of Modern Data Analytics

After working through the proof of concept over the course of five months, it was clear that the combination of Snowflake, Matillion and Tableau would be an excellent fit for TuneCore’s needs. More than that, it proved that this was a data and analytics stack that could scale with them in perfect lockstep as they continue to grow and bring in more data. For an independent company like TuneCore to compete at a high level in the music industry, it’s important to find solutions that are powerful yet cost intelligent. That’s exactly why the proof of concept from InterWorks was so valuable. It allowed TuneCore to discover whether this stack was right for their data and the people working with that data, without the pressure of a massive upfront investment.

The proof of concept didn’t just make believers of Raj and the TuneCore Data Analytics Team – it also impressed TuneCore leadership, as well as the other groups that interacted with the two dashboards built by the InterWorks Analytics Team. With those litmus tests passed, TuneCore pulled the trigger and went all in on the Snowflake, Matillion and Tableau stack in early 2021. They’ve officially rolled out the new data architecture and continue to iterate on several new dashboards.

“Knowing that we can now crunch the data and perform useful analyses, the Data Analytics Team is really invigorated. We’re at the point now where we’re telling teams across TuneCore that we’re opening shop and ready to work with them at a deeper level.”

– Raj Sarma, Director of Data Analytics, TuneCore

Beyond the technology itself, TuneCore also learned a lot about what kind of people they should add to their Data Analytics Team as they continue to grow. Working with InterWorks consultants gave them more clarity as to what roles and skills they really needed within their team, and they have since added several new team members with plans to add more in the near future.

So, what’s the bottom line on this new data architecture and analytics environment for TuneCore? Within the organization itself, TuneCore is now able to access insights faster, more easily and with more confidence than ever before. By eliminating manual processes and implementing scalable solutions, they’re able to think more proactively on new sources of insight vs. spending all their time on query troubleshooting or data pipeline maintenance. This means that the Data Analytics Team is able to help more groups at TuneCore in more meaningful ways. With clear and ample insights at their fingertips, those groups are more empowered to help TuneCore grow and better serve their artists. In an industry where every stream on every platform matters, that kind of insight is what amplifies success.

“Data is what drives artists’ understanding of their performance on digital platforms, and they will do better if they have tools that are engaging and intuitive. We want artists to be artists, not data analysts, so we’re constantly striving to make their data easier to interact with.”

– Raj Sarma, Director of Data Analytics, TuneCore