Have you ever noticed that when you complete a trip with Uber, your account is updated immediately? I often interact with apps on my phone only to be left confused by how they can stream my data between their source systems and my phone so seamlessly. Interacting with Uber as a consumer showcases the extraordinary engineering these companies use to solve data problems. While these final products are intuitive and simple, they are powered by exorbitant amounts of data, which leaves data engineers like me wondering: “How do they do it, and how can I do it for our clients?”
When we first heard about Data Council and saw that companies like Slack, Uber, Lyft and Netflix were presenting, I had a hunch my questions were going to be answered.
What Is the Data Council Conference?
In April, Dylan Bergey, Brenden Goetz and I traveled to San Francisco to attend the 2019 Data Council Conference. Data Council’s main goal is to create a global community of data professionals, and this week embodied that effort completely. Data Council shares a lot of the same values as InterWorks, and as data engineers, it was invaluable to spend a week listening to our peers discuss how data engineering is changing at organizations that define the problems we solve.
While most enterprises are starting to gather data at massive volume, it is obvious that some of the technology organizations in the Bay Area are running into issues never seen before, and as solutions emerge, InterWorks is afforded the opportunity to learn what’s changing in our field and how that change relates to our clients.
The week was filled with speakers from a variety of industries that highlighted how data is changing the way they work. The conference was organized around numerous tracks you could follow. Day one gave us the opportunity to choose to learn about Data Platforms & Pipelines; Machine & Deep Learning; or Hero Engineering. For day one, I chose to follow the Data Platforms & Pipelines track, getting to hear discussions from Netflix, Lyft, Astronomer and more about Data Discovery, ETL and all things data.
Day One: Data Platforms and Pipelines
As day one kicked off, I noticed how our speakers’ approach to data engineering seemed different from our clients’ typical approach. Listening to the style of projects these large tech companies pursue, there was one common theme: open source.
Open-source technology is an exciting space. The concept of using community-driven software to power your analytics workloads provides flexibility and allows you to give back through contributions and support of the system. However, for a lot of our clients, an open-source solution doesn’t make sense. At InterWorks, we aim to provide our clients with a comprehensive solution that can be utilized and iterated upon after we leave.
Exploring Open Source
There are a lot of benefits to open-source projects, such as almost always having the lowest up-front investment, but the skillset required to develop and maintain these platforms can quickly evolve beyond what is available to most companies. You don’t call in InterWorks to build something you’re afraid to touch once we leave.
As a result of the complexity created by these open-source tools, we often suggest that clients without the ability to manage them explore options with a fully managed solution. These tend to be easier to maintain and hand over, as well as give us the opportunity to provide value to the user and the business simultaneously. In the end, a solutions goal looks wildly different depending on what the client needs, but we want to provide a solution that makes life easier, not harder.
Day Two: Data Engineering Roles
The second day’s options consisted of Databases & Tools’ Data Analytics and Building Data Products. I hopped around tracks quite a bit and was able to hear talks about data from nearly every perspective available to a business. We started the day listening to a panel discuss the importance of security in the world of data. The speakers did an amazing job unpacking how technical advancements have moved at a quicker pace than regulation, and in order to continue rapid innovation, we need to learn to work with the regulators more effectively.
As the day progressed, we heard presentations from Looker, Maxime Beauchemin, Slack and Stitch Fix. Slack’s discussion about project management for data projects highlighted the importance of data engineering’s role to an organization. Just to provide some background on Slack’s presentation, they recently did a complete restructure for how their teams are supported by data engineers and are pushing that model across the business with great success.
This talk was particularly interesting to me because when working with clients, we are brought in to perform two roles: be the expert, and make the project succeed. Those two skills are crucial to our success as a consultancy, and learning how Slack successfully manages data projects shed some light on how we can continue to improve in that area.
Staying Current in the Data Industry
Attending conferences like Data Council is what keeps InterWorks in tune with the industry and what our clients need. On any given day, we’re sent all over the world to solve data problems for clients who are doing amazing things with their data. Being able to go and hear about what is on the horizon technically and professionally is crucial to our ability to serve clients effectively. The conference delivered and definitely offered up a few things we can grasp onto that will drive us to be better consultants who engineer solutions our clients love.
