Happy New Year to all the readers of the InterWorks blog; 2018 has been a whirlwind of a year! In 2018, InterWorks steadily provided value to our clients in the data space, and as we kick off 2019, it seemed appropriate to prep you guys on what we have up our sleeve in the new year.
The theme for 2018 was a different perspective on our core principles: Best People, Best Work, Best Clients. Instead of just looking at these phrases at base level, we dove in and identified how this philosophy differentiates us from other consultancies and began aspiring to be a Small Giant. Moving into 2019, being a small giant means a few things for our data practice.
2018 Recap in Data
In 2018, we continued working extensively with data that is generated in the cloud. As these opportunities emerged, we mastered cloud-native technology to wrangle and tangle this data into fast and available reporting solutions for our clients. Partnerships with companies like AWS, Snowflake, Matillion and Fivetran naturally emerged as our expertise and excitement around their tools took off. We were the Snowflake Managed Services Partner of the Year for 2018 and continued to perform exceptionally in the Tableau partner ecosystem.
We learned not only how to create reporting solutions surrounding underlying data with Snowflake but also how to create intuitive pipelines and automation techniques with AWS products, Matillion and Fivetran. The results derived from our team of experts across each technology are unmatched, and we quickly became known as the trusted partner in data. Our data practice is at a very exciting place, and we are preparing to make this the best year yet.
Above: Some members of Team Hood at our team meetup last August in Denver.
Winning the Year of the Pig
As we move into 2019, our goals as a company, and in our data practice, are clear: be the best, and work with the best. As a practice, we are starting the year with a few focal points:
- Continue to be at the cutting edge of Snowflake’s new features and tools
- Find ways for clients to maximize efficient, reliable and idempotent ETL processes with cloud-based tools
- Continue mastering enterprise-modeling techniques for consistent and reliable reporting
Snowflake’s Growing Presence
Snowflake has been a huge success story as a reporting layer for some time now. The performance is undeniable, and the cost structure is friendly to businesses of every shape and size. Some feedback we heard about Snowflake last year was a lack of features SQL Server gurus are used to. So far in 2019, we are already starting to see Snowflake roll out new features like stored procedures and materialized views, and as the year progresses, more will certainly come.
In addition to new features, Snowflake is beginning to vocalize the importance of clustering micro partitions for performance on large datasets and is introducing a proctored certification exam. The things we know and love about Snowflake will remain, but as we move into the new year, we will continue to be the experts on utilizing new features. A few Snowflake blogs to be looking for in 2019:
- Javascript Stored Procedures
- Clustering Micro Partitions
The Ever-Changing ETL
Extract, transform, load. Rinse and repeat. The age-old process seems to be changing by the minute; each tool offers a different benefit, and getting the most out of that investment can be difficult. Centralizing data reliably in a warehouse is the goal, and building a reliable pipeline is crucial to its success. By definition, a data warehouse is designed to host data from dozens—if not hundreds—of source systems. In 2018, we saw a large push to the cloud from many CRM and ERP providers, and with the data being generated in the cloud, it makes sense that it is most efficiently migrated to a warehouse using cloud tools. The ETL process is evolving.
Organically Formed Partnerships
In 2018, we formed partnerships with companies where we genuinely enjoyed the technology. Fivetran offers an incredible user experience and allows businesses to replicate data from a warehouse. Fivetran does not have transformation tools yet, but its replication process is fast, reliable, and easy to use.
Photo above courtesy of Fivetran
On the other end of the spectrum, we have Matillion: a development-friendly tool utilizing components to allow you to create a bespoke pipeline that centralizes and transforms data as it lands in your warehouse. One big piece of ETL in 2018 was the need for an efficient way to ingest and consume semi-structured data, and that will continue to be a key factor in maximizing the value of a data warehouse in the new year.
The most interesting piece of these cloud based ETL/ELT tools is the ability to constantly adapt to what is going on in the industry. With the cloud landscape exploding in the new year, this is a crucial piece that allows reporting systems to be agile and cost effective.
Modeling Techniques
When it comes to modeling, there is one technique that is quickly gaining buzz around the InterWorks office: the data vault. The way businesses use and collect data has been on an exponential curve for quite some time now, and with the implementation of agile methodologies in corporations across the world, we are seeing the need for reporting systems to be just as flexible.
Working with Data Vault is something that will be an exciting part of 2019. Adapting to change is crucial to have a reliable and consistent reporting environment. With clients adding new software to run different pieces of their businesses like never before, this approach to mapping events is poised to be huge for the success of data warehousing.
Tackling These Data Goals in 2019
In this new year, we are thrilled to continue studying technology that we love in order to solve the data problems of the world. Let’s make this the best year yet!
