This blog series unpacks everything from best practices to best philosophies when it comes to delivering the best analytics experiences.
We are beginning to enter a golden age for business intelligence. We have been steadily moving in this direction for the 10+ years I have been involved in the industry, but until now, there always seemed to be some blocker. The volume of data, the many places it lived, the niche skillsets required to use the available tools and sometimes the cost meant data could be difficult and slow to access.
Sifting Through for the Right Tools
We’ve seen an emergence of a whole range of new tools, many both cloud-based and easy to scale. There is now simpler integration and communication between tools. Solutions allow for the incorporation of AI or ML (machine learning) natively into a product to add greater depth. Further, the introduction of new entry points like Natural Language Processing (NLP) and ready access to beautiful front-end dashboarding tools should be make us excited for the possibilities of today’s data analytics.
The combinations of tools are seemingly endless, so I wanted to share some of our team’s favourites. I interviewed a few of our leaders to get a sense of their perfect end-to-end solutions. Before I list them, it is worth saying there were numerous caveats given and if I asked this same group tomorrow, I might get a different list! Like all good consultants, they inevitably started with the phrase, “Well, it depends.”
Please take this for what it is intended to be: a mix of possibilities and starting points for your further research. Or skip all that and come and chat to InterWorks directly!
The Team’s Go-to Solutions
Jim Horbury – Global Solutions Practice Director
Where better to start than with the guy who regularly puts together menus of tools for our clients to sample than our very own Jim. He couldn’t be pinned down on a single solution but wanted to share a few he has seen work effectively together:
Basic Starting Points
- Extraction/Load: Fivetran
- Warehousing: Snowflake
- Data Transformation: dbt
- Analytics: Tableau
- Front End: Curator by InterWorks
Ad Hoc and User-Driven Analytics
- Extraction/Load: Matillion
- Warehousing: Snowflake
- Data Transformation: Matillion
- Analytics / Ad Hoc / Self-Service: ThoughtSpot
Complex, Hard-to-Access Data with No Analytical Requirement
- Data Sources: Unknown / Custom
- Extraction/Load: AWS Lambda
- Warehousing: Snowflake
- Data Transformation: Snowflake
- Front End: Web App
Mike Dunning – Data Lead
Mike outlined a number of great tooling options but, when pressed, narrowed his choice to an entirely cloud-based combo:
- Extraction/Load: Fivetran
- Warehousing: Snowflake
- Data Transformation: dbt Cloud
- Analytics: Looker
Raphael Teufel – Solutions Architect
Raph was keen to stress that whilst the tooling is important, so is the hosting platform and the authentication. His experience has led him to favour GCP (Google) and OKTA for his selection. He has seen the huge importance many clients place on a self-service setup, so he was keen to point out Tableau’s Ask Data to handle this:
- Extraction/Load: Matillion
- Warehousing: Snowflake
- Data Transformation: Dataiku
- Analytics: Tableau (Ask Data)
- Front End: Curator by InterWorks
Chris Hastie – Data Engineer
Chris felt that the main choices are driven by the skillsets available within the organisation. His distinction therefore went towards a positive UI option with lots of drag and drop or a much more code-based solution. He was also keen for Azure-based hosting in general:
UI Preference
- Extraction/Load: Matillion
- Warehousing: Snowflake
- Data Transformation: Matillion
- Analytics: Tableau
Code-Based
- Extraction/Load: Fivetran
- Warehousing: Snowflake
- Data Transformation: Snowflake
- Analytics: Tableau
Chaitanya Joshi – Data Engineer
CJ was keen to explain a prominent Google option and its merits:
- Extraction/Load: Matillion
- Warehousing: Google Big Query + Omni
- Data Transformation: Matillion + Google Vertex AI
- Analytics: Looker
Kevin Pemberton – Regional Director, EMEA
Kev felt when there was an option for cloud-based tools, this route should be favoured. However, reality says this is not always possible, and some of the more well-known and established tools still have a place for an on-prem setup:
Cloud
- Extraction/Load: Matillion
- Warehousing: Snowflake
- Data Transformation: Dataiku
- Analytics: Tableau
- Front End: Internal: Curator by InterWorks ; External: Google Search
On Premises
- Extraction/Load: SSIS / Python
- Warehousing: SQL Server
- Data Transformation: SQL
- Analytics: Tableau
It would probably be a little wrong to shy away from offering my own opinion here:
- Extraction/Load: Dataiku
- Warehousing: Snowflake
- Data Transformation: Dataiku
- Analytics: Tableau
- Front End: Curator by InterWorks
Find the Best Solution for Your Unique Needs
I loved hearing about the variation of choice, and the more interviews I ran, the more it reinforced that there is definitely no “one size fits all.” There were some strong common themes around Snowflake as the core warehousing solution, and Tableau is still the firm favourite analytics tool, mixed in with InterWorks’ very own Curator for the end user entry point.
It is safe to say that business intelligence professionals have an incredible menu of choices available, and I wish you all the best in making your own a la carte selection. Please reach out if you need help. We’d love to talk through the options and strategise about the best plan for your data needs.
