Technologies Used:
Services Used:
Mining companies understand the importance of efficiencies scale. A common operational objective among the industry is to streamline as many processes as possible because even small efficiencies add up to massive savings in the end. Take iron ore for example: When you mine, process and ship millions of tons each year, shaving even a few cents off your operational costs can result in millions of dollars saved. In that sense, operational efficiency is truly a competitive edge.
In the case of this anonymous mining company with global reach, one strategy they’ve employed with great success is that of vertical integration. By owning the means of extraction, processing and shipment, they’ve been able to cut costs dramatically. Not wanting to rest on their laurels, they recognized another way to optimize operations was by empowering their decision-makers with faster access to reliable data. In order to provide that to their people, they needed the help of a trusted guide.
Trial and Error with Tableau Consulting
The mining company is one that, once set in a direction, will analyze every possibility to make the best decision they can – so it was in choosing an analytics platform. Knowing full well that their choice of analytics platform would ultimately guide the direction of their entire data environment, they did not make the decision lightly. In the end, they landed on Tableau as the best fit for enabling modern, user-driven analytics.
After purchasing Tableau, the mining company immediately reached out to a large analytics consulting firm to help them first implement Tableau and then help them grow in their use of the platform. Progress crept along, and this consulting firm soon became entrenched within the mining company. Seeking a change, they turned to their regional Tableau account manager to see if there was a different consulting firm that could take them further in their efforts. Knowing just the firm for the job, the account manager referred them to InterWorks. That’s where our relationship begins.
The Proving Period with InterWorks
Before going all in on InterWorks, the mining company first wanted to see what we could do. This began with InterWorks-led Tableau Combo training courses across the company in addition to some exploratory dashboarding work. The training courses went swimmingly, and the dashboarding work was illuminating, but they kept coming back to the fact that InterWorks, at least at the time, didn’t have a large local presence in their area. Some consulting firms might be set in their ways, but InterWorks took it to heart, going so far as to add additional qualified resources local to the area.
Thanks to extensive experience in the mining industry, our new people were an immediate fit with the mining company. The InterWorks team started out doing small projects such as ad hoc dashboarding as the mining company needed it. Eventually, all these successful small projects with InterWorks showed the mining company a pattern – InterWorks was who they said they were. With firm trust established, even more meaningful work could begin.
Scoping the Big Project
Now that the mining company was familiar with InterWorks and the broad capabilities our team possessed, they wanted the same level of holistic work InterWorks was known for globally. The thought of implementing the same modern analytics technologies and principles as the some of the world’s most sophisticated companies had the mining company excited about their own potential to apply these best-in-breed solutions to their organization as well as pioneer them for the broader mining industry.
Scoping Requirements
- Unification of disparate data sources
- Ability to handle large volumes
- Clean, reliable data for analysis
Knowing InterWorks offered full-stack data solutions beyond visual analytics, the mining company turned to InterWorks for strategic advice on what that ideal framework might look like for them. Tableau was clearly a big part of that framework, but they also wanted more impactful ways to take their disparate data sources, store them in a secure and easily accessible place, and then prepare that data in a way that would be readily consumable by Tableau. It was then that InterWorks recommended Snowflake and Matillion.
As a cloud-based data warehouse built for speed and convenience, Snowflake was an immediately attractive solution. Its ability to handle staggeringly large volumes of data while also being highly performant checked two big boxes for the mining company. Matillion would serve an equally important role, enabling them to perform robust ETL actions with a strong focus on automation.
With the appropriate pieces identified, all that was left to tie everything together into one cohesive POC. Leaning on our strong triumvirate of partnerships with Tableau, Snowflake and Matillion, we were able to map out the POC with ease, identifying the value each piece would bring individually as well as the collective benefits these technologies would bring once paired.
Breaking Down the POC: Tableau + Snowflake + Matillion
Specifically, the mining company wanted to apply data analytics to the complex and fast-moving elements of their supply chain, looking for opportunities to enable its workforce to analyze data and for opportunities to improve performance. The end goal was to launch a Centre of Excellence (CoE) to support the many analysts across the organization who needed a centralized and trusted repository of near real-time data analytics.
Before any of that became a possibility, the mining company needed capable architecture in place to enable that speed to insight. They also needed to rollout that architecture where it could make the most immediate impact, namely through the organization’s ERP, Manufacturing Execution Systems (MES) and mobile/fixed plant operations. The goal of this architecture would be to fulfill strategies surrounding conventional overnight/batch-based data processing, on-demand/push-based data processing and near-real-time data ingestion. With those priorities in mind, Snowflake, Matillion and Tableau would shine even further as part of their POC.
Snowflake
The foundation of this architecture would be a persistent data staging repository, integration layer utilizing Data Vault methodology and presentation layer for self-service discovery – all modeled and delivered rapidly via Snowflake. The data repository would be built incrementally from the 100+ data sources used within the organization. Internal and external security were additional focal points during the rollout of Snowflake. The mining company wanted to establish a security model encompassing Privatelink and two-factor authentication as well as external security audit and compliance. They also sought to establish models for ringfencing and governance within their internal data science and analytics communities. Looking to the future, a built-in provision for data sharing using a combination of Snowflake and secure shared portals would greatly benefit external vendors and joint ventures. Throughout this process, InterWorks would also advise on all performance best practices.
Matillion
Matillion would play a crucial role as a high-availability ELT platform capable of managing consumption of large and rapidly changing data volumes. InterWorks data engineers would assist the mining company with the loading and integration of the 100+ data sources in use. They would also develop templates and documentation to support trained personnel within the organization, providing training and mentorship along the way. Once established, Matillion would enable automation services to monitor, address and/or notify the mining company in the event of system downtime or arising data quality issues. Platform support and upkeep would be managed by InterWorks until the mining company is ready to take over.
Tableau
At the front end of the mining company’s analytics ecosystem would be Tableau Desktop and Tableau Server. First, InterWorks would help them establish a high-availability Tableau Server instance with 24×365 support via our ServerCare offering. InterWorks would also handle the administration of any content migration activities. In terms of visualization, InterWorks would provide ongoing consultancy in the creation, support and delivery of Tableau content. This would primarily be achieved through Tableau training and enablement services for the mining company’s internal Tableau community, as well as visual analysis and performance best practice support. The primary goal of these activities would be to establish excellent business analysis across multiple groups while building stakeholder engagement wherever possible.
From POC to New Data Opportunities
So, how did this ambitious POC go? Thanks to a clear and collaborative vision set by the mining company and InterWorks, it went off exceedingly well. Leveraging our multi-disciplined team of local and global resources, InterWorks was able to find the perfect people to deliver each component of the POC and even dive deeper into new sub-projects.
On the data architecture and engineering side, the InterWorks data engineers worked tirelessly to leverage Snowflake and Matillion to dig deep into the mining company’s data and ensure the establishment of a solid pipeline to Tableau. Building on this success, the InterWorks team was able to shift to producing similar results for the mining company’s safety, finance and HR data. While they delivered the POC and various sub-projects to spec, the InterWorks team also trained mining company staff to be able to carry the torch and continue their success once the initial framework was provided.
Immediate Benefits
- Reliable data pipeline
- Greatly improved speed to data
- Empowered user base
- Millions of dollars saved
An impressive example of how Snowflake provided immediate value is its application to a supply chain constraint with a single point of failure. When this part of the chain goes down, it cuts overall production by 33%. For this mining company, time is literally money. In 2018, the lost gross revenue from these outages totaled $3,000,000. A data scientist was tasked with investigating, but he couldn’t source the required data for his machine learning model. The dataset he needed was also too big for his software to process. InterWorks had a solution. Using AWS S3 and Snowflake to stage the dataset, InterWorks was able to take them from source to analysis in two hours. The mining company data scientist originally asked for 36 tags of information but InterWorks provided them with 200 tags at two-second intervals for a year’s worth of data. The result of this project was that the predictive model for outages went from about 50% accuracy to 99.6% accuracy with the added ability to predict failures six hours in advance. That kind of preventative analysis results in millions of dollars saved each year.
Equally impressive successes were made on the dashboarding and enablement fronts. A notable example was InterWorks’ work with an inherited executive dashboard. One of our analytics consultants sifted through and streamlined 52 pages of custom SQL, 200 Tableau worksheets and 1500 calculations, taking workbook load time from five minutes to 30 seconds. Even more InterWorks team members from all over the globe delivered crucial training sessions across a wide variety of cohorts, ranging from executives to data science citizens and analysts. Some Tableau training courses were even conducted at mine sites in full safety gear (after all, safety is a priority). Wherever mining company staff were, InterWorks was there alongside them to provide the training they needed to adopt Tableau more effectively.
Moving forward, the plan is to expand Tableau training to even more cohorts and further establish the internal CoE that the mining company desired from the outset, eventually enabling their organization to conduct their own trainings and produce their own resources. Beyond training, InterWorks will continue to provide strategic advisory for Snowflake, Matillion and any other data architecture and engineering initiatives. But as more mining company staff members are trained and a culture of self-service continues to take deeper root, the mining company can work the magic entirely on their own. In our view, that is the biggest success of all in this relationship.
