While there is no crystal ball that can predict exactly where the data industry will go in 2025, or which tools or practices will make the largest impact, we at InterWorks are uniquely positioned to help our customers take the next best step on the path with their data. Based on our wide-ranging experiences from last year, in addition to what we’ve seen so far this year, I’ve provided some data “ins and outs” below that span several categories. I hope you find them helpful.

Enablement

In: More advisory and help surrounding change management

Organisations need to adopt a strategic, multi-faceted approach that includes establishing a dedicated change management process with clear roles and cross-functional representation. They should also develop comprehensive and role-specific training programs to build necessary skills, while implementing robust communication plans. Engaging stakeholders through thorough analysis and involving them in planning fosters ownership and reduces resistance, while leveraging collaboration tools enhances process efficiency and tracking.

Continuous support mechanisms, such as Assist by InterWorks and regular check-ins, along with utilising InterWorks’ expertise to further strengthen change initiatives, can be a significant help. By integrating these strategies, organisations can achieve smoother transitions, higher employee engagement, sustainable change and a resilient organisational culture poised for long-term success.

Out: Large classes with generic tool/platform focus

Traditional training methods, often characterised by generic and technology-focused content delivered in large classroom settings, are increasingly ineffective in enhancing data literacy within organisations. Such approaches fail to address the specific needs of diverse roles within an organisation, leading to disengagement and lack of ownership among employees. By focusing on personalised and comprehensive training plans and robust communication, organisations can enhance data literacy more effectively than through traditional means.

Architecture

In: Better Interoperability via Iceberg, Delta Lake and Medallion Architecture

Improving interoperability in data architecture can be effectively achieved by leveraging Apache Iceberg, Delta Lake and the Medallion Architecture. Apache Iceberg provides an open table format that supports multiple processing engines, enabling seamless data integration and schema evolution across platforms. Delta Lake enhances reliability with ACID transactions as well as unified batch and streaming data processing. It is particularly well-integrated with Apache Spark. The Medallion Architecture structures data into Bronze, Silver and Gold layers, promoting clear separation, standardisation and scalability. Leveraging these technologies facilitates continuous integration and deployment, reduces complexity and increases operational efficiency.

Out: Default configurations, neglect of best practices and underutilisation

Organisations must address challenges such as integration complexity, boilerplate security, potential performance overheads, the need for skilled teams and balancing open standards with proprietary features.

Engineering

In: IaC data in Terraform to manage infrastructure through code

Implementing Infrastructure as Code (IaC) with Terraform significantly enhances engineering practices in analytics by enabling the automated, consistent and scalable provisioning of cloud infrastructure essential for large-scale data operations. Terraform allows analytics teams to define and manage infrastructure through declarative configuration files, ensuring reproducibility and version control across development, staging and production environments. Its modular architecture promotes reuse and standardisation, while seamless integration with major cloud providers supports multi-cloud strategies and optimised resource utilisation. Despite challenges like the learning curve and state file management, the benefits of increased efficiency, consistency and scalability make Terraform an invaluable tool for building resilient and adaptable analytics infrastructures that support evolving business needs

Out: Siloed and inconsistent documentation

Poor documentation significantly impacts engineering efforts in analytics deployments by creating barriers to communication and understanding among team members. When documentation is fragmented and lacks standardisation, it becomes difficult for engineers to access necessary information efficiently, leading to delays and errors in analytics projects. This inconsistency can lead to inefficiencies in infrastructure management, increased risk of errors or misconfigurations, and challenges in maintaining a coherent and scalable analytics architecture. Moreover, the lack of a unified documentation approach hampers knowledge-sharing, hinders onboarding of new team members and can lead to repeated work or duplication of efforts

Analytics

In: Continued migrations to cloud for Tableau and Power BI

Transitioning Tableau and Power BI to cloud-based systems provides organisations with greater flexibility for delivery locations, potentially enhanced performance and scalability, plus improved collaboration capabilities. There’s a cost to migration, but the benefit outweighs this due to reduced cost in purchasing and managing infrastructure by an IT team.

Out: Over-hyped search analytics

This is a complex area as it’s very early days in search-based analytics. The technologies are constantly evolving at an ever-increasing pace. Current LLM tech requires us to keep refining our questions to deliver a sensible answer and even then it an hallucinate and deliver a non-sensical response. Here are a few other thoughts on this:

• Functionality: While NLS has made significant strides, accurately interpreting and responding to highly complex or ambiguous queries remains a challenge. Ongoing advancements in NLP are essential to bridge this gap.
• Data Integration and Quality: Effective NLS relies on high-quality, well-integrated data sources. Ensuring data consistency, accuracy and accessibility across disparate systems is crucial for reliable NLS performance.
• User Trust and Adoption: Building trust in NLS systems requires demonstrating reliability and accuracy. Organisations must address concerns related to data privacy, security and the potential for misinterpretation of queries.
• Scalability: As data volumes grow, NLS platforms must scale to handle increased query loads and data complexity without compromising performance or response times. Training larger models is becoming cost prohibitive, both monetarily and environmentally.

Governance

In: Optimisation of key core workloads

By transforming data analytics into a well-governed ecosystem, organisations can improve data accuracy, foster innovation and drive strategic business outcomes while minimising risks and inefficiencies. Key self-service data models empower businesses by enabling employees across various departments to access, analyse and visualise data independently without relying heavily on IT or data specialists. However, successful implementation requires robust data governance to ensure data quality, security and consistency, as well as comprehensive training to equip users with the necessary skills and tools. AI and ML are still untested but very interesting. There’s more work to be done here, so watch this space.

Out: Wild West self-service models

The ungoverned realm of data analytics, often likened to a “wild west,” presents a multifaceted challenge that organisations must face head on to harness the full potential of data-driven decision-making. In the absence of stringent governance frameworks, data analytics environments can become chaotic, leading to disparate data sources, inconsistent practices and varying levels of data quality, which undermine the reliability and value of insights generated.

Addressing these challenges requires organisations to establish a comprehensive data governance framework that encompasses data quality, security and privacy protocols. This includes setting clear policies for data management, establishing roles and responsibilities for data stewardship, and implementing tools and technologies that support data governance goals. Moreover, fostering a culture of data literacy is crucial, ensuring that all stakeholders understand the importance of data governance and the role they play in maintaining data integrity. Training programs should be tailored to enhance the skills necessary for data handling, analysis and compliance, empowering employees to utilise data responsibly and effectively.

Need Help with Your Next Step?

It’s one thing to talk about data trends and best practices, but it’s another thing to apply them. It might be that you have the resources and knowledge to make meaningful improvements; in which case, I hope the above the commentary is helpful in bolstering those efforts. However, if you find yourself wondering what your next best step should be, or if you know what needs to be done and simply need some extra hands, we’d love to help.

Let’s start a conversation about how we can help move things forward for your organisation.