BI System maintenance is like a teenager’s bedroom—you should anticipate some challenges to maintain organization and reliability. Without discipline, the health and efficacy of your system won’t last. Establishing best practices for monitoring and maintaining the system’s health, security and performance is an integral part of the evolution of your BI platform.
Everything works well early in your deployment. Establishing solid security and governance practices will get things off to a good start. Have your consulting partner offer suggestions to avoid costly missteps. The real test comes when your project completes and your team members return to their everyday roles. Your successful early deployment will generate much enthusiasm. As people discover how accessible, profound and responsive your BI stack has become, they will want more.
Your BI system content will expand. The volume and velocity of activity surrounding the system will increase. The number of people accessing it will explode. Trouble will come to your doorstep in 1-2 years if disciplined system maintenance schedules aren’t maintained. You may believe a software update should be “no problem” because the updates have never caused problems. Refrain from falling into this kind of complacency. Hedge your bets by contracting a consulting partner with experience deploying the update processes to maintain your system’s health.
It is common for clients I haven’t spoken with for years to call me on a Sunday morning because an update brought their system down. They don’t know how to fix the problem. They may have failed to adhere to best backup practices, or they failed to follow good policies. Panic ensues because if the system isn’t up on Monday, they will have thousands of unhappy people on their hands. Unfortunately, discovering that on a Sunday afternoon makes it challenging to address before Monday morning.
As the size and velocity of your data grow, regular monitoring of system health is crucial to high performance. Establish KPIs on system usage, referred to as metadata. Typical metrics include the following:
- Number of dashboards
- The most popular reports by usage duration and user count
- Frequency of usage
- Size of your data
- Amount of storage capacity being used
- Response times for queries
- Load speeds of reports
- Number of stagnant files
- Number of developers who aren’t using development tools
- Publishing frequency
Monitoring your network’s technical details, web access and contracted capacity for query compute power. How your security systems affect performance provides insight into overall system health. In this situation, the best scenario is a slow system performance degradation. The worst case is a complete system failure. In either case, your employees and executives will be annoyed.
Establish system monitoring and maintenance as part of your initial plans. Define and communicate specific KPIs and performance thresholds. Imagine your BI system is like an airport. Airports run according to a schedule that is updated continuously. Everyone knows system performance. That’s an excellent model for defining and communicating your BI System performance and maintenance.
Vendor Recommendations
Every reputable software vendor has system maintenance recommendations. Some offer services or tools to help you maintain their systems in peak condition.
Figure 3.5.1 references some of our favorite platforms and software for each portion of the data workflow. The picture is not an all-inclusive collection, so don’t be concerned if the software you are considering is not included. The intent is to provide a glimpse of the complexity that BI systems can evolve into. I’ve selected my favorite tools because they appeal to most of our clients. Other software tools may provide similar features and a better fit for your workflow.
Figure 3.5.1 Example Reference Architecture, Best-of-breed BI System
Finding the Right Consulting Experts
Another layer of expert knowledge exists within the consulting companies that deploy these systems. Many of them learn the hard way what the software vendors need to provide or what they need to consider. Expert consultants are an excellent source of “real-world” know-how. Some consulting partners may have special-purpose tools and services for monitoring system health. They will be more efficient and effective at doing the work because their staff continuously deals with it for many clients for a fee. It must be better.
Some companies start with the idea that they can maintain their systems. Still, 2 to 3 years after deployment, they are distracted by the urgent. They may need more capacity to attend to maintenance properly. Maintenance = plumbing work, which is unexciting. Discovery and development are more exciting tasks for your team and where proprietary business knowledge pays off.
Consulting partners know how to do system maintenance according to best practices. They have also developed processes to do it effectively and efficiently. The cost to engage an outside expert for this work will more than pay for itself. Consider this realistically. Recognize where you will want to apply your internal technical team’s proprietary knowledge and skill. Contract the “plumbing” work with consulting companies that can do it faster and better.
In Part IV, you will gain insight into ways to advance a data culture in your company. You can’t just buy data literacy; you must develop it. Literacy is a necessary precursor for building a culture that creates accurate data, harvests that data as valuable information, and utilizes information to make more informed decisions.
