This blog series unpacks everything from best practices to best philosophies when it comes to delivering the best analytics experiences.
In this post, we’re launching a new blog series that will take a closer look at what makes good analytics. We’ll attempt to home in on various facets of the analytics experience, from the perspectives of developers and end-users alike, and walk through tested methods and approaches to help you create efficient, sensible and rewarding analytics solutions. To start things off, we’ll be focusing on scalability.
Finding the Scalability Balance
A sculptor commissioned to create a marble statue will first create a small-scale model in clay. The clay model may help the sculptor’s client understand what is to be created, but it will form no part of the final, full-scale statue. The whole thing has to be remade from scratch in its final form. Once carving of the final version has begun, it can’t then be altered.
An artist painting in oils has more flexibility. While she might produce sketches that are separate from the final painting, the painting itself can be altered many times before it’s complete. X-ray studies of many famous paintings show alterations with limbs in different positions, buildings changed or background details added or removed. While this pre-work isn’t visible, it indicates an iterative, evolutionary approach where the final painting will contain elements of the original sketches, refined and completed.
A development web server or database running on a single computer is comparable to the sculptor’s clay model. It will work at a small scale to demonstrate functionality with a few test users, but it won’t be able to cope with the much greater level of traffic when it’s put into production. Re-engineering these kinds of small-scale systems is not simply a case of upgrading the hardware. To support much larger loads, software components will need to be adapted or replaced, hardware upgraded and new features added to support load-balancing and redundancy.
If your system doesn’t have scalability built in, when you move to production, you will have to estimate the likely maximum load (even though you don’t have any history of actual usage to use as a basis for an estimate). Estimate too low, and your users will experience connection problems and poor performance. Estimate too high, and you will have expensive hardware sitting there, underused.
The Beauty of Scalable Systems
Above: Ophelia by John William Waterhouse, public domain via Wikimedia Commons
If you have an idea that may improve your company’s bottomline, a truly scalable infrastructure has powerful advantages. The cost of early experimentation is low, and you can move quickly to test new ideas and then iterate to improve them. You can try out many ideas on a small scale, discarding those that don’t work and moving those ideas that prove fruitful into production with no need to re-engineer them.
New cloud-based and serverless computing technology has come a very long way in solving the scalability problem. Vastly improved scalability is one of the most common reasons that companies are moving computing load from their own on-prem server rooms into the cloud.
Today, if you use a cloud-based data platform such as Snowflake, you can implement your system on a small scale at low cost with no hardware to manage. Processing costs a few dollars per hour, and you only pay for the time that processors are actually running queries. When you decide you’re ready to go live, you can increase or decrease the available processing power enormously within a few seconds or have the system scale up or down automatically following moment-to-moment demand.
The Dangers of Scalable Systems
Above: Ophelia by John Everett Millais, public domain via Wikimedia Commons
Danger 1: You Don’t Continue to Sell the System
If you needed to commit to a large up-front expenditure, you would have spent quite some time commissioning feasibility studies, market research and technical proofs in order to build a solid business case. If you got your project approved, there would be plenty of awareness of it at higher levels in the organisation. Once implemented, there would be the political will to make full use of whatever you had delivered to realise the promised benefits.
If a lower initial commitment means you’ve not had to go through that exercise of selling the idea to stakeholders at all levels, it’s tempting to miss out on that step altogether. Many a start-up founder has rushed to create some revolutionary new product or service, and perhaps succeeded in technical terms, but then wondered why nobody seems interested in what they’ve done. Very few shiny new products, even the great ones, are so amazing that they simply sell themselves. If nobody knows about the thing you’ve made, it will fizzle and die.
Danger 2: Partial Scalability in the Operation
Even if your database or website can scale up or down quickly and easily, there may be other parts of your operation that aren’t so flexible. If creating new users requires manual processes or security checks, that part of your operation can quickly become a bottleneck. Don’t promise super-fast growth if you’re going to get held up by tasks that still have to be done by hand.
Danger 3: Your Competitors Can Do It, Too
Highly scalable systems can greatly accelerate the rate at which you can develop and implement new products and services. These same technologies can also be used by your competitors. A large company can no longer afford to ignore a much smaller competitor if that much smaller upstart could expand their customer base 100x or a 1000x within a few months.
A Focused Approach with an Expert Ally
If you’re not currently using scalable cloud technologies as a core part of your infrastructure, you’re stifling much of the potential for innovation that exists in your company. After a while, you may find that you’re struggling to compete against smaller, nimbler rivals.
The ability to try out and prove or disprove ideas on a small scale then quickly put them into production at a much larger scale is a terrific technological advance that shouldn’t be ignored. Instead of trying to make a broad system that appeals to a large set of stakeholders, focus tightly on a small number of use cases, get feedback from a tight group of potential users, iterate quickly and get into production early. You will still need to actively promote what you’ve done, but it’s much easier to do that when you have a live system that is generating results, and you can call upon references from satisfied users and sponsors.
InterWorks can help you every step of the way, from sculpting in clay to painting in oils. We offer services and support from data migration and infrastructure management to advanced analytics and machine learning. Connect with our team to share your analytics vision. We’d love to help bring it to life.
