Solving Vendor Lock in Using a Composable Data Infrastructure
Vendor Lock-in
Almost everyone in the #Technology #Consulting industry has heard the term “Vendor Lock-in” when engaging with a client. Companies are desperately trying to reduce costs in their #cloud and #software spend, especially when it comes to #data. Example scenarios that have frustrated business leaders could be: “…our system is too slow”, “…our licensing agreements are going up every year, we don’t have room in the budget,”“…we are locked into the vendor’s ecosystem, we can’t share data with other systems,” and many more. A typical response to these issues often starts with the existing data platform and engineering teams, they start to build “shadow” infrastructure to (hopefully) scope and solve some of the problems, often resulting in short term victories and long-term problems.
Convoluted Inefficient Architectures
Engineering teams often create work-arounds to solve $/tech problems having to do with integrations, storage, and reporting on existing data. They try to mix legacy (on-premises) systems with cloud native and third-party systems creating a “Single Pane of Glass” or “Single Source of Truth” to ease the frustration of not being able to get a handle on where the business is from a numbers perspective. Some business teams (Team A) use the tried-and-true legacy system to report out the current numbers; the other team (Team B) combines legacy and other systems to get a better picture of the relative business data. Often, the numbers from Team A and Team B don’t match. This causes confusion for the business leaders when trying to make decisions. Team A tries to match what Team B is reporting, resulting in systems that become overly complex and extremely hard to manage. This is typically when business leaders decide they need to rebuild or re-platform the system to meet the updated needs of the business.
Build, Buy and the Dreaded Vendor Lock In
I, and the team I lead, meet the client where they are in their data maturity, always. Do they build, buy, or build and buy? (PS- Typically, it is the latter, to ‘build’ what makes sense and to ‘buy’ where there are obvious advantages.) Building could be as easy as some functions and scripts to bridge a small technical gap or using the cloud vendor’s services landscape. Building involves using skilled technologists, which may or may not be in the clients existing team. If I build it for the client, they will need to supply the talent to maintain the system; whether it is building from scratch using programming, or talent to manage the cloud infrastructure put in place, these are expensive resources to procure depending on the scale of the project.
Buying, this is where many business and data leaders have a specific adverse reaction. They recognize that the system replacement is what they want to avoid in the future, and when proposing vendors and software they immediately bring up the “Vendor Lock In” argument. What the clients misunderstand is that the data architecture and software landscape has evolved into a composable paradigm. There are a myriad of tools and architectures today to prevent “Vendor Lock In”. Having a distributed, composable framework of tools will help you avoid being locked in holistically to one monolithic system. Broken down, the essential blocks of a data platform really comes down to three basic things: 1. Extract and Load (Data Pipelines), 2. Transformations and 3. Reporting and Analytics. When looking at the amount of software available for each of these verticals, it becomes abundantly clear that many of them are interchangeable with competitors or open source. I believe this the best way to approach the complex nature of interconnected systems. This gives the client extreme flexibility, both from an architectural patterns and implementation of infrastructure perspective. If the client doesn’t want to manage a cloud infrastructure, there are plenty of Platform as a Service (PaaS) vendors to choose from. One platform can extract the data and deliver it to another platform. As the client matures and wants to cost optimize, instead of landing the data directly into a PaaS system, they are ready to create a data lake backed by cloud storage. The PaaS doing the extraction can now be switched to a data lake. This is what creating a Composable Data Infrastructure looks like and also allows the client to exert some control over the vendors as far as licensing costs. Changing from one tool to another may incur some re-engineering costs, however, the client now has flexibility on many of the components in their data eco-system. Some may argue that now the client must engage with many software providers and vendors, but it is the price of breaking free from a monolithic lock in.
Before engaging in a journey creating a Composable Data Infrastructure, the client will need to do a deep dive on the pros and cons of the current state. Is your company trying to drive greater efficiencies by reducing cost? Does the current system give your company what it needs, cost being the main driving factor for considering re-platforming? Does the current system provide 90% of what is needed and the purpose of looking for alternatives to supplement the existing system because of licensing or flexibility? There are lots of questions that need to be asked; evolv Consulting has experts that will answer those questions and implement the best solution, meeting you where you are.
