Winning with Modern Data and Analytic Strategy: Part 1

Winning with Modern Data and Analytic Strategy

Part One – Current State, Vision, and Tactics

This is part one in a series of posts dedicated to data and analytics strategy, written by Relus Cloud senior leaders.

The capability to effectively ingest, process, store, manage, and use data for actionable insights is absolutely essential for any modern business to compete in the marketplace.

This really isn’t new; it’s been well-understood since before the dawn of business technology. The fundamental truth is that in order for any business to out-compete strategic opponents, it must first be able to observe the reality of external circumstances and the business environment, then process it quickly and accurately to develop insights and take appropriate action – and to do this more efficiently and effectively than competitors. [1]

To reframe this, our analytic cycle needs to become more efficient – and we must drive value in the form of relevant information, not just wrangle ever-increasing amounts of historical structured and unstructured data. The business paradigm shift is now overwhelmingly about real-time and in-memory streaming analytics based on an exploding volume of data.

Yet today, most companies are badly challenged with effectively managing and deriving value from data they already manage – at least in any competitive way. More than ten years ago, Gartner reported that over 50% of Enterprise Data Warehouse projects would fail to achieve user adoption, a key measure of the value delivered by such projects. This has certainly been borne out.

One of the reasons that this is true is that the volume, variety, and velocity of data has increased. Another is that the enterprise data management ecosystem has become rapidly more complex and difficult to understand. Still, another reason is the fact that established companies typically have years of technical debt in the form of legacy databases, legacy data warehouses and the like, which must still be governed and maintained.

But really, the business demands are shifting. The main challenge for technologists today is that - while we must still establish governance and controls, data integrity, security, privacy and while we must still establish ways to standardize data and reporting – we must also help our company to more rapidly identify and act on opportunities in real time. Those are the table stakes for an emerging business epoch based ever more on advanced analytics and hyper-personalization.

So, to sum up, we must continue to do everything we’ve done before to govern data and drive good decision-making from a single source of truth - but do it faster and better with less waste; and we must establish advanced analytics capabilities – such as predictive and prescriptive analytics - to derive business value from real-time and near-real-time data sources.

A Hard Look at our Current State

Is the current situation meeting our needs? For many companies Relus Cloud has consulted with, after all these years of investment in complex on-premise data warehouses and operational data stores, we are left with some hard realizations:

Top 10 Problems

  • We still have years of legacy technical debt in the form of old database and data warehouse platforms which, somehow, can never be retired and continue to consume more of our resources than they’re worth.
  • Proprietary data platforms have become a software licensing nightmare / prison, providing little additional value over open-source alternatives and not actually helping us solve our key problems.
  • We still have rogue, siloed data sets springing up outside of centralized data management and incorporated into ad-hoc business
  • We still have difficulty managing access to data. By some estimates, well over 70% of employees - even in regulated industries - have access to data they should 
  • Discovering and preparing data still takes too much time – and then we argue over the source of truth and disagree over the business rules used in its
  • We are challenged to effectively estimate, engineer, scale and maintain our platforms over their lifespan – not to mention execute change with any agility or
  • Data Sprawl and Data Fragmentation: Micro-service architectures generally require data stores to be owned by the micro-service itself, leading to architecture sprawl, another source of
  • Many teams lack the skills or available cycles to incorporate key enabling technologies or advanced analytic capabilities required for digital transformation
  • Data Latency Decreases Relevance – Less than 30% of a company’s historical data is actually used today. That’s because stale data may be of some use to spot trends, but business stakeholders today demand fresh insights based on streaming or near-real-time
  • Lack of User Adoption: After all the investment, 65% of companies still make limited use of centralized, structured data in data warehouses. 84% of established companies are unable to incorporate unstructured data sources. And very few enterprise data teams are able to integrate real-time streaming analytics.

That sure seems like a lot of corporate energy expenditure (in the form of time and money), for ever-decreasing value in what’s shaping up to be a data-driven arms race.

Future State Vision

If we were to define a future in which many of our current state problems were fixed, it might look a little like

  • We have a “bi-modal” enterprise data strategy and a modern data platform to support all aspects of our data and analytic value
  • We use policy, process, and technology to create a business-driven, virtuous cycle of innovation that will modernize our platform while finally eliminating technical
  • We have added advanced analytics technology capabilities – not only descriptive analytics based on historical data, but also predictive, streaming, and prescriptive analytics based on perishable data – to drive detective insights and improved digital customer
  • We are more iterative, agile and responsive - able to rapidly explore and incorporate real-time and near-real-time data to provide relevant, actionable
  • We are able to rapidly and effectively scale aspects of the data platform up and down as needed without long procurement cycles, enabling greater enterprise
  • We are able to provide both controlled data sets and access to raw data for exploration by data scientists (with appropriate
  • We can address the creation and existence of rogue data sets by addressing the reasons they spring up in the first place and better limiting access to data based on
  • We’re no longer stuck in proprietary software platforms that add little additional value and complicate our ability to
  • Our teams are out of undifferentiating work and focus primarily on creating business value through data and advanced
  • As analytics professionals and as technologists, we are key enablers of digital transformation, driving outcomes for our business and those we serve.

So How Do We Get There?

In the Relus Cloud strategy & consulting practice we help our clients win by helping them define an actionable strategy to drive transformative results. We use several main techniques:

  • Strategic engagements resulting in actionable transformation
  • Proof of Concept and Pilot architecture
  • Kick-starting the Analytics Innovation Engine and Analytic Centers of
  • Building a Data Transformation Factory

Unfortunately, most companies can’t just start over from scratch, but rather need to innovate their way out of a tangled set of past architectures. Ultimately, we counsel our technology-focused clients to make some critical moves in the right order and to do so in collaboration with the business.  This frees up the team’s energy by degrees to be reinvested in evolving new business capabilities, building a virtuous cycle of innovation.

As part of this process, there are some key data modernization tactics that we often employ. Here are the Top 10:

Top 10 Data Modernization Tactics

  • Tactic 1: Create a compelling vision, data strategy, actionable roadmap, and ensure both business and technology stakeholders are aligned to the execution
  • Tactic 2: Migrate away from proprietary databases to scalable open-source engines (MySQL, PostgreSQL, MariaDB) and platform services like Aurora and
  • Tactic 3: Get out of the datacenter business - unless your business needs to be the best in the world at building and maintaining data center
  • Tactic 4: Where possible, separate processing and storage, dynamically autoscale all the things, and stop “building the stadium for Super Bowl
  • Tactic 5: Build a zone-based data architecture incorporating at minimum raw, refined, and curated “for-purpose” zones with appropriate access
  • Tactic 6: Update your ETL - Integrate real-time streaming and in-flight processing with batch-based ETL/ELT pipelines and event-based
  • Tactic 7: Control and track data extracts using ephemeral data exploration “
  • Tactic 8: Use the right tools for the job, e.g. refactoring from SQL to NoSQL
  • Tactic 9: Update BI and Visualization capabilities to support an extended
  • Tactic 10: Implement an API-based data virtualization layer in front of curated data

The “secret sauce” in the Relus Cloud approach ties back to business and technology alignment and empowers all the others: we help our clients build an iterative, business-driven transformation engine in which process transformation and underlying data and analytics transformations are tied at the hip. It’s never just about the tools - maybe not even primarily so. Without appropriate alignment and without the associated people and process transformations, any of the tactics above will be of little use in driving measurable business outcomes. Relus Cloud can help your company identify how and when to use these tactics to drive measurable business outcomes.

In part 2 of this series, we’ll dive in to some of the tactics above, including when it’s appropriate to ditch proprietary databases, why you might want to do that, and how to easily migrate to open source databases using the power of the public cloud.

[1] Readers who are interested in exploring this idea further should review John Boyd’s OODA loop.