Big Data: The New Oil (and potentially just as messy)

minute read

Most organisations, big or small have by now learnt to appreciate the transformational power of data. If managed properly, structured transactional data captured in IT systems has proven to be a valuable corporate asset.

In recent years, this is being supplemented by unstructured data formats – e-mails, chat transcripts, social media interactions, to name a few. This has led to a veritable explosion in data footprints under management.


The data has now become “Big”!


Traditionally, structured data was managed in relational or hierarchical platforms provided by established names like Oracle, Teradata or IBM. However these platforms started proving to be inadequate to meet the ever larger footprints of data. The vendors did move with the times, providing for example dimensional implementations instead of relational ones to support data warehousing programs as against operational projects.

But the onslaught continued! The data volumes were going through the roof. So was the sophistication of requirements posed by a new breed of knowledge workers. The existing data modelling constructs and query management approaches were inadequate to meet these expanding requirements. There was only so much you could do for performance tuning!!

A new set of data management constructs are being introduced – columnar databases, NoSQL (not only SQL), document stores, MapReduce to name a few. This meant that the new database platforms were not merely an evolution of the old relational / dimensional paradigms, they were completely new. Any migration from old platforms to new would mean a step change.

The good news is that this point is not lost on most CTO/CIO organisations. Hence Big data POCs are being undertaken, with plans to eventually “roll it out”.

The bad news is that this rollout is usually not thought through. Big data strategy in most organisation goes something like this – “Buy a big data box, a BIG box. Find an analytics program that requires a large data footprint and migrate it to this box!” Even worse, a scaled down scope of the self-same analytics program is considered as the problem statement for the first POC. If the POC works, the program adopts the platform thereby concluding what is called a “rollout”.

This is clearly not a rollout.


So what IS it then?


A rollout is a plan to take the POC mainstream in your organisation. More than merely adopting a technology platform, it involves asking key questions such as –

  • Which business areas can we take this POC immediately to? Typically marketing, product definition or campaign management functions
  • Which business problems can this big data paradigm solve? More importantly, which problems would it NOT be able to solve? E.g., tread carefully when using big data for regulatory compliance
  • Can existing Data Governance, Data ownership and Data Quality Management processes be applied to big data? If not, what needs to change and what is the plan for implementing this change?
  • What infrastructure is required for this platform? Can we work with the platform vendor to achieve economies of scale in pricing without incurring upfront Capex, e.g. through staggered uptakes etc.
  • How are these new assets going to co-exist with legacy estate? What are the plans for rationalising the estate based on the realisation of key business / technology benefits?
  • What change is required in the project delivery life cycle for implementing a big data platform as against a relational platform?

Just because the Big Data paradigm revamps traditional data handling constructs (like query management or data modelling) does not mean you need to forget the best practices for a successful Information Management program.


How can EB help –


  • Making it real for YOU by helping to define the journey, setting out what it is you are trying to achieve, through a Big Data adoption roadmap
  • Help you through the vendor selection to ensure best fit (all the while reminding you that silver bullets do not exist!)
  • Stand by you through the journey, by being part of governance team ensuring no leakage in project scope
  • Create centres of expertise within your enterprise with the aligned management framework
  • Hand hold the new office bearers through the transition before stepping out