We are currently living through not only the adoption of Big Data by companies at an ever-increasing rhythm, but also an acceleration in the spread of strategies for transforming them into Data-Driven organizations. Of course, in the beginning, it was the technology departments leading projects looking to save on storage and mass data processing costs, but marketing departments are now also adopting Big Data with the goal of supporting their digitalization strategies and searching for the greatest efficiency in commercial campaigns thanks to a better understanding of the client, among other benefits.
Major international companies have already implemented the adoption of business strategies based on Big Data, a massive adoption led by business and more and more by executive directors. In fact, according to a study by the IDC, only 10% of Big Data projects are purely IT projects.
The key to this unstoppable trend lies in the verification that technologies such as Big Data can offer a clear return on investment for the company, specified in new forms of income as a result of anticipating demand, developing new, personalized products and services, improving the level of operational efficiency, developing a greater analytical capacity for decision-making, and preventing and predicting risk and fraud. However, Big Data is also a source for new business models based on the monetization of data, which is inciting a transformation in businesses that offer data products and services, a transversal phenomenon that is affecting all industries and sectors.
The biggest difference is that, for the first time, the demand for rapid change and adaptation required by companies is not necessarily measured by the rhythms of the technological transformation. Big Data is strengthened by the cloud, virtualized infrastructure and advanced analytics as a service. Therefore, while IT departments design evolving roadmaps in order to integrate their traditional architectures with Big Data, the business can continue to develop tactical projects based on use cases with a short-term return on investment.
Recognizing the importance of Big Data, however, does not mean being capable of adopting it successfully. And that is because fully realizing the potential of Big Data is a challenge of not only processes and technological capacities, but above all one of organization and strategy. In my experience, the adoption of Big Data as a part of a corporate strategy includes five great challenges on a management level: leadership, technology, talent management, decision-making and organizational culture.
As we know, disruptive technologies provoke exponential changes, yet organizations evolve at a slower pace and organizational leadership is necessary to maintain the rhythm in spite of internal and external tensions that may be provoked by implementing our strategies in business models based on data.
From a technology point of view, Big Data implies that the center of gravity of information systems will no longer be applications. This changes the manner of processing, analyzing, and exploiting information, as it will be necessary to reconsider the fit of traditional systems. Some technological decisions of the last five years could have been mistakes or even obsolete, and this has to be recognized in order to react in time.
The third great challenge is the management of talent: new professional profiles are necessary that know how to manage and analyze Big Data. A new generation of professionals, such as data scientists, Big Data architects and data engineers, who not only have differential knowledge and abilities, but also various motivations, interests and expectations, and whose job positions are totally different from previous generations. Thus, there is a challenge in detecting that talent, forming it, and retaining it, but, above all, being capable of generating an organizational environment appropriate for its development.
From another point of view, Big Data is not only changing the way in which we discover new hidden relationships and unique behavior patterns from the information, but it also forces us to make decisions differently. One of the biggest difficulties on a business management level is showing directors how to make decisions based on the data, instead of basing them on reports reflecting on the past or on their own intuition or prejudices.
Finally, Big Data requires a transparent and open organizational culture for sharing information, understanding that it is an asset of strategic value that does not belong to any department or system, but rather to the organization itself. From the culture of “this data is mine,” we need to move towards publishing and sharing data on a corporate level, through departments and channels, while obviously following security and government policies. This cultural move towards a more open, transparent and agile environment is one of the main obstacles when it comes to realizing the potential of Big Data, especially in large corporations with rigid structures that must compete in a market with smaller, more dynamic and more innovative companies.
The phenomenon of Big Data cannot be reversed. It is a dominant factor in all sectors and in our lives, that will forever change the way in which we interact with information as a society. Companies as well as countries need to keep this in mind. Only those that know how to generate an adequate framework for data exchange between companies – obviously with security and guarantees – but a framework that allows for the generation of business value, will be competitive in the new economy of data.