Will Big Data Change Your Business? Yes, and Very Soon!


Andrei Povarov

Emerging technologies take different roads to market, sometimes taking a route via unexpected shortcuts to maturity and productivity. This means that two groups of technology users coexist: those who have vague ideas about a new technology, and those who are actively using it to generate revenue. This pattern can clearly be observed with Big Data today. Big Data mysteriously disappeared in 2015 from the Gartner Hype Cycle (a reputable annual report summarizing cross-industry perspectives on emerging technologies and trends). Indeed, in 2014, Big Data was about to leave the Peak of Inflated Expectations but then it skipped three remaining Hype Cycle phases and found itself to be out of range as a fully matured technology. Which group do you belong to? If the former, can you jump to the latter? If yes, all you need to do is to figure out how Big Data can work for your business.

Big Data does not mean operating large databases. Volume is only one component, the others being Velocity referring to unprecedentedly high data collection and processing speed and Variety allowing structured and unstructured data formats such as documents, email, images, audio, video, etc. These characteristics (Volume, Velocity and Variety) make the classical ‘3V’ definition of Big Data. Sometimes one more V is added for Veracity representing data uncertainty: inconsistency, incompleteness, inaccuracy. Whatever is the number of ‘V’s’, the term Big Data refers not only to data, but to technologies and processes helping to extract useful information and to generate practical business insights.

What are the sources of these huge unstructured data masses and why should their processing be useful? The digital economy we have entered is creating much more information than before. If five exabytes (five billion gigabytes) is what the whole of mankind produced from the earliest days of civilization until 2003, between 2003 and 2010 the same amount of information has being generated every two days, and from 2013 five exabytes were recreated every 10 minutes.

In a similar process to extracting rough materials from mines and refining useful minerals; collecting and mining Big Data can bring priceless findings especially when put in a real business context. Today we only process 0.5% of data collected4, but the Big Data technology and services market is growing at a 26.4% compound annual growth rate with estimates to achieve $41.5 billion vendor revenue derived from sales of related hardware, software and services in 2018, or about six times the growth rate of the overall information technology market.

Is the army of trained analysts doing this mining? Not at all, and this is the key point in understanding Big Data. No human manual work can cope with 3V/4V complexity, instead self-learning algorithms can do the trick while the function of humans (called Data Scientists) becomes to set up algorithms, help the programmes to learn, interpret and visualize results, monitor and correct. Algorithms learned from huge data sets can find not obvious, not scientifically justifiable, not even understandable dependencies and patterns. The amazing fact is that this works in practice really well! Similarly to how in ancient times people could farm without any scientific knowledge about soil, temperatures, climate, grains, referring only to empirical observations and experiments gathered many generations before, today any large volumes of data collected in a particular business area can be processed through self-instruction.

There are many examples of Big Data’s practical use. For example, clients’ purchasing behaviour patterns can inform about other needs sometimes not known by customers themselves, which generates considerable cross-selling revenues. Telecom companies are retaining clients by recognizing signs of coming ‘unloyalty’ even before clients themselves realize this themselves. Banks are assessing risks of fraud and doing credit scoring. Recommender systems select for a customer what films to see or what music to listen based on feedback from people with similar taste. In politics – Obama’s presidential campaign is considered the first ‘data driven’ 6 campaign in presidential history.

Taking business aside – how can Big Data affect your private live? Is it good or bad for you and what are the legal aspects here? A simple test: what do you feel when an advert appears on your screen on the subject you’ve mentioned just few minutes ago in your private email? Not to worry! Algorithms again, not human beings, are reading your e-mails and they do no more than usual spam filters.

Can someone challenge your privacy using Big Data? Not really. Collecting data is normally anonymous, while results of Big Data algorithms apply to your personal situation only within a closed environment (your internet space with a particular shop). You should only be aware that the existence of hidden dependencies (discovered by Big Data methods) makes your profile a much deeper source of information about you by involving your behaviour/purchase history, social networks interactions etc. Having this in mind and being careful with your digital traces (just common sense) while supported by personal data protection laws including Russia, creates protection.

Can consumers benefit from Big Data? Yes, personal decisions can be made more effectively, cross-selling can be in buyers’ interests, recommender and personal insights systems can directly answer customers questions such as: ‘Am I spending too much time in social networks during sunny days?’.

Is Big Data working in Russia? Quite a lot, and there are international and local companies deeply involved. For example, Yandex is using Big Data technology for more than 70% of its products and applications8, most of them have already served the general public for years, while some of them have revolutionized business in Russia.

How can companies find out more about Big Data and start using it? Most available training programs are still aiming at techno-analytical skills suitable for data scientists (e.g. Yandex Data Factory, Digital October). Very recently professional Big Data training programs for top- and mid- level managers have started to appear (e.g. New Media School/Higher School of Economics) giving practical knowledge on how to start Big Data in your company, whom to recruit, what tasks to formulate and how, without digging into technical details.

A general piece of advice on assessing Big Data’s potential benefits for your company is to consider data supply and demand. What role do data collected in a natural way play as part of your normal business processes (supply)? Can you or someone else create value out of it? Can you use or start selling data? What data can bring value for your customers disregarding sources of such data (demand) and if you do not collect all this by yourself, can you buy or obtain such data from open sources? The remaining processing part can be done by deploying in-house facilities (e.g. Hadoop-based systems) or by using a cloud environment (e.g. Amazon Web Services).

To summarize, Big Data is a powerful tool that can create a lot of value for your business. Not using it puts you in a disadvantage, as your competitors will certainly be there. If you do not change your business to accommodate Big Data, it will be changed anyway by others and probably not in a direction you want. And note: all this is actively happening right now – remember that only the last year, Big Data jumped out from the Hype Cycle!