We are seeing more and more that Big Data can be kind of a big deal. Corporations and even governments are addressing the issue and making investments to get ahead of it. There are 2.5 quintillion bytes of new data created EVERY DAY. To put that number into perspective, it means that 90% of the data in the world today was created in the last two years alone! We can think of Big Data in terms of the 3Vs : Volume, Velocity and Variety
This term describes the inability to process the sheer amount of data received. This is the main attraction of Big Data analytics - the ability to process more data results in a more encompassing report and more detailed analytics which results in your organization’s ability to better predict trends and demand.The first major example of this is the Human Genome Project the goal of which was to sequence the 3.3 Billion base-pairs of the human gene. It took 13 years and US$3Billion. Today, modern Big Data analysis solutions can sequence DNA in less than a week for $1,000!
More locally, with over 25.5 million tax payers in Canada, the Canadian Revenue Agency’s 40,000 full-time employees have to manage all of the tax information and data. With tax returns, benefit administration etc this amounts to hundreds of millions of data streams annually. Not only is an efficient Big Data solution required to manage all the data from a variety of sources such as Netfile and paper filings, but it also needs to be processed within a timely manner as expected and demanded by the tax payers.
This data is time-sensitive, being able to analyse data faster – in real-time or as close to real-time as possible – not only help you make better business decisions but will also give you a competitive advantage. For example, if you’re able to analyze your customers’ movements throughout your website in real-time you have the ability to make suggestions and recommendations based on what they’re looking at. Providing this immediate insight gives you an advantage over competitors who may not have access to real-time reporting and could be following up too late, or not at all.
An example experienced by every organization is the importance of up-to-date budget reporting. In any organization, especially larger organizations there can be hundreds or even thousands of revenue/expense transactions made a day. These daily fluctuations can have a dramatic effect of purchasing decisions – the more real-time budget reporting can be, the better and more informed purchasing decisions are going to be. A famous IBM commercial demonstrated this concept by making the point that you wouldn’t cross the road if all you had was a five-minute old snapshot of the traffic. In most business, and in our everyday life, we need data analysed as quickly and efficiently as possible in order to make informed decisions.
Data comes from everywhere in many formats, from social media posts, field staff reports, videos posted online, transaction records, the list goes on…
All this data is nebulous - it comes from different software, different versions, and different vendors. The golden rule for Big Data is: if possible, keep everything. Keeping all your data will ensure you don’t discard something that could have been extremely useful. However, the key here is finding a Big Data analysis tool that has the capability to monitor all of your source data to combine, analyse, and report on the data in a timely manner.
An example is Ontario College Application Service (OCAS), an organization that is responsible for managing over 200,000 college applicants annually. They need to manage incoming applications for their 24 colleges but they also need to manage data received from the colleges going back out to applicants. Also, OCAS has a recruitment program consisting of social media, advertising, school visits, etc. – all of this data is coming in from a variety of source formats. In order to sort through recruitment data, applicant data, and college offer data a Big Data solution is required to create meaning and develop trends from these sources. With OCAS having the ability to combine data from different data sources they are able to see trends that wouldn’t have been visible otherwise, to the benefit of students, the colleges, and those running OCAS.
In summary, Big Data is well – BIG! The first step in managing your Big Data problem is to figure out what V is your biggest challenge. It is possible to be struggling with all three, but in order to start managing your Big Data you need to decide which V is going to be crucial in getting your organization ahead of its ‘Big Data.’ This will determine what Big Data software you are most likely to benefit from, providing focus in your selection effort.
"See the ordinary become extraordinary" Watch the short video below to see an illustration of how Big Data permeates the entire world as we know it from the smallest, most innocuous items to the global macro economy.