You may have heard of the term ‘big data’ – it’s one of those technologies that some people are making a lot of noise about. It’s been written about in many publications, e.g., Harvard Business Review blog, Forbes, MIT Sloan Management Review, McKinsey Quarterly. However, as one analyst has commented, the hype about it is reaching “the levels of SOA in the early 2000’s, cloud in the late 2000’s, and social in the past few years.”
A nice simple definition of big data is this:
- Extremely large volumes, of mainly unstructured data, which are streamed (not batched), and which require real-time analysis.
The main characteristics of big data are described as:
"Volume - This characteristic describes the relative size of data to the processing capability … Overcoming the volume issue requires technologies that store vast amounts of data in a scalable fashion and provide distributed approaches to querying or finding that data ...
Velocity - Velocity describes the frequency at which data is generated, captured, and shared. The growth in sensor data from devices, and web-based click stream analysis now creates requirements for greater real-time use cases …
Variety - A proliferation of data types from social, machine-to-machine, and mobile sources add new data types to traditional transactional data. Data no longer fits into neat, easy-to-consume structures …"
So who is using big data? Forrester Research indicates that the businesses using big data are the ones that have always handled lots of data in the past – e.g., banks and insurance companies, telcos, oil companies, large retailers, and the international CPG companies. An analysis by McKinsey defines those industries where big data has high value vs. those where its value is low, and also indicates that currently the value of big data isn’t that great for manufacturing businesses. The same McKinsey article mentions that the analysis was done for businesses with over 1000 employees, but if you look at the US Census bureau’s 2010 statistics for business you will see that businesses of this size make up less than 9000 (0.15%) of the total of over 5.7 million businesses in the US.
How relevant is big data? It seems that most of the hype is coming from database vendors and consultants, which you might expect. A recent article had the following advice:
"One of the key lessons from the history of marketing science is that when a new data source becomes available, everyone is quick to fall in love with it. But smart companies take a step back and strive for a more holistic view of their customers and markets."
Is big data appropriate for small- and mid-size business (SMB)? Some comments:
- Most organizations don’t yet have the data sources for it – whether from customers, manufacturing plants, operations, or inventory.
- The SMB market is very conscious of risk and cost – big data technology is currently ‘bleeding edge’ and the cost of implementation is high. There are also other issues – sorting out the technology isn’t even 50% of the work; there are new skills, processes, and management approaches that need to change to make proper use of it.
- Many organizations that have large data volumes (in the gigabyte range) can easily handle them using existing relational database (RDBMS) software. The combination of existing Microsoft technologies – e.g., SQL Server 2012, Power View, and Power Pivot – provide the capability to manage and analyse large volumes of data without having to get into new technologies.
From another external perspective, TechnologyEvaluation.com some very complementary things to say about SYSPRO:
"The company develops its own nifty gadgets and functionality, but it often waits for Microsoft’s products to get mature (i.e., Azure, SharePoint, Windows 8, etc.) and then it comes up with some great tools on top of it, often outdoing Microsoft...big data is also in SYSPRO’s roadmap, and my guess is that it is again waiting for Microsoft’s in-memory analytics technology to mature in SQL Server."
Finally, what could make big data become more important to the SMB market?
- The data available gets above a certain critical threshold
- Measuring and recording instruments become a part of operations to generate very large volumes of data
These conditions could arise with the growth of the Internet of Things. This refers to how objects are becoming embedded with sensors and linked through networks, often communicating via the Internet. This could create huge volumes of data that flow to computers for analysis, and could therefore have an impact on SMB manufacturers and distributors.
If you are part of an SMB organization, is big data on your horizon? We would be very interested to hear from you.