Big Data is nothing new, just more accessible than ever. Look at LIMRA and the reports they have been gathering for years on the life industry. It is literally collecting policy details on hundreds of companies to identify trends in the industry. Or the pharmaceutical industry in the US where you can query the database of all the medical prescription refills of an population.
A single company can collect big data on their exposures or policy holders (telematics) but that only provides a small piece of the puzzle. What about Big Data on the industry? In this case, you absolutely have to have standards to make this work. One could argue that big data in the industry is only possible when we all agree on a single standard for the data to be collected.
I guess there is a third piece which is big data in the very broad sense of everything available on an individual (e.g. Facebook, Airline Travel, Medical Information, Telematics). This calls into the equation the need for common identifiers and structure on party information.
Big Data is new. It's only partly the volume of data. It's also the amount of unstructured data from nontraditional data sources in such a variety of formats that the data warehouses and data marts that we have today can't effectively handle it. That's where all of the vendors are scrambling to figure out solutions. Do we need a strategy as respects standards and big data?
A strategy for how Big Data and Standards work together, I think this is an area that is still somewhat confusing. The core concept of Big Data is that there is an extremely large amount of data that needs to be processed and managed. The data is coming in as various formats, unstructured and with a lot of noise. If standards were used for the data collection, the need for new ways of thinking and processes is significantly reduced. However, the problem with Big Data is not just in the processing of the unstructured data but in the sheer volume. For many of the current Big Data projects, the data is already structured (more or less) but there is a need for zeta bytes worth of it in order to get the appropriate analysis done. As these projects continue to gain traction, there will be needs for standards that people aren't thinking about right now.
No doubt that Standards will play a key role .