The next frontiers for big data and analytics

What to analyse is one of the biggest questions for the big data and analytics world today. While large volumes of data can be handled, systems can still be overloaded, and resources wasted on looking at data which does not produce useful insights.

There is still an element of serendipity with analytics, said Manik Narayan Saha, CIO, SAP Asia Pacific & Japan, during a media roundtable at the IDC & Intel Big Data and Analytics Forum in Singapore, as a problem that has been defined differently might require data that had been discarded. “It's a learning process to understand what data can do for us, a learning process of trying to figure out what should we capture, keep, and archive, and what should we completely discard. We're at the initial part of the journey,” he said.

Craig Stires – Associate VP, Big Data, Analytics and Software, IDC Asia Pacific, shared the experience of Westpac, which has a data framework that scores datasets by relevance multiple times before deciding if that data is worth keeping.

He suggested researching to see if information can be further categorised. Iproperty.com, which owns property portal websites in Asia, knows that viewing and browsing behaviour does not matter beyond 18 months before a property purchase, when customers first make an initial assessment of the market, he said. Six months before their purchase they check specific properties, and three months before the purchase they apply for financing. “Tiering data is a real part of the conversation,” he said.

Eddie Toh, Data Center Platform Marketing Manager, Intel, suggested collecting the most useful data with the help of an Internet of Things (IoT) gateway, hardware which acts as a gatekeeper to decide if information should be processed on the spot, discarded, or forwarded for further analysis. If there has been no change in the scene in a video surveillance environment, there is no need to capture it, he pointed out. “Intelligence at the edge ensures that the right data has been captured,” he said.

Richard Jones, VP, ANZ/ASEAN, Cloudera, noted that a sensor taxonomy standard may also help. "That's the next evolution of big data,” he said. “The manufacturers and government bodies need sensor standards by vertical industry for telecommunications, for healthcare.”

Need context?

Read the TechTrade Asia blog post on Dell's latest IoT gateway.