Experts debate data scientist shortage

From left: Ramanathan, Jones and Saha at a media roundtable during the IDC & Intel Big Data Analytics Forum. Ramanathan is holding a sample Intel Xeon E7 v3 processor.

While the benefits of big data and analytics are well known, they gloss over one slight problem - that insights still require the human touch, and that there are not many people who have the skills to tell what should be analysed and how to interpret the data.

At a media roundtable organised at the IDC & Intel Big Data Analytics Forum 
Intel's Eddie Toh, Data Center Platform Marketing Manager, said that the problem begins much earlier, when companies do not start on big data projects because they believe they do not have the expertise. "Companies think they need to hire people to start a project, but that's not neccessarily true. They need to understand what they want to achieve, what problems they want to solve. That's where a lot of companies get stuck. What they need to do is articulate the problem better, and get others to help,” he said.

Richard Jones, VP, ANZ/ASEAN, Cloudera, added that big data may not necessarily require extremely specialised skills. "The biggest barrier is people thinking you need to have a PhD,” he said. “The reality of this is that you don't. It's a business analyst role; it's how you access the data.”

Such skills are however in great demand. Jones shared that Cloudera is helping customers on building competency with an internal hiring agency “because the skills gap is really hard.”

Stires.

Craig Stires – Associate VP, Big Data, Analytics and Software, IDC Asia Pacific, said that good data scientists are still hard to find. “The majority will understand the business and the market, but to break existing models of thinking you have to have people who are real scientists,” he said. “Those people are few and far between.”

He also shared that the clout of two sectors in particular – financial services and defense – enable them to take in the majority of available talent, leaving few skilled personnel for others to hire, though Australia, Singapore, South Korea, and Taiwan tend to have people with more of the required skills.

Stires added that the best insights will come from someone in-house. “Telcos, manufacturing, and logistics firms have a lot of infrastructure investment and are short on top-end innovation. They outsource a lot of the analytics. Data management is fine to outsource, but an idea of where the business is going next, (should come from) someone who sits internally,” he said.

Deepak Ramanathan, CTO, SAS Asia Pacific, said that it might be a team that big data projects need, rather than focusing on a single person who has all the skills they need. "They need to write code, be top grade statistical guys, and also understand the business," he said. "If you try to find all three in one individual, you're always going to be scraping the bottom of the barrel... you might need a team of people rather than these guys (who are) supermen. They are hard to find and hard to keep and there is disappointment when they leave," he said.

Manik Narayan Saha, CIO, SAP Asia Pacific & Japan, suggested relying on more intelligent software instead. “Business intelligence is about things you know,” he said. “It's now become discovery based. How do I start discovering patterns and data? That's where the shift is taking place. That leads to advanced analytics.

"Putting tools into the hands of every user so they can go into existing data and find out things could be the gold mine for many companies. In one case, savings that a company got out of an implementation paid for the whole implementation, he said. “If every employee has full transparency there is a lot of optimisation potential, maybe he doesn't need any training. Every user (would be) a power user.”

Stires added that questions and answers may require interpretation. “Whatever they are called - centres of excellence, competency centres - those structures provide an important bridge. They can translate requests from the executive level..to people who can actually run the ship,” he said.