IBM Watson to understand us better

- First commercial availability of key technologies from Project Debater

- New capabilities for IBM Watson will help enable businesses to begin mining and analysing some of the most challenging aspects of human language

IBM, the leader in artificial intelligence for business*, is announcing several new IBM Watson technologies designed to help organisations begin identifying, understanding and analysing some of the most challenging aspects of the English language with greater clarity, for greater insights.

The new technologies represent the first commercialisation of key natural language processing (NLP) capabilities to come from IBM Research's Project Debater, the only artificial intelligence (AI) system capable of debating humans on complex topics. For example, a new advanced sentiment analysis feature is defined to identify and analyse idioms and colloquialisms for the first time. Phrases, like 'hardly helpful,' or 'hot under the collar,' have been challenging for AI systems because they are difficult for algorithms to spot.

With advanced sentiment analysis, businesses can begin analysing such language data with Watson application programming interfaces (APIs) for a more holistic understanding of their operations. Further, IBM is bringing technology from IBM Research for understanding business documents, such as PDFs and contracts, to also add to their AI models. APIs link one technology to another, or to different types of data.

"Language is a tool for expressing thought and opinion, as much as it is a tool for information," said Rob Thomas, General Manager, IBM Data and AI. "This is why we're harvesting technology from Project Debater and integrating it into Watson – to enable businesses to capture, analyze, and understand more from human language and start to transform how they utilize intellectual capital that's codified in data."

IBM plans to integrate Project Debater technologies into Watson throughout the year, with an 'ABC' focus on advancing clients' ability to exploit natural language:

A. Analysis

IBM has enhanced sentiment analysis to be able to better identify and understand complicated word schemes like idioms (phrases and expressions) and so-called sentiment shifters, which are combinations of words that together take on new meaning, such as "hardly helpful." This technology will be integrated into Watson Natural Language Understanding this month.

In addition, IBM is announcing a new classification technology that will enable clients to create AI models that can more easily classify clauses that occur in business documents, like procurement contracts. Based on Project Debater's deep learning-based classification technology, the new capability can learn from as few as several hundred samples to do new classifications quickly and easily. It is planned to be added to Watson Discovery later this year.

B. Briefs

Summarisation technology pulls textual data from a variety of sources to provide users with a summary of what is being said and written about a particular topic. An early version of summarisation was leveraged at The GRAMMYS this year to analyse over 18 million articles, blogs and bios to produce bite-sized insights on hundreds of GRAMMY artists and celebrities. The data was then infused into the red carpet livestream, on-demand videos and photos across www.grammy.com to give fans deeper context about the leading topics of the night. It is planned to be added to IBM Watson Natural Language Understanding later in the year.

C. Clustering

Building on insights gained from Project Debater, new topic clustering techniques will enable users to "cluster" incoming data to create meaningful "topics" of related information, which can then be analysed. The technique, which is planned to be integrated into Watson Discovery later this year, will also allow subject matter experts to customise and finetune the topics to reflect the language of specific businesses or industries, like insurance, healthcare and manufacturing.

IBM has long been a leader in NLP, developing technologies that enable computer systems to learn, analyse and understand human language – including sentiment, dialects, intonations, and more – with increasing accuracy and speed. IBM has brought its NLP technology, much of which was born in IBM Research, to market via Watson. The resulting products include Watson Discovery for document understanding, IBM Watson Assistant for virtual agents, and Watson Natural Language Understanding for advanced sentiment analysis, are all infused with NLP.

ESPN Fantasy Football uses Watson Discovery and Watson Knowledge Studio to analyse millions of football data sources each day during the season to offer millions of fantasy football players real-time insights. By processing natural language, Watson identifies the tone and sentiment of news articles, blogs, forums, rankings, projections, podcasts and tweets that cover everything from locker room insights to injury analysis. ESPN Fantasy Football surfaces these insights in player cards that snapshot the "boom" and "bust" potential of each player, as well as in a Player Buzz section that summarises the positive or negative commentary about a player.

"The thumb rule for the success of any NLP project is simple and straight forward - the better the ability to understand language and complex topics, better the ability to deliver outcomes for the business. A recent study by Greyhound Research confirms the increasing use of chatbots and similar intelligent platforms using NLP by organisations across the globe. The study confirms that while over 73% of large organisations globally are either already running a project or looking to launch one in the next 12 months.

"The fact that the work on NLP is moving from IBM’s research labs to commercial availability now, goes to show the progress that has been made with Project Debater and the technology. Also, the project’s effort to cluster a variety of incoming data by topics, summarise it for business users and provide enhanced sentiment analysis is a significant step forward in the field of NLP. Such technologies can significantly benefit organisations looking for better insights from their data and, most importantly, establish a better connection with customers and stakeholders," said Sanchit Vir Gogia, Chief Analyst, Founder and CEO, Greyhound Research.

Explore:

Check out the trailer for The Debater, a behind-the-scenes look at the making of Project Debater though the lens of researchers who take AI into uncharted territory. An Official Selection of the Copenhagen International Documentary Film Festival.

^*IDC Market Share: Worldwide Artificial Intelligence Market Shares, 2018: Steady Growth — POCs Poised to Enter Full-Blown Production (Doc # US45334719, July 2019)