Natural language understanding: the secret to content analysis
Computer scientists have been trying to get computers to understand natural, human language since the 1960s. As wave after wave of brilliant innovation synergized with increases in computing power, the idea of Natural Language Processing (NLP) went from the realm of science fiction to being reality, albeit one that needed significant compute resources to produce limited results.
Then, in a rather quick leap, NLP, and Natural Language Understanding (NLU), a subset of NLP, have become so commonplace, we scarcely even notice it anymore. Not convinced? Ask Siri to explain it to you.
The work is not finished, however. Advances continue to come on the market, pushing computers’ language comprehension abilities ever further. New use cases are also emerging. They demonstrate the value of NLU in novel contexts like cognitive search and automated reasoning.
Natural Language Understanding should be a fairly easy concept to grasp. A computer equipped with NLU capabilities can understand natural language, such as the text of a written document or a spoken sentence. That’s why the technological capability is sometimes referred to as natural language interpretation.
NLU is part of NLP, which deals with the overall process of getting machines and humans to interact using human-like language. NLP contrasts to the standard mode of human-to-machine interaction, wherein the human’s input is translated into a machine language the computer can understand. NLU is the listening side of the equation.
With NLU, the computer comprehends natively human communications. NLU works by combining software logic, linguistics, ML, and AI to parse the unstructured data that comprises natural language.
Where things get a little tricky is in understanding what NLU is capable of—and what it isn’t, yet. For example, it’s one thing to get a computer to understand a sentence like, “I would like to order a bus ticket.” It’s quite another for a computer to accurately comprehend a comment like, “Every man at the party spoke to one woman.” The latter sentence can have two quite different meanings. Did each man at the party speak to a separate woman, or did all the men speak to a single woman? The sentence could have either meaning. Context might help, like if the next sentence read, “There were five men, and five women at the party, each deep in private conversations.”
The predominant modes of NLU today are relatively simple, more “I drove the car” than “I can’t admit I resent my sister-in-law for being a drama queen.” Examples include various Robotic Process Automation (RPA) workloads like ticket and email routing, along with Machine Translation (MT) services like Google Translate.
Some NLU solutions are capable of “automated reasoning” and question answering, a la Siri. The thing to keep in mind is that even the “simple” NLU workloads are quite sophisticated. They represent decades of incremental improvement and the application of a great deal of Artificial Intelligence (AI) and Machine Learning (ML).
How does NLU work? Although there is a wide range of approaches, most NLU systems employ a standard set of components that work in concert.
The most basic element of the system is a lexicon, which defines the language. A lexicon is more than a dictionary. It puts words and phrases into a structure that NLU can use.
Then, working with a grammar rules component and a semantic guide to comprehension, the system can start to break down the language it is presented and transform it into something meaningful for the user and itself. Collectively, these components are called a “semantic parser.”
The semantic parser performs text categorization. In the sentence, “I would like to order a bus ticket,” the semantic parser is able to identify that “I” refers to the system user, “order” is an action the system user wants to take and “bus ticket” is a product or service to be delivered to the user. In effect, the system is using the basic English subject (noun), verb and object sentence structure and converting it into an expression the computer can use—in this case, to sell a bus ticket.
As one might imagine, getting this to work, even in a simple use case, represents a massive amount of work. Lexicons and accompanying grammar rules engines can take years to develop.
Cognitive search has emerged as a compelling use case for Natural Language Understanding. Representing a new generation of enterprise search, cognitive search uses advanced unstructured data analytics to conduct content analysis and extract relevant information from multiple, diverse data sets and improve the results of users' search queries.
Cognitive takes enterprise search further by combining indexing technology with AI, ML and NLU/NLP to scale a wide assortment of data sources. The technology uses NLU to put natural language to work in optimizing enterprise search processes. For example, if an enterprise search user enters the query, “Find contracts for Acme, Inc.,” a conventional enterprise search engine would dutifully discover all available documents that referenced contracts for Acme, Inc.
With NLU, however, the search might find other, relevant but not closely connected results for the same query. It might, for instance, find an email that contained the phrase, “I want to re-negotiate all agreements.” An advanced NLU function, such as the one found in Sinequa, could link the word “re-negotiate” with “agreements,” and know that “agreement” is a synonym for “contract.”
Adding this email to the search results gives the user more information about what’s really happening with the Acme contracts. With NLU in cognitive search, an organization's employees gain the ability to discover and access information relevant to their work contexts. Developers can also build NLU-powered search applications and embed them into business process applications.
This simple example offers a glimpse into how Natural Language Understanding can be the secret to dramatic improvements in content analysis. With NLU, the enterprise search solution gains a better understanding of content, as well as the connections between pieces of content. The result is a more effective enterprise search experience and ultimately better outcomes from business processes that employ enterprise search.
Related blog posts.
One of the many exciting futurisms predicted by the original Star Trek TV series was the notion that people could have conversations with computers. “Computer, show me a map of all ...