Since a few decades ago, science fiction books and movies have rarely missed a chance to show any kind of an intelligent computer system that understands precisely what humans tell it and acts accordingly. Of all the predictions done by sci-fi writers, this one is being fulfilled frustratingly slowly.
Granted, there has been a significant progress: Google Now, Siri, and Cortana understand most of the queries by users reasonably good. And yet, no one would say any of them get 100% of what’s said to them right. The case of Facebook M, the personal assistant that appears to be still powered by humans, is a good example of problems that AI wouldn’t be able to solve.
The main issue here lies with Natural Language Processing (NLP), an interdisciplinary field that’s aimed, simply speaking, at creating an interface between the computers’ and humans’ languages. The applications of an interface like this are numerous if not unlimited, including all kinds of chat bots and personal assistants, machine translation, consumer electronics, and so on.
Let’s take a look at the main problems and issues the scientists face on the way to building a universal machine that would understand any human’s queries in a natural language.
Tagging and structuring
Among the main steps of understanding the natural language are part of speech tagging and named entity recognition. These steps are relatively easy, and in most cases (and in most languages) machines are reasonably good at it.
Akshay Sharma on Quora shows a nice example of how a text would look after being parsed for speech parts and named entities:
INPUT:
Profits soared at Boeing Co., easily topping forecasts on Wall Street, as their CEO Alan Mulally announced first quarter results.
OUTPUT:
Profits/N soared/V at/P Boeing/N Co./N ,/, easily/ADV topping/V forecasts/N on/P Wall/N Street/N ,/, as/P their/POSS CEO/N Alan/N Mulally/N announced/V first/ADJ quarter/N results/N ./.
KEY: N = Noun, V = Verb, P = Preposition, Adv = Adverb
INPUT:
Profits soared at Boeing Co., easily topping forecasts on Wall Street, as their CEO Alan Mulally announced first quarter results.
OUTPUT:
Profits/NA soared/NA at/NA Boeing/SC Co./CC ,/NA easily/NA topping/NA forecasts/NA on/NA Wall/SL Street/CL ,/NA as/NA their/NA CEO/NA Alan/SP Mulally/CP announced/NA first/NA quarter/NA results/NA ./NA
KEY: NA = No entity, SC = Start Company, CC = Continue Company, SL = Start Location, CL = Continue Location
Structuring the data from the natural language to a database or something similar is a bit less developed. There are, however, a number of startups that work on a similar problem, offering automatically generated summaries of long texts in the natural language, like newspaper articles or even books.
It works even better the other way around. For example, there’s Narrative Science, a company that makes money on Natural Language Generation. In this case, the company’s service called Quill creates analyst reports based on (extremely boring) financial documentation.
Sentiment analysis
Another aspect, in which the machines are getting better, is sentiment analysis, i.e. understanding of the basic attitude of the person speaking or writing. It can range from positive/negative deeper to things like happy, or angry, or excited, or depressed, and so on.
There are lots of startups doing sentimental analysis of social media posts and reviews. However, there’s also an additional value of understanding emotions for the purposes of NLP:
People can be excited, they can be nervous, or they can be angry or frustrated, and they speak differently and choose different words and key phrases when each of these emotions is presenting. Natural language processing algorithms must cut through this emotion to get to the point of a query. But there is also predictive value in that emotion, and these algorithms can use the emotion to build context and predict questions that a user might want to ask.
And yet, even the best sentiment analysis tools might have hard time trying to figure out the emotions behind these three sentences:
- I wanted to like my new XYZ dishwasher, but the controls were too confusing.
- The XYZ is my new best friend! I didn’t like replacing my old dishwasher, but now I am hooked on brand ABC.
- I would have liked the dishwasher better if it had a timer.
Even though all three have the word “like” in them, only one is actually positive; in addition to that, the positive one is negated with “didn’t.” This brings us to one of the main issues of NLP.
Ambiguities
The task of understanding the sense of a sentence, in which a coreference or an ambiguous word is used is something humans do without even thinking of it. For the machines, however, it’s an issue that’s still far from being solved.
For example, in the sentence “Carter told Mubarak he shouldn’t run again” the word “he” could possibly refer to each person. Another example is the English pronoun “it” that has many uses, from “It’s raining” to “Sometimes it’s the loudest who have the most influence.” Try to think about it from the computer’s perspective, and you’ll see how tricky our everyday language is to parse correctly.
Same goes for words that have more than one meaning. If you’ve ever had to use machine translation, you might have experienced this issue when certain words are translated correctly but in a different meaning than in the original sentence. This issues is solved relatively simple if the word in question is a part of a longer text and can be related to a context. However, a virtual assistant would need to deal with very little context when parsing a query, which makes the task orders of magnitude more difficult.
Speaking of virtual assistants, there’s another interesting problem to solve in this area. In many cases, it could be difficult to understand the actual intent of user’s question. This means that sometimes we ask for one thing, while what we actually want is something different. To be honest, sometimes parsing of things like these is difficult even for human beings, leave alone the machines.
These are the most significant issues standing between us and an ultimate natural language processing solution that would be able to make our lives easier. If you’re working in this area or care about NLP, feel free to share your insights in the comments section!