Lets talk about a hypothetical software program to solve a simple problem of taking a natural language question like – “What was the last painting of Leonardo Da vinci?”. which would give out St. John the Baptist as the answer.
The first solution to strike your mind would be to use Google Search API to simply post the question, parse the responses and select an answer based on keyword search on google responses. But you are going to be disappointed after building your software because it won’t have the solution you are looking for. Take a look at Google search results for my question. Not a single mention of the answer anywhere on this page.
Google is nowhere close to what I asked! All major search engines are keyword based who probably strip out keywords from my question and try to find those words on the pages it has crawled. How I found the answer was by clicking the Wikipedia link which contained the List of paintings of Leonardo Da Vinci.
This is the high level diagram of what the search algorithm should have been.
Why couldn’t Google or any other search engine figure this out. If this was exciting and you are planning to build this software then let me warn you – this problem has never been solved by any Human in the last 60 years since Alan Turing published his famous Turing Test.
We humans are way more powerful and intelligent that we give our self credit for. The reason the problem seems so easy is because the problem is extremely trivial at human level intelligence. But when it comes to replicating this intelligence in a computer program, its just too difficult.
So what are the options we have?
- Deep learning Neural Networks? No, this is not a pattern recognition or learning problem and even if it was, Google would already be trying to solve this problem instead of making his Neural Network experts write AlphaGo to entertain the Media and scare the human race that machines are going to take over them soon.
- Linguistics and NLP? No, just finding the parts of speech in a sentence is not going to give us anything. Symantics and Pragmatics research is way behind.
- Some crazy algorithm from AI textbook? No, most algorithms are designed for Computer Games and the rest are for specific applications which are not even remotely related to anything we are trying to solve here.
- Calculus, Linear Algebra, Bayesian Theory or any other crazy chapter from Advanced mathematics? Nope! We are over engineering the solution.
- Create a program to read a pre-populated facts database like Freebase and find the solution. You may succeed but I wouldn’t call that the right solution. That is not AI. You might argue what is wrong with this solution, the problem is with the way you will solve the problem. Your program should be the one to go to Wikipedia, find the name of painting and give it to you.
60+ years of research in the field of AI, Machine learning, Natural Language Processing, Cognitive Science and Mathematics and we still have no answer to such a simple problem of Information Extraction. There is a lot for us to understand in the field of AI before we over hype its potential.