Hi Thomas and AccessibleSoft,

As Grulex guessed the Q&A API is only a small part of the Jeopardy system. It is limited to answering with a single answer type: passages extracted from the provided content. In the coming month, we will release a series of services that will allow developers to build the full Jeopardy stack. In particular, allow developers to answer factoid-type questions as well as select the proper answer-type pipeline.

So no "national security" or "cheating" theory needed ;-). This is just an engineering problem. We are trying to release the research assets in consumable bytes. We've found that passage answers provide the most value to our early customer and that's why we started there.

Now we realize that these two examples, based on some limited fixed content and limited training (they will get better overtime as we collect more questions), aren't very useful. They are really meant as a quick preview of the APIs. The real value comes to those who can provide their own content and do their own training - a functionality we'll open to everybody early next year.

Jerome

Hi Thomas,

The Travel and Healthcare corpora that you are working with have not been fully trained. Over time, you will find that these publicly available corpora will become more accurate as we have the opportunity to train Watson with questions from the community.

The version of Watson that we are using is domain independent. Think of it as a child that needs to be taught. Of course this child has been pre loaded with a lot of content!

Domain adaption requires training (educating) Watson on a set of question/answer pairs. Generating question/answer pairs for both the travel and healthcare corpora would have been a very time consuming task for the small team creating the corpora.

The questions asked by the community and provided by Watson are captured in Watson's adaptive learning component. When the Feedback API is used, the relevancy rating provided by the user is captured and aids in training Watson.

Right now, you may receive answers that are not very relevant, but overtime, Watson will learn.

You should check the start of first episode of Jeopardy when it explains what is watson!!

Think of this as a three part solution for you....a large cloud auto-scale environment for this super computer, An intelligence engine which can 'understand' and a end user API -- this question answer api.

Now we need to teach this middle thing. For example If we do a big data analytics, we will be writing those map-reduce jobs or statistics analytic rules/queries for R. Similarly we need to train the system to 'understand' in the way we like it to understand. It's a wonderful child we need to put through education for a engineer, designer, chef, pilot etc etc. I would be surprised if Watson is already trained for some case as we then trouble to use this for some completely different domain specific solutions.

so it work once you implement for the domain and for your customer. Little information is useless and this is what you are facing. Try to train for hundreds of question and grow your content. It's like giving books to watson to learn and then taking exam and giving marks!

Thanks and great points. To your point about training, we are in the process of actually doing that for travel and healthcare. The QA service is currently trained with a minimal set of ground truth questions and corpus. Also, it can only answer certain types of questions that are more descriptive that factoid. No support for polar questions either.

By using the Feedback API (see documentation for QA Service) we are hoping to get more real world questions and hope to get better results with training. Having said, there is more than just training that is going to do the trick. For instance, in Jeopardy we had to induce other resource types such as glossaries, lexicons, knowledge bases for open domain. I call this Domain Adaptation.

From your example, to put an engineer through a test he need to be able to understand the nuances of the domain by learning from different sources, being able to assimilate the jargons, acronyms, relations between various subjects etc. This also has not been done for this QA Service. Over the next several revs, we will be adding capabilities that would allow the QA service to come pre-trained with various resources and have the ability for users to upload their content / resources to train and test.

Hope this helps.

Hi Chris,

thank you for answering, but given my example, I don't understand how Watson can even learn/improve. No answer qualifies for a "learning process"...or the process is simply too long. (I've tried multiple questions). E.g. you might also try "Do I need any vaccinations in the Dominican Republic?" (since there was a sample question for visas already) Since most of the questions I tried might have been thrown at Watson in Jeopardy as well, I don't understand why you don't simply train Watson with Wikipedia and several encyclopedias. Shouldn't it be able to provide significant answers to easy questions at least? Or if there are really question/answer pairs needed, maybe to feed Watson with Trivial Pursuit or Jeopardy games? Why don't you offer to ask the (improved) Jeopardy version of Watson some questions. In the video we all know the answers always were one clear sentence. As a decision maker it is rather risky to build on top of Watson after testing it and having the expectation, that IBM improved this over the last few years. I really would love to see Watson working in one given context representing the 2014 state of the art. Please don't get me wrong, I want to see this thing happen and understand the difficulties of asking highly qualified scientists and engineers to "train" Watson instead of improving him/it. But unless the community learns, that Watson is on a good way to "pass the Turing test" in near future there is a gap. As soon as you can build a bridge here things should work in favor of IBM.

@60R7_Chris_Madison

I don't think Watson won Jeopardy by using only the Q&A API that you guys are testing. It might be that they used something like this just to generate thousands of hypotheses to narrow down the search area. The Developer Cloud has other Watson services as well. That is my guess. Many big companies use Watson. They would not be paying for something that is not worth it. Personally, I really need a solution like this to build an AI system and I really hope it works better than this example (watson-qa-demo.mybluemix.net).

The 2011 version of WATSON appeared to have a "natural language processor" to identify the question and the nuances of human language within the query. The 2014 version available to developers certainly does not have the same processor. It has a very weak string match ability that any novice programmer could have written. It does appear to have some vaguely related content, but answers are not easily identifiable or retrievable by the current software implementation. Training it could be an even bigger waste of time, because the NLP probably will not find that answer as it did in 2011.

Yes, Thomas poses some very good points. Also look as his example question "Do I need any vaccinations in the Dominican republic?". In my software I break the NLP down, and first identify the format of the question. It is polar implying a yes or no answer usually. Then the subject, is "I", and the action "need" then the preposition "in" and place Dominican republic implies a place. The AI has what it needs here to narrow the search right down. This is baby stuff. In my software the encoding of factoids is kept short and non-ambiguous for better retrieval. Humans speak with what is known as "the principle of least effort", as long as the brevity does not muddle the meaning. This is what is expected of WATSON, or any serious AI. When I ask WATSON: "Is a stroke always fatal?" A long factoid about rabies is not what I want to hear. lol

Based on whatever time I invested so far, I am disappointed at least with Q&A API / Service. I did spend time with Twitter bot app and tried searching nearby "Indian restaurants" (with my "location enabled" properly) and the Watson recommendations and confidence levels were really disappointing. Is existing version not ready for beta testing as well? If so why did IBM release this to developer community? Really curious to know if it is Watson or am I missing something?

Regards

Swarraj