Advance epub: Facebook AI Research developed a set of tasks that they believe are a prerequisite to full language understanding and reasoning.  The article includes a comprehensive set of testing question types for QA; from ‘Basic Factoid QA with Single Supporting Fact’ all the way to ‘Reasoning about Agent’s Motivations’.  If nothing else, it makes for a good check list of areas to have developed for your bot.

Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

If this thread’s title were true, then no scientist currently in the world is going to solve AI , so yeah, let’s go with “checklist”.

More accurately this is a list “To measure progress towards [the goal of] building an intelligent dialogue agent.”, at which Facebook and I clearly differ in priorities. My program can make inferences and construct opinions from multiple supporting facts, but in all 3 years of my work I never seriously considered having it answer “How many apples do I have?”, because there is no doubt in my mind that computers can count. That said, I’ll write that down somewhere in case I get bored some afternoon.

Don Patrick - Aug 17, 2015:

If this thread’s title were true…

It is a quote from the article itself, click-bait click-bait if you will. “To measure progress towards [the goal of] building an intelligent dialogue agent.” sounds so… boring, and FB knows it (I mean, they literally “Know” it). 

An intelligent dialog agent will likely be, at least, the front end of any human accessible AI. Therefore developing an intelligent dialog agent seems to be the practical stepping stone to creating a competent and intelligent artificial assistant.

Don Patrick - Aug 17, 2015:

My program can make inferences and construct opinions from multiple supporting facts

That is pretty cool, but also becoming more routine (even my AI, which is online, ehem, has those functions).

Don Patrick - Aug 17, 2015:

...in all 3 years of my work I never seriously considered having it answer “How many apples do I have?”, because there is no doubt in my mind that computers can count.

Yeah, but not intuitively or intelligently- I am sure my computer can not count how many apples I have unless I very carefully craft my query:

“1+1=”, or even
“$ans = count(user_id_ppn,vb.poss_have,thing_apple,temporal_now)”

are much different than

“how many apples am I holding right now”,

which, if known, could also be used to make inferences/suggestions about what you could do with those (would especially cool as a persistent local system too).

Anywayz, the original link has a nice list of types of testing questions for QA with examples (and also links to complete lists of human readable test sets for each question type) that any bot dev will find useful.

which, if known, could also be used to make inferences/suggestions about what you could do with those (would especially cool as a persistent local system too).

This illustrates my point though, that compared to the ability to make suggestions on what to do with two apples, it is but a trivial task to program for the input “how many X * Y?”, do a count(), and produce the numerical answer.
Reversely, if one uses this as a checklist, one might be tempted to reproduce only the trivial ability to answer these particular questions, without the underlying flexibility of intelligence. It’s a decent list though.

The thing I like about the Face Book AI tests, is that they are small, focused tests. Unlike a Turing test or many of the open chatbot competitions, you can focus on a single aspect of the AI. I believe these types of exercises are much more helpful than wide open challenges (like the Winograd schema).

Don Patrick - Aug 17, 2015:

but in all 3 years of my work I never seriously considered having it answer “How many apples do I have?”, because there is no doubt in my mind that computers can count. That said, I’ll write that down somewhere in case I get bored some afternoon.

Although computers count well, the challenge is to take various ways people express potentially calculable inputs and use AI/NLP to convert it into a form a computer can understand. I put a lot of work into Skynet-AI’s math functionality and you may be surprised how non-trivial it is to handle even 4th grade math once you delve into it.

The Allen Institute for AI’s flagship project ARISTO is organized around a collection of QA
tasks derived from increasingly difficult exams.

Elementary School Science and Math Tests as a Driver for AI: Take the Aristo Challenge!

Please note that I said “compared to”. I don’t mean to undervalue your hard work and know that math linguistics and their ambiguity takes effort. It took me 5 days to program for processing e.g. “How much is three thousand and five x 246 divided by twenty-four”, but on the other hand it took me 500 days to program for regular English language and inference abilities. So math makes up 1/100th of my problem space.
Anyway, carry on.