http://www.extremetech.com/extreme/161383-artificial-intelligence-has-the-verbal-skills-of-a-four-year-old-still-no-common-sense

ConceptNet 4, an AI developed at MIT, was recently put through a standard IQ test given to young children. The result? ConceptNet 4 is a slightly odd four-year-old child.

The article describes the strengths and weaknesses discovered in ConceptNet during the testing process. One of the people commenting on the article suggested that some of the problems could be overcome if the software could experience things for itself, if it could become embodied.

What if this could be accomplished by putting more emphasis on implementing empathy than intelligence. There are tests for that now too. Has anyone attempted to facilitate empathy in their software?

Sounds like an interesting test. A couple of the questions were:

What do apples and bananas have in common?
Why do we shake hands?

Does anyone know of a sample paper online? The only ones I could find were the full testing kit at $1200!!

As regards to empathy, I believe Mohan Embar (Loebner Prize winner 2012) is a strong advocate for this approach. He sometimes visits the boards but nort very often.

$1200 for a “testing kit”? Are you sure that wasnt part of the IQ test?

V

I dunno, the examples the article mentions are just knowledge, not feats of intelligence. It could know that shaking hands is for greeting in the same way that it knows that saxophones are for music. They suggest however that such facts are derived at through “rational judgement by leveraging implicit facts”, and while I disagree, that sounds exactly like what an inference engine does.
Children learn these facts by seeing when people shake hands, or feeling when ice is cold (and being told that it’s zero degrees). Seems to me you’d just need sensors, but then it would still take a childhood to learn so that wouldn’t be very efficient.

I found a few more questions they asked it:

Where can you find a penguin?
Why do we keep ice cream in the freezer?
What are pancakes made out of?
What does a snake and an alligator have in common?
Where can you find a teacher?
What is a saw used for?

Seems like the sort of standard Loebner questions we are all used to. It would be interesting to see how our bots fare against the same test. I ran these by Mitsuku and got them all correct apart from the freezer question.

lol @ V

I don’t know what pancakes are made of… Does that mean I have a low IQ?

To be honest, I doubt many 4 year olds would know what a pancake was made from.

RE: Pancakes LOL

If RICH has knowledge of the object , it will answer “made of” interrogs with a direct response (list of materials etc..). But my favorites are the answers that come up when the logic\KB modules do not produce a direct response

Teacher: What are pancakes made out of?
Vince: Pancakes like all things, would be made up of subatomic particles. These are collectively referred to as Quanta. That is the correct answer by the way.
Teacher: What does a snake and an alligator have in common?
Vince: They are both objects that were mentioned in your interrogatory. . . . BWA HA HA (whine all you want that answer is 100% correct)
Teacher: What might a car and a truck have in common?
Vince: They both have the dubious distinction of having been mentioned in this inane interrogatory.

I agree with Patrick. The real goal should be “leaps in logic” and a “world view” (sensors etc) as well as a temporal sense may well prove to be the key.  For example, knowing that an alligator and a snake are both reptiles is fairly simplistic although when you consider what goes into arriving at that “simplistic” answer reliabily, is still pretty impressive. Being able to query the bot for a deeper association where the bot is able to detect that both have been used to make articles of clothing, is a further step. Being able to infer given this knowledge whether something else might be used for clothing , is getting pretty close to home. Being able to factor aesthetic values and formulate independent opinions that might qualify as “personal taste” on clothing and you begin to have something that might qualify as “intelligent”.  Whatever that is

V

Subatomic particles is a great answer

By the way, is this, in fact, a test for children up to the age of 4, and was age 4 therefore the highest age one could attribute to the computer?

Don Patrick - Jul 16, 2013:

I don’t know what pancakes are made of… Does that mean I have a low IQ?

No…it simply means that we would rather not have you make our breakfast!

Ok, that’s it. You’ve gone and made me hungry! Off to go get breakfast!

Pancakes for breakfast?
Revisiting this topic, I read the test was for children age 2 - 7, and Conceptnet was only tested on the verbal parts of the IQ test.

I think this says it best:

Sloan said ConceptNet 4 did very well on a test of vocabulary and on a test of its ability to recognize similarities.
“But ConceptNet 4 did dramatically worse than average on comprehension—the ‘why’ questions,” he said.

Commonsense has eluded AI engineers because it requires both a very large collection of facts and what Sloan calls implicit facts–things so obvious that we don’t know we know them. A computer may know the temperature at which water freezes, but we know that ice is cold.

“All of us know a huge number of things,” said Sloan. “As babies, we crawled around and yanked on things and learned that things fall. We yanked on other things and learned that dogs and cats don’t appreciate having their tails pulled. Life is a rich learning environment.

Of the questions Steve retrieved, only the “why” and “in common” questions require more than simply knowing a fact. The “in common” question is a matter of comparing two lists of facts and extracting the ones that match, which computers do well enough.
That leaves “Why do we shake hands?” and “Why do we keep ice cream in the freezer?” as the only questions requiring “implicit facts”, which are usually learned by witnessing two observations, such as that ice melts when ice is out of the freezer, and assuming a connection between them. It seems our magical “common sense” is mainly a matter of knowledge due experience.

Instead of encyclopedic data or sensory observations, suppose an inference engine were to read novels, records of life experiences, wherein it says “Tom took the ice out of the freezer. It melted.”, or “He greeted him with a firm handshake.”, it should be able to establish the same connections. I’ll just put that on my to-do list.

That’s gonna require a lotta books, mate.

As a general rule, popular published works of literature, both fiction and non, leave out exactly the sort of “day to day minutiae” that you’ll be looking for. Instances of such tiny details of life (the “little things” that we all learn from or about as infants/toddlers/small children) also happen to be mind numbingly boring when found in the context of a literary work, so they’re not only left out, they’re also “glossed over” whenever possible. This is a large part of why such books or articles are so popular.

Please note that I’m not trying to discourage this avenue of data collection… I’m just trying to help quantify the scope of it.

Good point, Dave. I’ll tell you what I had in mind: Most of the children’s books I read back in the day described ordinary countryside childhoods, often describing in great detail how exactly a leaf fell from a tree or what the texture of a rock looks like or how cold water feels after you fall into a pond, which implicitly suggests that ponds contain water and are larger than a human, etc, etc. Most literary texts I read are rich in trivial information to ‘paint a picture’. Possibly we read different prose
I agree it would take a lot of reading (literally a childhood), but it seems to me a more likely source to find everyday facts described than educational sources of information. And considering even my little AI’s text processing speed, reading one 200-page book shouldn’t take more than 2 minutes, so I don’t think the quantity would be so much an issue, theoretically. If I had an abundance of trustworthy novels in digital text format and if my inference engine were enhanced by several months of work.

I wonder how did they “pass” the test. Can ConceptNet recognize the pictures? Or can it read the questions in natural language?

It was only asked the textual questions of the IQ test. Conceptnet can read normal English.

The verbal portion of the test asks questions in five categories, ranging from simple vocabulary questions, like “What is a house?”, to guessing an object from a number of clues such as “You can see through it. It is a square and can be opened. What is it?”

To answer a question from the test, like “What do you wear on your head?”, ConceptNet searches its database for the object that is most closely related to the pair “wear” and “head”.

ConceptNet is a freely available commonsense knowledgebase and natural-language-processing toolkit which supports many practical textual-reasoning tasks over real-world documents right out-of-the-box (without additional statistical training) including

  topic-jisting (e.g. a news article containing the concepts, “gun,” “convenience store,” “demand money” and “make getaway” might suggest the topics “robbery” and “crime”),
  affect-sensing (e.g. this email is sad and angry),
  analogy-making (e.g. “scissors,” “razor,” “nail clipper,” and “sword” are perhaps like a “knife” because they are all “sharp,” and can be used to “cut something”),
  text summarization
  contextual expansion
  causal projection
  cold document classification
  and other context-oriented inferences