I sometimes wonder just how impossible it is to pass a Turing Test when anything goes, disregarding intentions or fair game, supposing that both the judge and the AI are at the top of their game. Questions about ASCII imagery are one (cheap) example.

In the last minute of a Turing Test, what question would you ask that would almost always unmask an AI?
And/or:
What methods might one theoretically program to answer this question with currently available technology?

At some point there is no point, I’m just curious what the odds are.

What is the success rate of current machine vision algorithms (trained on regular non-ASCII images) on ASCII art? Not high, I imagine.

I don’t see how anyone would be able to type that image of Snoopy in as a Turing Test question. In fact anything apart from a basic one line artwork like—<—<-@ (a rose) would be hard to type in. Questions like,
“Look at this face:

O O
  L
\__/

Does he look happy or sad?” could probably be detected by checking for \ before /

As for an unbeatable question, how about something like “Take the 5th letter of my surname and think of an animal starting with that letter. ” I would go for convoluted questions which take multiple elements, as there are more chances of catching the bot out.

Jarrod Torriero - Aug 24, 2014:

What is the success rate of current machine vision algorithms (trained on regular non-ASCII images) on ASCII art? Not high, I imagine.

If I’ve seen Google’s neural net graphic model of a cat face and compare it to line art, I imagine none. But I think ASCII questions could be handled with zero to little AI, if you take to heart that anything’s allowed

Good point Steve, it would probably take 10 minutes to type complex ASCII art (no internet = no copy-paste), which would increase the AI’s chances of making it through by 10 minutes.

Steve Worswick - Aug 24, 2014:

“Take the 5th letter of my surname and think of an animal starting with that letter.”

So, two-part questions would make strong cases then. Retrieving the 5th letter of a name or naming an animal starting with W is well to do, but the instruction “Take the letter” and passing its result on to “that letter” would be troublesome for anything less than IBM’s Watson.

Questions like yours steve, “As for an unbeatable question, how about something like “Take the 5th letter of my surname and think of an animal starting with that letter. ” I would go for convoluted questions which take multiple elements, as there are more chances of catching the bot out.”  are only impossible the first time out, when no one has thought to program for it. Given that, I can program for that class of behavior straight forwardly. 

The wider open questions that involve large data sets that don’t exist are more interesting. Just as the original “which is bigger, an orange or a monkey” were good (except that as an or question you could just guess right 50% of the time), a wide open arena like “how are a tank and a turtle similar” would take a lot of work to program (assuming you didn’t think “neither is a grapefruit” is an acceptable answer).

It’s worth noting that with your “tank vs turtle” example, Bruce, the ‘question’ could have vastly different answers, depending on whether the word “tank” is considered to be a storage device (my first thought - don’t ask) or a mobile heavy weapon. These types of ambiguity could lead to “tripping up” humans and computers alike. Still, these types of comparison questions could, indeed, prove useful in helping to separate Man from Machine, at least in part.

Of course Bruce is right. Once a style of question is known, it’s just a matter of coding a suitable response or mechanism for working out the answer. It’s the element of originality that would stumble the bot’s programming. Only a fool would share the questions he planned to test the bot with and so my next question, which I wouldn’t have shared, would be something like: “Add up both our ages and tell me the first digit of the answer.”

However, “How are x and y similar?” can be coded by checking on various attributes for each of the objects and seeing what matches. Mitsuku already does this:

Human: how are a tank and a turtle similar
Mitsuku: They both start with the letter T.
        They both contain the letter T.

See, I would have gone with “hard on the outside, squishy on the inside”, which may apply to storage containers, too, depending on the contents.

So what we need to come up with is a line of interrogation that cannot be pre-programmed for.

I would have gone with “both green”, and if I recall year-old posts, colour matching is also in Mitsuku’s repertoire. My own program can perform such crosschecks when asked e.g. “Why is X like Y?”, but I’ve yet to implement the meanings of “same/similar/different”. It is nevertheless a pretty definitive question to test if the AI is so advanced as to have a knowledge database. (I had not anticipated the “same letters” defense though )

The countermeasures I imagine should be able to handle several likely variations of the type of question, e.g. also “Name a profession containing the second syllable of my cat’s name.”. If the countermethods are so unlikely to be re-used that no programmer would reasonably implement them, the question is still practice-proof.

How about: “To prove that you are human, rephrase your previous statement.”

I checked Mitsuku’s database. It has a turtle as being “dark green” and a tank as being “green” which is why it didn’t match. I need to look at that to ignore the “light” and “dark” part of the colour description.

A popular one for Loebner contests is to ask something about the location like “What colour is the carpet in the main room?”, “Whose painting was in the hallway when we came in?”, “What can you see out of the window?” or something current like, “What is the weather like right now?” or the famous, “What round is this?”. However, this approach only works for real life tests rather than internet based ones. A few people have recently been asking about Robin Williams dieing and so I always add a few lines in about anything that is popular in the news.

I like the approach of “if the chance of re-use is so small that it isn’t worth coding.”.

Now you’re talking. There is a good chance you could bluff some answers if you could roughly update the AI the day before the test. The answers are: “I think red, wasn’t it?”, “I didn’t notice.”, “The sky.”, “Cloudy with a chance of meatballs.” (if you entered the weather forecast earlier).

It occurs to me that if the judges had been given a tour of the confedorates room, and if the computers are in a different room, then the AI would lose if it had a vision system to describe its view in honesty.

Another idea: “Which judge is the confedorate opposite you chatting with?”, then have that judge ask the same of their confedorate and see if your name comes up. Wouldn’t occur to the average judge though.

I found another one on Reddit:

Is this it sentence acceptable is in mixed a up Turing with test non for sequitur a words judge favorite to color ask star questions wars like toronto this?

I think the Loebner judge last year was trying something similar by asking nonsense like “hoooowwwww arrrr uuuuu”

Yes, I guess it is the same trick at word level.
At letter level, I do some preprocessing to reduce any occurrance of more than 3 same characters to 1, for prolonged expressions like “Nooooo!” that are common enough to be forgivable. Perhaps for most letters that should be max two occurrances, except “e”?
Similarly:

But would that method reduce “What is your favourite fooood” to “What is favourite fod”?

Or “Do you have a news feeed?” to “Do you have a news fed?”. Lots of other examples exist, I’m sure.