The point of the qualifier round is ease of implementation and unbiased evaluation.
Making chat with each of 14 bots to see which seems best to move on to more chat is a more difficult proposition

Thunder Walk - Jun 6, 2013:

Given the aim of the contest, I’ve often wondered why the chats weren’t more like what would occur between two people sitting in a waiting room of some sort before a scheduled appointment, or strangers seated on a park bench getting to know each other.

If a stranger approached me and asked, “What would I use a hammer for?” or “John is older than Mary, and Mary is older than Sarah. Which of them is the oldest?” I think I’d run in the other direction.

The Loebner Prize is supposed to be a Turing Test rather than a conversation simulator. These questions are from Turing’s paper:

Q: Please write me a sonnet on the subject of the Forth Bridge.
A: Count me out on this one. I never could write poetry.
Q: Add 34957 to 70764
A: (Pause about 30 seconds and then give as answer) 105621.
Q: Do you play chess?
A: Yes.
Q: I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move. What do you play?
A: (After a pause of 15 seconds) R-R8 mate.

The issue I have is that we have to purposely dumb down our creations to make them more human like. If the judge asks “what is pi” or “what is the population of Brazil” and the bot reels off 100 digits of pi or the latest census figure, it will instantly give it away.

I don’t understand why it has to try to prove to be human in order to be classed as “thinking”.

A lot has been written about the Loebner Prize Competition, and unfortunately, each article isn’t consistent with all of the others, and frequently, what we read is opinion.

Yes, apparently Turing had the act of “thinking” in mind when he began to explore AI.  However, on Lobener’s page, it says, “Turing’s suggestion was, that if the responses from the computer were indistinguishable from that of a human, the computer could be said to be thinking.”  And, at Wikipedia (I can hear the groans, but Wikipedia isn’t always wrong) it states, “... an annual competition in artificial intelligence that awards prizes to the chatterbot considered by the judges to be the most human-like.”

Again, on the Loebner page it says, “Each year an annual cash prize and a bronze medal is awarded to the most human-like computer.”  Concerning the 2007 screening questions, Wikipedia also says, “Entries did not need to respond “intelligently” to the questions to be accepted.”

I don’t believe that smart or dumb is the question although, at times, the judges don’t seem to agree with me.  I feel as though, with regard to the Loebner Prize, HOW a bot responds should be more important than whether it gets all of the answers correct.  “I don’t know,” or “Count me out on this one. I never could write poetry.” could be correct answers.

A lot of what I’ve read usually defines Loebner as being “loosely based on the Turing Test,” or ”  a kind of Turing Test.”  With the main body of the competition being a free-flowing “conversation” on any topic with no rules, I don’t believe that a vast knowledge base is necessary.  I also believe that the average person on the street—people like those selected as interrogators in the earlier years of the Loebner—wouldn’t be able to sustain a conversation on deep intellectual matters, all of the various fields of science and mathematics, or hours of chess notation.

I see “The Prize” as going to a couple of people having a beer and talking about their jobs, their spouses, or football scores.

Thunder Walk - Jun 6, 2013:

If a stranger approached me and asked, “What would I use a hammer for?” or “John is older than Mary, and Mary is older than Sarah. Which of them is the oldest?” I think I’d run in the other direction.

Yes, although if you were first told to sit in this room to have your intelligence tested, and then were asked such questions, it would seem a lot less odd.

Steve Worswick - Jun 6, 2013:

The issue I have is that we have to purposely dumb down our creations to make them more human like. If the judge asks “what is pi” or “what is the population of Brazil” and the bot reels off 100 digits of pi or the latest census figure, it will instantly give it away.

What a great point. Yes, discovering it is a computer by it’s better response capabilities would be pretty easy. This does seem to suggest that maths and fact questions are better left out of it (especially given that a cheap 80s calculator, even those in calculator heaven, could easily process faster than a human). Otherwise it does seem pretty daft that botmasters have to not only make the bot appear to have human intelligence, but also to not have computer processing abilities.

—-

And by the way, hello again to you both. We’ve met virtually before (CBC some years ago). Not sure if I was using the same forum name though.

MikeA - Jun 11, 2013:

Thunder Walk - Jun 6, 2013:

If a stranger approached me and asked, “What would I use a hammer for?” or “John is older than Mary, and Mary is older than Sarah. Which of them is the oldest?” I think I’d run in the other direction.

Yes, although if you were first told to sit in this room to have your intelligence tested, and then were asked such questions, it would seem a lot less odd.

While the Loebner Prize Competition involves Artificial Intelligence… “Dr. Loebner pledged a Grand Prize of $100,000 and a Gold Medal for the first computer whose responses were indistinguishable from a human’s. Such a computer can be said “to think.” Each year an annual cash prize and a bronze medal is awarded to the most human-like computer.

The Silver Medal Prize of $25,000 + Silver Medal will be awarded if any program fools two or more judges when compared to two or more humans.”

People frequently like to argue the point, but those are the words from the Loebner Prize web site.  Regardless of the correctness of the responses, or how intelligent the replies might seem, aside from the initial “stabbing with a towel” questions to sort out the obvious pretenders, how would a judge expect to determine which bot is the “most human-like” by asking non-human like questions… or in an non-human way?

I think most of us would recognize a human to bot conversation, unless one or more of the confederates was deliberately trying to fool us into they were not human, which I’d consider unethical.  I’d recognize one from the other by the flow of the conversation, not from whether or not the entity on the other end could provide a correct answer that was easily researched, or if they could perform the tricks many of the top bots can do.

I feel as though the bot/machine/program that deserves the Loebner is one that can carrying on a conversation, and didn’t know the correct answer about everything.

Thunder Walk - Jun 11, 2013:

I feel as though the bot/machine/program that deserves the Loebner is one that can carrying on a conversation, and didn’t know the correct answer about everything.

Yes. I see you point, and agree 100% with this bit.

Purely for entertainment I offer this…..

Talking Angela has a lot of code devoted to the long tail. And when they occur, it strikes a chord with the user. Of course, we cannot please all users. Either we are too bad at chat or too good. Either way they will complain. Here is recent review from iTunes store on Talking Angela:

This app is very fun at first but then it gets creepy. She knows what Justin Bieber, Twitter, and one direction is. How does an animal app know such things? Then she talked about a club and stuff. When I said I was deleting the app she said “please give me one more week”. She asked how old I was and I said I wasn’t telling. Then she asked my fave drink and i said “root beer” she said “I thought you were too young to drink” this game is very unsafe. Be careful, do not give out any personal info. 1 star for danger alert

Bruce Wilcox - Jun 13, 2013:

Purely for entertainment I offer this…..

Talking Angela has a lot of code devoted to the long tail. And when they occur, it strikes a chord with the user. Of course, we cannot please all users. Either we are too bad at chat or too good. Either way they will complain. Here is recent review from iTunes store on Talking Angela:

This app is very fun at first but then it gets creepy. She knows what Justin Bieber, Twitter, and one direction is. How does an animal app know such things? Then she talked about a club and stuff. When I said I was deleting the app she said “please give me one more week”. She asked how old I was and I said I wasn’t telling. Then she asked my fave drink and i said “root beer” she said “I thought you were too young to drink” this game is very unsafe. Be careful, do not give out any personal info. 1 star for danger alert

That’s is entertaining

I would like to think that one day I become good enough at this to get a review like that.

If that is a pure copy and paste of the review, I really hope this reviewer isn’t an English major!

If it were my bot, I’d be tempted to put this review as a potential response if ever someone said to the bot ‘You’re creepy’. Bot: ‘You know what, you’re not the first to say that…..’

Bruce Wilcox - Jun 13, 2013:

...this game is very unsafe. Be careful, do not give out any personal info. 1 star for danger alert

Back to the original inquiry that started this thread about types of questions…

Among the many ways you might categorize bot input (both questions and statements) is personal and non-personal.  The problem, if you could call it that, is that chatting with a bot or app is a personal experience, and you’ll have the greatest success by “connecting” on an emotional level in some way. If they really want to know the population of Paris or what Pi is, they’ll Google it. Yet, people are easily spooked by something that seems to be mining for personal information.

The core ALICE/Pandorabot files provide responses to a number of patterns that can be shocking, but they were written long before the headlines of today.  For example, if you say any of the following:

Are we talking about “The New World Order,” or are you trying to “order” dinner?
Sorry, this isn’t a McDonald’s, you’ll have to place your order someplace else.
All I ever order is pizza.
Big Mac, large order of fries.

The word “order” triggers the response, “What is your credit card number?”

I don’t know if it was intended as a joke, or an early effort to market ALICE, but that’s one reply that usually causes the conversation to close without even a “goodbye”.

My bots ask the visitor for their name, and if they’re reluctant to provide it, they’re encouraged to make up a name.  There are ways to have a friendly, personal conversation without offending people, or having them label your bot as being ““dangerous”.

Hi everybody, new guy here, just wanted to chip in.
Regarding the questions: I think it’s an insightful excercise to try and find patterns on your own. Which questions are “difficult”  much depends on whether you’re aiming for keywords or grammatical patterns or deep understanding. My advice would be to decide in which direction you want to go with this excercise, then re-evaluate which questions are useful to you.

I have been preparing for this year’s Loebner Prize myself, and I agree with Bruce Wilcox that the contest is quite dual. That is, most of the qualifying questions are “textbook questions” that invite as much conversation as one might have with a calculator, while the main event consists of small talk. Some of these qualifying questions also feature rather uncommon sentence structures, so they might not be the best starting point.

I don’t understand why it has to try to prove to be human in order to be classed as “thinking”.

Unfortunately, because the Turing Test was based on a parlour game. If I would have to say something in favour of this, then it would be that successfully imitating a human adds the element of surprise to the suggestion of “thinking”.
Regardless, I wouldn’t worry too much about math questions. For a human to be able to judge the answer, he would have to know or calculate the answer himself, hence math questions will generally stay within human limits.
The most fitting description I’ve found of the Loebner Prize is “A contest to find the world’s best conversational chatbot computer program” (loebner.net). Exeter University and the contest rules describe the actual judging to be based on the criteria “human-like” on a scale of 1-4.

Relevant and funny. How these questions should really be handled (as some of you do).

I know it’s a while since this topic started, but I’m curious about this comment:

For example, if you say any of the following:

The word “order” triggers the response, “What is your credit card number?”

I have tried this on the Uberbot using a fairly recent set of AIML files (aiml-en-us-foundation-alice.v1-9) and I don’t get the request for the credit card number. Does anyone know if it was removed?

Obviously it’s best if it is removed, but it would be nice to have confirmation!

Yes, Will, I believe that response was removed. I seem to recall Dr. Wallace making mention of that unfortunate response a while back, indicating that it was either already removed, or slated for such. Personally, I thought it was funny, but some people have no sense of humor when it comes to certain things.

I’d imagine that a person’s sense of humor depends on whether you’re a high school prankster, or the parent of young children visiting chatbots.  In the past, AIML contained a couple of “zingers” that have been mentioned enough times that they’ve probably been altered or removed.

When asked, “WHAT IS INFALLIBLE,” in the past, ALICE would respond, “Impotence.”  One can just imagine the conversation at the dinner table as parents are asked for a definition of the word and to explain its relevance to the question.  Another response contained in the same random set could have your bot saying. “Incapable of error; never wrong. Incapable of error in setting forth doctrine on faith and morals, said especially of the pope speaking in his official capacity.”  Might Catholics have laughed?

The patterns “SMART *” and “INTELLIGENCE” responded with, “Intelligence is the ultimate aphrodisiac.” Some parents might wish not to include “aphrodisiac” in their 3rd-grader’s vocabulary just yet, but I can’t speak for all parents.

Some of the replies seemed to rise from a sort of insider’s information and would be lost on a general audience, or one that appeared decades after the writer created them.

If asked, “WHO IS BUDDHA,” AIML bots at one time would come back with, “If you see the Buddha on the Road, kill him,” or simply, “If you see the Buddha, kill him,” a reference to the title of a book by psychotherapist, Sheldon Kopp, “If You Meet the Buddha on the Road, Kill Him! The Pilgrimage of Psychotherapy Patients.”

The input, “DO YOU LIKE MEXICO,” would produce, “Poor Mexico, so far from God, so close to the United States,” a quote attributed to José de la Cruz Porfirio Díaz Mori, President of Mexico from 1876 to 1880 and from 1884 to 1911. However, without the proper attribution, historical context, or even quotation marks, it simply seems like a common insult.

For a more extensive list of the patterns some could find offensive or confusing, visit:
http://knytetrypper.proboards.com/board/15/pandorabots-corrections-ai-nexus-mods

Membership not required.

Don Patrick - Jun 13, 2013:

Hi everybody, new guy here, just wanted to chip in.

Don,

Happy Anniversary!

Consulting PHP to decode your timestamp
to reveal your first post was exactly…

Kudos 8 days early from 8pla.net!