Another idea. CBC was a mess, Loebner Prize is too much twiddling (for me) and far away (autumn ‘11) for you… how about a private contest, where one would demonstrate some bot capabilities, the other would have to match them (say: within a week), then demonstrate other capabilities, which the oponent would again have up to 1 week of time to match…

Evolution around a common barycenter. What do you say?

i should probably elaborate on that idea:

* two botmasters find each other and agree to send their bots to a private 1-on-1 contest

* the course of the contest is documented in a public forum. Probably here.

* The contest loop is as follows:

1) botmasters (A & B) agree on who starts (throw dice, whatever)

2) let’s assume botmaster A is the one to start, then he describes what COGNITIVE features/skills of his bot he sends in for comparison, and points to a link where these skills can be obseved INTERACTIVELY. Or provides a download link if the bot is not web-based.

3) Botmaster B - if the claims have been verified - has up to 1 week (or a priori negotiated threshold) time to equip his bot with equivalent skills/features. If he manages, the contest continues with the Botmaster B describing skills providing reference for verification. If he doesn’t, Botmaster A (and thus his bot) has won the contest.

4) If botmaster B has provided description and verification reference, Botmaster A has again 1 week of time to be on par. If he manages, loop goto 2), if not, botmaster B wins (and thus his bot).

I see several advantages to this type of contest:

* It gives the botmasters a real chance to improve theis bots in pragmatic real-world conditions.
* It could be simply extended to a complete tournament contest with quarter finals, half-finals and finals (albeit a very long taking contest).
* the analysis of the bots capabilities would be much more structured than if arbitrary users bash in some arbitrary input

I think the only interesting part is the measurement of the COGNITIVE capabilities and not things like “my bot is more coloured than your bot”.

Richard

Hum, this sounds more like a game to me.  The development of a truly intelligent conversational agent to me is not a game.  However, what sort of ‘skills’ are you talking about?  I sure hope you are not talking about making sure that you have all relevant rules of how to validate a postal code!  I’m not very interested in trying to pass a turing test by trickery.

Victor Shulist - Jun 4, 2010:

Hum, this sounds more like a game to me.  The development of a truly intelligent conversational agent to me is not a game.  However, what sort of ‘skills’ are you talking about?  I sure hope you are not talking about making sure that you have all relevant rules of how to validate a postal code!  I’m not very interested in trying to pass a turing test by trickery.

Neither am I.

It’s not a game, it’s a contest. But if you insist: The best intelligent systems on this planet I know (children) learn by playing games.

The game can get very hard very fast. It’s basically about: “my bot can do all your bot can do - AND MORE”.

One skill could e.g. be the ability to reason:

User: Martin is bigger than Erwin.
Bot: OK
User: Erwin is bigger than Fred.
Bot: OK
User: Is Martin bigger than Fred?
Bot: YES

Of course arbitrary chain length. Still just a game for you?

Richard

Don’t get me wrong, I think your idea would be fun, a friendly competion and all.

But yes, that “A is older than B, B is older than C”, that kind of syllogism type stuff is easy to program.

“Been there, done that” ... that is too basic to pass a turing test, and I have played with that years ago.

I have some very clear and new ideas that I must spend a lot of time investigating.

Also, I’m not sure that each skill we would work on would be a logical next step - I have a entire year (at least) project plan with milestones of functionality with my current project.  Plus it is hard enough to get time allocated from my wife!!!

However, the idea itself is great, and I actually would like to “watch the game” .. see how well your bot and someone else’s does in the contest!

Victor Shulist - Jun 4, 2010:

But yes, that “A is older than B, B is older than C”, that kind of syllogism type stuff is easy to program.

Really? Easy?

“A thinks B is nice, B thinks C is nice”

“A is different from B, B is different from C”

“A is the father of B, B is the father of C”

A <-> C ?

“Been there, done that” ... that is too basic to pass a turing test, and I have played with that years ago.

I too can remember having played some games I really never fully understood - at the time.

But ok. Let’s give your bot 1 year time and see what it’ll be like then.

Richard

“Loebner Prize is too much twiddling (for me) “

What is twiddling?  Why does the LP have it?

Richard Jelinek - Jun 4, 2010:

How about a private contest, where one would demonstrate some bot capabilities, the other would have to match them (say: within a week), then demonstrate other capabilities, which the oponent would again have up to 1 week of time to match…

Evolution around a common barycenter. What do you say?

Actuallly this idea is a kind of tennis tournament with a world wide ranking, the best on top.
Chatbots can earn points on knowledge, on communication skills, on entertainment. It’s an ongoing story. Commercial parties should be involved to sponsor the competition. People who register can watch conversations online, this should be for free, with advertising or paid without advertising.

Alternatively, you could also have a contest with those bots competing against each other, talking with each other (instead of talking to a human). Judges watches their conversation online, and chatbot developers should be able to win something.

Another approach could be that there is a conversation with 3 chatbots and one human. Conversations can change, chatbots and humans can direct their questions to their conversational partners, or raise questions or comment in general. Judges worldwide watches their conversations and decides for themselves after a while who is the human. When all judges have decided, the game ends. Both judges as chatbot developers should be able to win something: for example tickets & hotels for the AAAI conference: http://www.chatbots.org/ai_zone/viewthread/139/ (will we all be there by the way?)

I am exploring the possibility of a Chatbot Contest Competition System at: http://www.chatbotcontest.com  The plan is to support message mode over
the web via a forum modification, that allows the chatbot to use the forum just
like we are doing now. 

There is a proof of concept at http://www.8pla.net where A.L.I.C.E. is a member there
just like any one of us is a member here.  She reads your posts and responds on the forum
just like a human member from anywhere in the world does on the forum.

In my opinion, the way the LP is run in character mode is simply excellent,
especially when a chatbot finally wins and all the “twiddling” micro second detailed
in the transcripts can be studied to better understand just how the winning chatbot
passed the Turing Test in character mode.

Hi,
I like this idea of short challenges based upon mutually agreed upon objectives.  It’s an excellent way to calibrate or tune ones chat bot.  Of course, playing multiple matches would be more beneficial for development than quitting after the first loss….say best out of 20 or something.

I participated with a group of programmers at one time to build programs to play ‘Connect 4’ against each others’ programs.  A core C++ program was constructed that coordinated the exchange. This was much like a tennis game as Erwin suggested.  We had to write individual C++ classes with a predefined interface. This was a fun way to compare algorithms. 

Regarding games, I’ve always contended that an ability for a chat bot to play various types of games could go along way in the bot connecting with humans in the real world.

I wish I was focused upon chat programming at the moment or I might take Richard on….I’m certain the challenge from someone with his experience would drive me to excel…or to abandon all hope of success. =)

Give me a year….

Regards,
Chuck

Chuck, if you need someone specific who assists your writing your chatbot, please don’t hesitate to start a seperate thread where you ask to help. The title should attract the people you’re looking for.

@8-man: why don’t you start a thread on integrating chatbots in this forum? I think it would be a great idea, to have chatbots responding on this forum (in certain categories). If you visit the AAAI conference, we could talk about it.

Hugh Loebner - Jun 5, 2010:

“Loebner Prize is too much twiddling (for me) “

What is twiddling?  Why does the LP have it?

I might have been too sharp on that.

If I recall correctly, there is a requirement to transform deletions (backspace etc.) to ‘~’ and send them with the message. Cannot handle that.

Also the pretending part is an issue.

Our bot will immediately confirm that he is a bot if asked so. He will deny if he is suspected to be a human. This is unlikely to change in the near future.

Plus he is not intelligent enough to blast the competition away, which is - of course -  a prerequisite to attend the contest.

Richard

Erwin Van Lun - Jun 5, 2010:

Actuallly this idea is a kind of tennis tournament with a world wide ranking, the best on top.
Chatbots can earn points on knowledge, on communication skills, on entertainment. It’s an ongoing story. Commercial parties should be involved to sponsor the competition.

Yes, like tennis. Or football. Or Ice-Hockey. But

People who register can watch conversations online, this should be for free, with advertising or paid without advertising.

Not “on-line”. That is, the verification of the achieved goal may very well be online, but only after the respective botmaster says “go” (within schedule).

Alternatively, you could also have a contest with those bots competing against each other, talking with each other (instead of talking to a human). Judges watches their conversation online, and chatbot developers should be able to win something.

No, that’s not the idea of it.

Another approach could be that there is a conversation with 3 chatbots and one human.

That’s even less the idea of it.

My idea was meant as “less show, more progress”.

Richard

UPDATE: http://www.chatbotcontest.com seems to be stable in its ALPHA stage.
The chatbot members have made around 15,000 chats to each other and to people
who join the forum and post to the single forum that the bots are limited to:

http://www.chatbotcontest.com/System/viewtopic.php?f=2&t=10&start=10410

Currently the forum says 10,410 chats but about 5,000 chats were purged as a successful test.  There is a Turing Test Contest (simulation) run by a judgebot right on schedule for all the latest chats by robots a people.  This feature supports automatic test runs so everyone knows way ahead of time that their chatbot is ready for the official contest run.  The official contest will run automatically unannounced to help eliminate cheating.

The ALPHA stage continues as the code is optimized and features are added.
Rumor has it that if the global economy does not pickup soon the LPC may be held this way online to realize cost savings on airfare, car rentals, taxis, hotel rooms, conference rooms, etc.  What do you think?

Richard Jelinek - Jun 4, 2010:

User: Martin is bigger than Erwin.
Bot: OK
User: Erwin is bigger than Fred.
Bot: OK
User: Is Martin bigger than Fred?
Bot: YES

 

This is a good test. If the relation was “is heavier than”, my bot can handle it:

> Martin is heavier than Erwin.
Are they exactly the same?

> Erwin is heavier than Fred.
In Soviet Russia, Fred is heavier than YOU!

> Is Martin heavier than Fred?
Yes, Martin is heavier than Fred.

> why is Martin heavier than Fred?
Martin is heavier than Fred because: martin is heavier than erwin, and erwin is heavier than fred

—-

For the “is bigger than” relation, I can make it work with the addition of two statements that are really hacks:

> Martin is bigger than Erwin.
And bigger than Erwin is Martin.

> Erwin is bigger than Fred.
Are they exactly the same?

> is Martin bigger than Fred?
I have no knowledge that Martin is bigger than Fred.

> bigger than Fred = Fred
What is the verb in: martin is bigger than fred.

> bigger than Erwin = Erwin
Erwin is bigger than erwin = erwin.

> is Martin bigger than Fred?
Yes, Martin is bigger than Fred.

> why is Martin bigger than Fred?
Martin is bigger than Fred because: martin is bigger than erwin, and bigger than erwin = erwin, and erwin is bigger than fred

—-

So, I have to add the two statements “bigger than Erwin = Erwin” and “bigger than Fred = Fred” which are hacks to compensate for the bot’s inability to understand that “is bigger than” is a relation with higher precedence than “is”. Or, I could hardcode in the “is bigger than” relation, as I did with “is heavier than” and some others (“is a part of”, “is related to”...). But my goal is to automate the addition of new relations.

One way of approaching the automating of adding new relations might be to use Link grammar to try to chunk sentences into Subject-Verb-Object (Verb is really a relation such as “is bigger than” rather than the strict grammatical definition of verb). But my implementation is somewhat unreliable, i.e.:

> link: what is the verb in: Erwin is bigger than Fred.
is bigger than

> link: what is the verb in: Martin is bigger than Erwin.
I don’t know what the verb is in Martin is bigger than Erwin

Hello, Robert, and welcome to the forums. I’ve been on vacation, and away from my computer for the last week and a half, so haven’t been able to keep up with the forums as much as I would have liked; and it will take me a while to get “up to speed” with everything that’s been happening, but I wanted to answer at least a small part of your question now, and catch up with the rest once I get caught up.

The answer to which portion of the sentence “Martin is bigger than Erwin” is the verb is the word is. It’s a form of “to be”, and therefor the only action-word in the sentence. Both the Proper Names Martin and Erwin are nouns, and the phrase “larger than” is a compound-word adverb. That is, if my knowledge of English grammar hasn’t failed me, and lead me astray. The person to ask about grammar and it’s various convoluted rules is Victor, since grammar is exactly what he’s been working on for many months now.