Hi all

I have to at all costs design and implement a chatENGINE from scratch in strictly 7 months. (Please dont refer to an old post of 2010 of 4 pages)

Similar questions I found on forums required more information so that is provided under:
1) I have real dream to make a chatbot worth talking.
2) I have explored AIML, chatbos, how they talk etc
3) I have explored grammar rules a bit.
4) I am skilled at programming C/C++, C#, Java, SQL, SQLServer, Oracle, ASP.NET
5) I can devote 5-6 hours daily for 7 months
6) I have great learning ability
7) Can undergo complex things with patience
8) Have programmed for 18 hours continuously (Cant do more than this, thats limit for me)

I want to make a chatbot for loebner prize I mean to say no specific purpose Just a College project.

Should I use AIML type approach, or Can I go with writing a grammar parser?
Another limitation is the hosting server might not be that powerful as I have to bring it online.

Can anyone limit the scope for me so that I dont overestimate myself, however usually I dont

Thanks to all in advance, You people are small still great community!

PS: Many times in several forums my questions have been closed for insanity. This community is the last resort for continuous 7 months. btw what does (PS stands for)

1) I have real dream to make a chatbot worth talking.

Great! You’ve come to the right place.

2) I have explored AIML, chatbos, how they talk etc

Have you looked at ChatScript? There’s a part of the forum here dedicated to it and it might be worth looking into for your project. Also, since you are familiar with Java, you might appreciate RiveScript. There are threads here dedicated to that as well, and the creators of both languages are contributers to this forum.

3) I have explored grammar rules a bit.

This is especially important if you want to build your own parser, but also useful for building more advanced AIML sets. Since you only have 7 months to work with, it might be in your interest to integrate a developed parser (see below), especially if you build your engine from scratch.

4) I am skilled at programming C/C++, C#, Java, SQL, SQLServer, Oracle, ASP.NET

I recommend taking a look at stanford’s LEX parser. It’s written in Java and released under GPL. If you’re interested in learning Python, the Natural Language Toolkit and extensions are highly recommended.

5) I can devote 5-6 hours daily for 7 months

Great! You should definitely be able to get something workable in time. (And oh boy am I jealous you have so much free time available!) Your best bet is to build off of an existing chat engine, such as ChatScript, RiveScript, or those for AIML. However, as it is a class project, it might be more interesting to build your own.

Definitely keep the forum updated on your progress.

PS: PS stands for postscript.

Making a chatbot and making a chatbot engine are two different things.  There are a number of good chatbot engines already available, such as Program O, ChatScript, and RiveScript.  I honestly do not see the point for you to create another chatbot engine from scratch.  AIML is a chatbot language, for instance various chatbot engines use the AIML language, such as Program O.  Creating a chatbot, or virtual personality, on a chatbot engine is a whole nother can of beans.  One does not enter a chatbot engine per se into the Loebner Prize; one enters a chatbot, or virtual personality built upon a chatbot engine, using a chatbot language.  And, it could easily take 7 months just to create a virtual personality, which might be barely competitive in the Loebner Prize.

A grammar parser is neither a chatbot engine nor a chatbot language.  In fact, I wish someone would provide an example of a natural language interpreter for a grammar parser.  My understanding of a grammar parser is that one would more likely convert a parsed grammar into a chatbot language for use in a chatbot engine, or chatbot language interpreter.

IMHO, what would be more productive would be to use your enthusiasm and programming skills to create a cloud-based commercial web service, dialog system API, with Program O, ChatScript, or RiveScript.  Of the existing cloud-based consumer dialog systems, including @bot_aco, @chatbot4u, @mycybertwin, and @pandorabots… Pandorabots is the only one that offers a fully customizable knowledgebase, downloadable and uploadable, as well as an API http://bit.ly/Ihg9eH .  What it does not offer is a backend API for modifying the knowledgebase automatically, without downloading and uploading for offline modification.  It also does not offer turnkey bindings to popular social networks, for the purpose of Social CRM.

@C R Hunt
Thats for sure that I am not going to use ChatScript, Program O, RiveScript because I would not be able to call it my own (I am not talking about Licensing ) So I used word scratch. The idea of using a developed parser is good.
So I am investigating link.
Chatbot is my Final Year Project so I have all the time for it.

@Marcus Endicott
You almost killed my passion to make it from scratch, is it really that difficult ;-|

And by chatEngine I mean I want to design the chatbot talking algorithm completely. Sorry for my less understanding of Jargon.

I already have a chatbot who doesnt talks i.e. all the work of hosting, database, chat logging, web, domain is done.
Making it using C# as the only language.

By grammar parser I mean something like this http://goo.gl/Bpp1F

By your talks what I can identify is that whether I am going to fail completely or reinvent the wheel if I go by make-from-scratch approach, in 7 months.

LEX Parser must be great but I am worried about the time it takes to do the computation and the memory it requires.
The site says at least 100MB is required and from this I guess I will be kicked by the web hosting service…

These days, 100MB of memory usage, even on a web server, is almost trivial, and we’re talking about “burst usage”, not something that would require that much memory load on a constant/consistent basis. Now, if we were talking about BANDWIDTH, that’s another matter altogether. But we’re not, so no worries.

I do have to say that I admire your ambition, and that I wish you success in your efforts. What you’re wanting to do is no easy feat, though I firmly believe that it’s well within the realm of possibility. If there’s anything we can do to assist, just let us know, and we’ll advise you to the best of our abilities.

@Muhammad

Hi, I’ll tell you a short-story (mine)
I once wanted to do the same thong, I am also a skilled programmer, knowing C, C++, Basic and USCD Pascal and many other odd-flavored computer languages like assemblers. This was the year 82 and the Apple2 I had in the Biomedical Research Institute at the Buenos Aires University had 64k memory and a 6502 Rockwell microprocessor running a few megahertz. I begun the odyssey.. no way! only a odd-gaming Eliza clone-came out, very funny but stupid an useless. I gave up.

Then I studied Electronics Engineering, Computer Science and Biomedical Sciences, Got to became a researchoer and entrepreneur in communications and many other stuff.

Once again.. on 2005 I sat down with the same ‘old’ idea, to make a good chatbot.

Then studied AIML 1.0, and searched for some other.. no luck! (RiveScript or ChatScript) were not available. So I decided to make my own engine, just luke you, but with no time-event horizon!

I begun going into matching engines, parsing, regular expressions, partial matching (Soft comparisons) and many more techniques, the whole were complicated and to build a system you had to make some sort of language-interpreter similar to JavaScript, but obeying your own chat-enabling rules. Gotcha! - I thought! (but I was terribly wrong)

An interpreter solution.. (my firt guess) was slow, very slow!, you must build a system that while it interprets your script, builds (in memory) the whole runtime-logic, but you cannot operate easily w/o variables, adn flexibility, which AIML dont have (unless you are a skilled programmer and change the guts of the AIML-engine (Program-xx…) , and you need much luck to do so and dont break the system)

Finally I got to build a compiler, this compiler gets the script, and generates high-level logic in language suchj as Java or C++, or C# (any of this) then you compile this down to a ML-library (or MSIL/CLR)  ane run it inside a runtime-engine.

All this process got to take almost 6 years, because meanwhileI had to build my own tokenizer, lemmatizer, scanner, chunker, grammar parser, named entity recognizer, build your own language-specific-grammar, make spell correction modules, etc.

So If you want to do something which is worth, in 7 months, concentrate into a small part, and do it VERY WELL, then add it to a working engine and see if it gets a better behavior..

To show you a number.. my whole project is worth a few years of work, grow to over >500k lines of code, >900 classes, >85 libraries and testing units, etc. etc. The documentation is >200 pages (raw, no details) and I work 12 hour a day to get it better every day!

And I don’t even dare to take it to beat a Loebner Prize because I wanted to make a smart-bot-engine (not a personality) and quite not a bot with capability to trick a judge, but to show a rather human, humble but intelligent behavior!

Hope this words help you out!

@Dave
So its a FEAT… and withing “realm of possibility” These two words huh! However, I will need your help in the way…

@Andres
So you are saying its immmpossible…

I admire your dedication, you are nearly pushing limits.

Ok, Thanks to all for the time you people put in the responses and special thanks to Dave, you discouraged the least

Well, now I have selected the project, currently writing proposal. now stopped… Give me time. Will let you all know if I dive into..

Thanks a lot for opening my eyes

I’m more than happy to assist wherever I can. Just bear in mind that I have, in all probability, one of the least experienced, lowest educated resumes here, which is probably why I was the least discouraging. It may well be that I don’t know enough to know when to quit. But that’s ok; some of the world’s best discoveries came from someone who was “too stupid” to quit (or, at least, refused to listen to the “experts” of the day, which amounts to the same ting)!

Thanks Dave! You have a chatbot thats enough to show pose you as an expert.

I have taken the first step, have started writing proposal and let see how far I can go.

That’s for sure I need to make it, by hook or by crook

Given the limited time frame, I strongly suggest that you consider starting small and building from there. It’s also a very good idea to look for inspiration from other chatbot applications that already exist. that’s not to say that you should copy someone else’s work, but you can certainly look at what works for other folks, and see if something similar would work for you.

Also, I can tell you from personal experience that working on a project of this type too hard, and for too long causes a person to get “burned out” VERY quickly, and is something to be mindful of. “All work and no play”, and all that.

Affirmitive

Well! actually it might not be appropriate to suggest sumthing cause you already called it off by “affirmative”. But as I see its not that much of time you had last post here so I might add sumthing here. With all the other thing considered you say about yourself…

Should I use AIML type approach, or Can I go with writing a grammar parser?
Another limitation is the hosting server might not be that powerful as I have to bring it online.

Well, all those experienced senior ppls here has provided their view which have to have some real meat about them. But let me add one more perspective - an novice outsider view. If you say you gone through AIML things even if just a overview than you must be knowing its plus points and limitations. You don’t need that much of time or dig deep to know such things. As far as grammar parser it all depends on what you call grammar.

In my view present state of AIML is too primitive for something as you call “worth talking”.  And its more cause how you use it what you use rather than being it in itself being not worthy.  It seems to me that you haven’t gave much time to your own thoughts yet. What I wanna say is since you just college student and have so much time why not use it to ponder about the potential problems you might be facing. You first have to identify the problem you need to provide solution for. And at least this not a field that you just don’t know about. Even a layman know something about it cause he himself is a engine in a sense and been dealing with those issues since his birth. You are a apt programmer so seeing through cracks or logic or its practical implementation wud not be that much a problem for you as well.

But as C R Hunt or other senior members here say its always good to look at some of the existing system a little bit at start. At least it’ll kick you out of primary inertia phase you might be feeling as novice here. To make thing going!

As far as grammar parser is concern. I won’t call it a grammar parser but some sort of parser you will eventually need to implement at some point of time. Even if you go AIML way you will have to make some changes at core level to make it suitable for your so call “worthy bot”. So just pick up at least one piece of comfortably good enough open source project from both sides, give yourself a bit of time to browse their basic logic then put your mind to basic problem you have to deal with. And then if you think whatever way, you go for it. But tell you it’ll be eventually a hybrid way . Actually more than hybrid way, if to be exact. Cause its not that something is very metaphysical or too typical about it that we mere souls cann’t handle or design such thing. Even so called GODS will go bonker if they have to design something within those constraints, no pun intended ...smile. What really makes it hard that you expect a man to run who don’t have feet. It’s difficult to design such perfect logic in the line of human being cause as a human being we have lot of senses or input method which eventually get encoded into text. whether directly or indirectly. You don’t expect me to tell you the meaning of word “bientôt” if I don’t know french or have any idea bout it, right? So eventually you objective should be to identify such basic differences and provide some means to compensate for. for example, what if our so call hypothetical man might become a gymnast and start waking upside down on his hands ...haha. Or its just really all AI (artificial Intelligence) no meat. got my point? ...smile.

And boy its really a best of your time for you to take a such a long shot at core logic. You won’t find that much opportunity or freedom once you get engage somewhere professionally or proceed through student life to more mature one.  It’s somewaht paradoxical but still ...smile

And everybody here already promised you for possible support. As myself being a nobody kinda guy so I might add a few hours or so to your daily time if need comes by ...

Wish you best of luck! Its really nice to waste our time on this earth at least in some sort of respectable way. And last of all, me no native English so pls do ignore my literal mistakes.

Cya! some other time some other place maybe

Sorry to interrupt again, but one last and formost important thing. I don’t know what you’ll make out of it but do make honesty one of the main attribute of your bot. Their must be something typical or magical about that thing that we make so much hue and cry over that concept. It’ll put lot of problem out of valid context if not anything else. Its easier to digest somebody’s honest limitation than fake alibis, no? Same here in life as well ...smile.

And I give you a head start pointers. Just grab all the different e-text you can lay your hand on whether as simple as stories, novels or film scripts. Just merge them into a overlapping multi-link database.  And you have the best collection of valid templates there ever could be. With some of the problems we talk here already solved i.e spell or grammar check things like that. And try make it linking at alphabet level if possible or even audio sample level if you can go that far. But I don’t think it wud be better choice for you at this point of time, just keep eye on that side of concept so when time comes you can adjust your logic for its incorporation. And designing something on this line might look like something shooting long or overdrive but believe me its really gonna make overall search logic far more efficient and effective in the end. It’ll solve most of the NLP/NLU problems we talk about automatically so that we can concentrate on more appropriate things or constraints. In this way let your software auto generate most of the valid phonemes/prefix/suffix or word phrases their ever could be. Might be all who cares Then try give them the meaning rather than keeping separate dictionary. This neural network sort of approach will provide you further benefit of being able to analyze things as they come by in real time. Alphabet by alphabet, word by word or possible phrase wise or possible meanings. With extra benefit of best place to place some of the basic auto learning logic. All you have to do is to design your logic in such a way that you take care of exceptional conditions as well within the same logic. i.e its good to try middle bulb check approach when checking for faulty bulb within a Christmas lights but when things comes to 5 or 6 bulbs linear approach not that bad a choice I not computer profession so dunno know your vocabulary. You guys got some “sort” name for it, no?

And try not to loose the original format or sequence of text. Design your database that you can get the original text in proper sequence any time. If we loose somewhere doesn’t mean we shouldn’t gain somewhere. See between the lines. And obviously you have to forget size constraints here for some time. Or have to adjust for exceptions sometime during the way. But eventually modify it into a knowledgeable resource in steps or in incremental scanning /rescaning, one thing at a time in our friend victor shulist way of saying. i.e you can incorporate AIML categories into it. They are dialogue based if not anything else. But tell you the problem there is they are writern with some weirdo approach in mind so not that use in its totality but you sure can learn by gleaning over the AIML files itself or its documentation, if not the software code. Other etext might be better. Add various dictionaries into it ..whatever you can lay your hand on. Some sense impairment especially visual one it too much a liability or problem in design of such system but you know some wise person says “being unreasonable is also one of the valid reason in itself”.

Write logic for logic not logic like our respected friend gary dubuque “that is, rules about rules”.

Hard code some most common basic logic into engine itself this that is where when how…haha . Give it a power to ask about problemos. something like that.

And finally kick the dirt off high enough! ...and let us all smell the euphoria of it….

Hope its far more than enough for time being ......You see I am a totally waste person by some standards here in this world so most probably I got far more free time than even you…haha But I do understand the logic of limit or implied boundations. So with out any further boring lecturing I QUIT! chio!

one last quote kung fu style

“Kick where the right spot is and even the mightiest man of size Tyson will go down on his knees without much force or else its just as good as fingering the bull from behind without thinking eventual consequences.” ...smile

is it kinda fuzzy logic here?

And as final dedication just for you only.
Not everything make sense in first place….

http://www.youtube.com/watch?v=c0z73WDPb20