Chatbot listing, virtual agents, virtual assistants, chat bot directory, conversational agents, virtual human news, chatterbot list
NEWS: Chatbots.org survey on 3000 US and UK consumers shows it is time for chatbot integration in customer service!read more..
Evaluation of Conversational Agent Systems
The evaluation of conversational agents or chatterbots question answering systems is a major research area that needs much attention. Before the rise of domain-oriented conversational agents based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when chatterbots began to become more domain specific, evaluation becomes a real issue. This is especially true when understanding and reasoning is required to cater for a wider variety of questions and at the same time to achieve high quality responses. This paper discusses the inappropriateness of the existing measures for response quality evaluation and the call for new standard measures and related considerations are brought forward. As a short-term solution for evaluating response quality of conversational agents, and to demonstrate the challenges in evaluating systems of different nature, this research proposes a blackbox approach using observation, classification scheme and a scoring mechanism to assess and rank three example systems, AnswerBus, START and AINI.
| Download Paper | |
| Full Title | A Black-box Approach for Response Quality Evaluation of Conversational Agent Systems |
| Year | 2006 |
| Language | English |
| Journal |
International Journal of Computational Intelligence
2006, number 3, volume 3 |
| Keywords | conversational agents, chatterbots, question answering system, evaluation, response quality, Response Quality Evaluation, Black-box Approach, natural language understanding |
| Synonym | Conversational Agent |
New Comment
Only registered members are allowed to comment.
or login
Browse Topics
