Web-Based Evaluation of Talking Heads: How Valid Is It?

Web-based tests are an established method for the evaluation (e.g. [1]). However, the validity of web-experiments has not yet been evaluated for the field of embodied conversation agents (ECAs). In this paper, evaluation results obtained in a non-interactive scenario in our lab [2] are compared to a similar web-based experiment. 3 different
talking head components and 2 freely available speech synthesis systems (TTS) are combined. The head components used are the thinking head (TH), Massy (MS), and a speaker cloning system (CL). The TTS include Mary ('hmm-bits3') and Mbrola ('de2'). We recorded videos of the talking heads speaking sentences related to the smart home domain like: The following devices can be turned on or off: the TV, the lamps and the fan.'