With the advancement of computer technology and the popularity of the Internet, there are gradually more online learning resources that can serve as English-learning tools for Taiwanese students. Among those resources, chatbots can process typed messages and respond by showing texts on the screen. Chatbot technology has been incorporated by some English-speaking countries, whose online ticket reservation system and stores adopt conversational agents to serve as information staff. Customers can pose questions online and chatbots can immediately offer appropriate responses using keyword matching, database searching, or pattern transformation. Literature showed that there are now five major types of chatbot construction mechanisms, though only three of which offer complete interaction transcripts. Previous studies on chatbots mainly focused on user perception. Few studies were conducted on chatbots’ effects on language learning. The main purpose of this study is to explore the effects of open-ended chats with three chatbots on English learners’ syntactic complexity, grammar accuracy, message appropriateness, and fluency. This study also examines learners’ interactions with different chatbots and investigates if learners’ perception of chatbots for English learning changes after chatting with different chatbots.
The participants were a cohort of 42 10th graders from a community-based senior high school in Nantou. This study lasted for eight weeks. In Week 1, all students spent forty-five minutes chatting with three chatbots. The transcripts were collected and analyzed for learners’ syntactic complexity, grammar accuracy, message appropriateness, and fluency as initial proficiency. Between Week 2 to Week 7, participants chatted in an open-ended manner with all three chatbots for two thirty-minute sessions on their own and filled out a perception questionnaire for each bot they used. In Week 8, all participants chatted again with all three chatbots for fifteen minutes each. The transcripts were once again collected and analyzed for syntactic complexity, grammar accuracy, message appropriateness, and fluency to compare if pre- and post-tests were significantly different. During the six weeks from Week 2 to Week 7, participants were required to turn in their transcripts and questionnaires to the researcher to prove their participantion.
However, during data collection, the researcher found that only 20 students finished the two thirty-minute sessions with each chatbot; hence, participants were put into two groups based on their degree of participation. Those who spent three hours chatting with bots were labeled as complete experiment group (CEG), while those who finished less than one hour were categorized as partial participation group (PPG). The rest of 10 students failed to finish either pre- or post-tests and were excluded.
To examine the effects of bots, paired sample t-tests were first used to explore if within group differences in the pre- and post-tests achieved significance. After ensuring comparability between CEG and PPG, cross group analyses were computed to investigate the effects of interacting with chatbots on English learning. Moreover, repeated measure ANOVAs were adopted to compare if the same students performed differently when interacting with different chatbots. As for perception, descriptive statistics and one-way ANOVAs were computed to check if students had different perception of different chatbots.
The five major findings are summarized below.
1. After interacting with bots for three hours, students’ syntactic complexity did not change.
2. Regularly chatting with bots raised English learners’ grammar accuracy, but did not significantly influence message appropriateness. In the interactions with bots, students might sporadically pick up some new grammar concepts or simply mimic chatbots’ language and developed their grammar level as a result.
3. Interacting with bots for three hours significantly increased students’ fluency. Students indicated in the questionnaire that using chatbots offered them more time to use English for communication; thus, fluency might have improved due to increased time to practice English.
4. Chatbots made using different construction mechanisms had some influence on syntactic complexity, but they did not affect fluency.
5. Students’ perception indicated that Skynet-AI responded at a faster rate than Cleverbot. Learners also indicated that Meg’s message appropriateness was much weaker than Skynet-AI and Cleverbot, suggesting that different mechanisms might also influence bots’ processing speed and chatbots’ message appropriateness.