Measuring Efficacy of Speaking English Chatbot NUMLINA: A User Study

Abstract

The utilisation of Natural Language Processing has enabled us to build conversational chatbots. In particular, pedagogical settings necessitate the utilisation of a speaking chatbot because writing chatbots are already functional in the industry. There was a need for such a conversational chatbot which could fulfil pedagogical needs. It was hypothesised that a chatbot could teach English speaking skills better than a conventional teacher. First, the research team built the NUMLINA chatbot with DialogFlow, a Google built-in automated infrastructure supported by Artificial Intelligence and machine learning. This study aimed to teach speaking English skills effectively to students with NUMLINA chatbot. This user study followed an experimental research paradigm to measure the effectiveness of the newly built NUMLINA chatbot while speaking English with a human being. Comparing the post-test of the controlled and experimental group's means, 1.4 and 1.9 in fluency, 1.1 and 1.8 in vocabulary, 1.3 and 1.8 in pronunciation, 1 and 1.4 in learning idioms, and 1.3 and 1.7 in communication. The experimental group outperformed in the five speaking English categories validating the hypothesis. Thus, it promoted autonomous learning by advocating the modern teaching method for learning English.