Of the 514 chats that were rated and conducted exclusively with the chatbot, 141 chats were rated “good” and 373 were rated “bad”, resulting in a chat satisfaction score of 27.4% from January to June 2023. The factors contributing to low chat satisfaction derive from a complex range of perspectives, which need to be addressed to provide improvements and increase customer satisfaction.

This study examines conversations between users and the chatbot. These conversations undergo analysis through reading and employing thematic analysis. The goal is to identify areas where issues arise in understanding users' needs within different intents. Furthermore, the study observes instances where conversations become more intricate and require human intervention. Metrics such as chat duration, language, and presumed intent were also used for the analysis.

By focusing on key areas of concern, such as problematic intents, the study uncovers critical pain points and challenges that users often face when engaging with the chatbot.