Predicting the success of Conversational Task Assistants (CTA) can be critical to understand user behavior and act accordingly. In this paper, we propose TB-Rater, a Transformer model which combines conversational-flow features with user behavior features for predicting user ratings in a CTA scenario. In particular, we use real human-agent conversations and ratings collected in the Alexa TaskBot challenge, a novel multimodal and multi-turn conversational context. Our results show the advantages of modeling both the conversational-flow and behavioral aspects of the conversation in a single model for offline rating prediction. Additionally, an analysis of the CTA-specific behavioral features brings insights into this setting and can be used to bootstrap future systems.
|IR: Research and Development in Information Retrieval
|Association for Computing Machinery
|46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023
|Taiwan, Province of China
|23/07/23 → 27/07/23
- Conversational Task Assistants
- Rating Prediction