Analysis of a State Machine-based Interactive Dialogue Management System Público

Abstract

The sports topic handler has been one of the key components in `Emora', an open-domain chatbot system that competes for the Alexa Prize 2020, which is a university challenge for creating the best socialbots. This thesis first gives a comprehensive description of Emora’s sports topic handler, including its architecture and innovations. These innovations involve effective approaches to opinion-based state transition dialogue management that uses a daily updated database as well as derivation of engaging conversations on flashing events in sports. Given this topic handler, Emora is capable of making multi-turn dialogues on any game, player, and team upon request by inferring statistical facts from the database and sharing its own opinions about the latest topics. These unique features help the sports topic handler to become the highest rated components in Emora in February and one of the highest rated components of all time. This thesis also presents the result of a user study on the sports topic handler which evaluates the impact of various modular improvements on the overall ratings provided by random users interacting with the chatbot. The impact of each update is evaluated extrinsically through the overall ratings. Our analysis finds a strong positive correlation between these updates and the user ratings, while also finds a negative correlation associated with uncovered topics and ignorance to the user input.

   

  This thesis makes the following contributions. It demonstrates how associating a database with a state machine allows chatbots to easily and quickly generate multi-turn engaging dialogues based on real-time information ; It provides a possible way of incorporating question answering into conversational agents; It gives a user study which could serve as a reference to future researchers who want to understand the impacts of modular improvements on a chat-bot, specifically for the sports domain .

Table of Contents

1 Introduction 1

2 Related Work 3

2.1 Conversation flow controlling . . . . . . . . . . . . . . . . . . . . . . 3

2.2 Real-time information based dialogues . . . . . . . . . . . . . . . . . 4

2.3 Open domain question answering . . . . . . . . . . . . . . . . . . . . 5

3 Background 7

3.1 Alexa’s prize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.2 Dialogue system architecture . . . . . . . . . . . . . . . . . . . . . . . 8

3.2.1 Dialogue manager . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2.2 Topic handlers . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4 Sports Topic Handler 11

4.1 Database associated state machine . . . . . . . . . . . . . . . . . . . 11

4.1.1 State Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.1.2 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4.1.3 Open domain question answering . . . . . . . . . . . . . . . . 17

5 Evaluation 24

5.1 Logged information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.2 Unweighted user-ratings . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.3 Weighted user-ratings . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

6 Results and Analysis 26

6.1 Events summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

6.2 Weighted ratings vs. unweighted ratings . . . . . . . . . . . . . . . . 29

6.3 Error (Low-rating) Analysis . . . . . . . . . . . . . . . . . . . . . . . 30

6.3.1 Overall statistics . . . . . . . . . . . . . . . . . . . . . . . . . 30

6.3.2 Qualitative error analysis . . . . . . . . . . . . . . . . . . . . . 30

7 Conclusion 33

Bibliography 34

About this Honors Thesis

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Emory College
Department	Computer Science
Degree	B.S.
Submission	Honors Thesis
Language	English
Research Field	Computer Science
Palavra-chave	Chatbot Dialogue Management Spoken Dialogue System Conversational Agent Open Domain Conversation State Machine
Committee Chair / Thesis Advisor	Choi, Jinho D., Emory University
Committee Members	Fossati, Davide, Emory University Nagy, James, Emory University

Última modificação

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Analysis of a State Machine-based Interactive Dialogue Management System ()	2020-04-28 00:41:00 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions