Intent Prediction and User Preference Modeling in Conversational Search and Recommendation Öffentlichkeit

Abstract

Conversational systems have emerged as potent tools for increasing the accessibility of user-facing applications, in particular, search and recommendation. This dissertation addresses two key challenges in conversational systems: intent prediction and user preference modeling.

  

Identifying the intent of each user utterance in conversational systems is a crucial step for all subsequent language understanding and response tasks. The high cost of collecting conversational training data makes the task of intent prediction challenging. The first main research question addressed in this dissertation is: Can we use external knowledge and synthetic data to improve intent prediction? I propose methods for knowledge-aware intent prediction in three settings, including open-domain social bots, conversational information elicitation systems, and web-based domain-specific search systems. In addition, I study the impact of synthetic data on intent prediction in these systems.

User preference modeling is another essential part of effective conversational systems. The second main research question addressed in this dissertation is: Can we anticipate the user's next topic of interest by constructing a user profile using conversation context? I propose methods to represent the user based on the conversation history. Moreover, a sequence modeling approach is proposed to predict the user's next topic of interest in conversational systems.

Despite the capacity of large language models to implicitly perform end-to-end intent detection and user preference modeling, they are not universally applicable. They cannot be used with private data. Moreover, modular systems with specialized components allow for more interpretability and control over the system. Systems with modules for intent detection and user preference modeling are thus still relevant.

Together, the proposed methods enable a better understanding of the user's immediate needs and long-term preferences in all types of conversational systems. The findings of this research hold implications for improving the accuracy and performance of conversational search and recommender systems.

Table of Contents

1 Introduction 4

2 Related Work 7

2.1 Intent Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1.1 Intent Prediction in Open-Domain Conversational Systems . . . . . . . . . 7

2.1.2 Intent Prediction in Information Elicitation Conversational Systems . . . . 8

2.1.3 Intent Prediction in Web Search . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2 Conversational Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2.1 Entity linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 Entity-based Text Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.4 Conversational Systems for Interviewing Users . . . . . . . . . . . . . . . . . . . . 11

2.5 Knowledge Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.6 Dialog Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.7 Clarification Question Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.8 Conversational Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3 Intent Detection in Conversational Search and Recommendation 16

3.1 ConCET: Entity-Aware Intent Classification for Open-Domain Conversational Agents 16

3.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.1.2 ConCET System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1.3 Conversational Entity Linking . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1.4 ConCET: Concurrent Entity-Aware Intent Classifier . . . . . . . . . . . . . 22

3.1.5 Conversational Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . 25

3.1.6 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

3.1.7 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.1.8 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.1.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.2 Contextual Intent Classification for Automated Structured Interviews . . . . . . . 34

3.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2.2 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.2.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.2.4 Experimental Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3.2.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

4 Query Intent Classification for Domain-Specific Web Search 45

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.1 Knowledge Injection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2.2 Data Augmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.3 Experimental Setting: Data and Methods Compared . . . . . . . . . . . . . . . . 50

4.3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

4.3.2 Knowledge Base and Representation . . . . . . . . . . . . . . . . . . . . . 52

4.3.3 Validation Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

4.3.4 Methods Compared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.3.5 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.4.1 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.4.2 Data Augmentation Results . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.4.3 In-Context Learning Results . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.5.1 Ground Truth Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.5.2 Entity Linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.6 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.8 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5 User Modeling 59

5.1 User Modeling for Topic Recommendation in Open-Domain Conversational System 59

5.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.1.2 Conversational Topic Suggestion (CTS): Problem Definition . . . . . . . . 60

5.1.3 CTS-Seq Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

5.1.4 CTS-Seq: Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.1.5 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

5.1.6 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

6 Discussion and Conclusions 76

6.1 RQ1: Intent Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.1.1 RQ1a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.1.2 RQ1b . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

6.1.3 RQ1c . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

6.1.4 Relevance in the LLM Era . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

6.2 RQ2: User Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.2.1 RQ2a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

6.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

A Prompts for Experiments with LLMs 81

A.1 Prompts for Synthetic Data Generation . . . . . . . . . . . . . . . . . . . . . . . . 81

A.1.1 Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

A.1.2 Entity-based . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

A.2 Prompts for In-Context Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

B Additional Results 84

Bibliography 85

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Computer Science and Informatics
Degree	Ph.D.
Submission	Dissertation
Language	English
Research Field	Artificial Intelligence Computer Science
Stichwort	information retrieval conversational systems intent prediction user representation
Committee Chair / Thesis Advisor	Vanessa Murdock, Amazon Eugene Agichtein, Emory University Jinho Choi, Emory University Joyce C. Ho, Emory University

Zuletzt geändert

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Intent Prediction and User Preference Modeling in Conversational Search and Recommendation ()	2024-08-09 15:17:16 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions