Knowledge-Aware User Intent Inference for Web Search and Conversational Agents 公开

Ahmadvand, Ali (Fall 2021)

Permanent URL: https://etd.library.emory.edu/concern/etds/tq57ns28g?locale=zh
Published

Abstract

User intent inference is a critical step in designing intelligent information systems (e.g., conversational agents and e-commerce search engines). Accurate user intent inference improves user experience and satisfaction, but is a challenging task since user utterances or queries can be short, ambiguous, and contextually dependent. Moreover, in an e-commerce setting, the collected datasets are often labeled by weak supervision (e.g., click-through data), resulting in an imbalanced and sparse dataset. To address these problems, my dissertation proposes integrating entity knowledge-bases, conversation context, and user profile information to improve user intent inference for conversational agents. Additionally, I investigate joint learning, product taxonomies, and unlabeled domain-specific corpora (e.g., catalog) to improve query intent inference in e-commerce search.

To evaluate the proposed models, I examine the user intent inference for two main settings: 1) open-domain conversational agents and 2) e-commerce search engines. The conversational agent research is evaluated on conversations collected from real users as part of Amazon Alexa Prize competitions, and the e-commerce efforts use real query logs collected from The Home Depot's search engine. My dissertation shows that leveraging entity knowledge-base, conversation context, and user profile information accounts for most improvements for the conversational setting. The results demonstrate that the proposed models significantly enhance topic classification accuracy by 15% and dialogue act accuracy by 8% for conversational agents. For e-commerce search, the dissertation shows that joint-learning, product taxonomies, and unlabeled domain-specific corpora can significantly improve intent inference accuracy. The proposed models improve the performance of the top-1 retrieved documents by 6%-8% on standard metrics for e-commerce search. The results in both settings offer a significant improvement over state-of-the-art deep learning methods. The insights and findings in this dissertation suggest a promising direction for developing the user intent inference in both open-domain conversational agents and e-commerce search.

Table of Contents

1 Introduction and Motivation 1

1.1 User Intent Inference in Open-Domain

Conversational Agents . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Challenges for User Intent Inference in

Conversational Agents . . . . . . . . . . . . . . . . . . . . . . 4

1.2 User Intent Inference in Web Search Engines . . . . . . . . . . . . . . 6

1.2.1 Challenges for User Intent Inference in

E-commerce Search . . . . . . . . . . . . . . . . . . . . . . . . 8

1.3 Dissertation Research Questions . . . . . . . . . . . . . . . . . . . . . 10

1.4 Contributions and Dissertation Structure . . . . . . . . . . . . . . . . 10

1.4.1 Contributions to Open-Domain

Conversational Agents . . . . . . . . . . . . . . . . . . . . . . 11

1.4.2 Contributions to E-commerce Search . . . . . . . . . . . . . . 12

1.5 Summary and Dissertation Structure . . . . . . . . . . . . . . . . . . 13

2 RelatedWork 14

2.1 User Intent Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.2 User Intent Inference in Open-Domain

Conversational Agents . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2.1 Topic Classification: NLU . . . . . . . . . . . . . . . . . . . . 17

2.2.2 Dialogue Act Classification: NLU . . . . . . . . . . . . . . . . 19

2.2.3 Smart Topic Suggestion for Open-Domain Conversational Agents 21

2.3 User Intent Inference for Web Search . . . . . . . . . . . . . . . . . . 24

2.3.1 Joint Learning for User Intent Inference . . . . . . . . . . . . 26

2.3.2 Label Representation for User Intent Inference . . . . . . . . . 29

2.3.3 Pseudo-relevance feedback for User Intent Inference . . . . . . 31

2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3 Integrating knowledge in User Intent Inference 33

3.1 Integrating Knowledge for Conversational Agents . . . . . . . . . . . 33

3.2 Integrating Knowledge for E-commerce Search . . . . . . . . . . . . . 38

3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4 User Intent inference in Conversational Agent 42

4.1 Emory IrisBot: An Open-Domain Conversational Bot for Personalized

Information Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

4.1.1 Language Understanding and Entity Recognition pipeline . . . 44

4.1.2 Domain-specific Components: . . . . . . . . . . . . . . . . . . 45

4.2 Topic and Intent Classification . . . . . . . . . . . . . . . . . . . . . . 46

4.2.1 Contextual Topic Classification . . . . . . . . . . . . . . . . . 46

4.2.2 Language Understanding and Entity Recognition Pipeline . . 48

4.2.3 Domain-specific Components . . . . . . . . . . . . . . . . . . 49

4.2.4 Topic and Intent Classification . . . . . . . . . . . . . . . . . . 49

4.2.5 Intent Classification . . . . . . . . . . . . . . . . . . . . . . . . 51

4.2.6 Dialogue Manager and Response Ranking . . . . . . . . . . . 53

4.2.7 Dialogue Manager: Implementation . . . . . . . . . . . . . . . 54

4.2.8 Personalized Topic Suggestion . . . . . . . . . . . . . . . . . . 55

4.2.9 Personalized Sequential Topic Suggestion Model . . . . . . . . 56

4.2.10 Cross-Component topic suggestion . . . . . . . . . . . . . . . 58

4.2.11 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 58

4.2.12 Topic and Intent Classifier: Internal Evaluation . . . . . . . . 59

4.2.13 Topic Suggestion Results . . . . . . . . . . . . . . . . . . . . . 60

4.2.14 E↵ects of Personalization on conversation behavior and ratings 61

4.3 ConCET: Entity-Aware Topic Classification for Open-Domain Conversational

Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.3.1 ConCET System Overview . . . . . . . . . . . . . . . . . . . . 63

4.3.2 Conversational Entity Linking . . . . . . . . . . . . . . . . . . 65

4.3.3 DBpedia Spotlight . . . . . . . . . . . . . . . . . . . . . . . . 66

4.3.4 PMI-based Entity Linker (PMI-EL) . . . . . . . . . . . . . . . 66

4.3.5 ConCET: Concurrent Entity-Aware Topic Classifier . . . . . . 70

4.3.6 Textual Representation . . . . . . . . . . . . . . . . . . . . . . 70

4.3.7 Entity Representation . . . . . . . . . . . . . . . . . . . . . . 73

4.3.8 Merging and FeedForward Layer . . . . . . . . . . . . . . . . . 75

4.3.9 Conversational Dataset Overview . . . . . . . . . . . . . . . . 76

4.3.10 Amazon Alexa Prize 2018 . . . . . . . . . . . . . . . . . . . . 76

4.3.11 Obtaining True Labels for Alexa Data . . . . . . . . . . . . . 77

4.3.12 Self-Dialogue Dataset . . . . . . . . . . . . . . . . . . . . . . . 77

4.3.13 Synthetic Training Data Generation . . . . . . . . . . . . . . . 79

4.3.14 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 80

4.3.15 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.3.16 Training Parameters . . . . . . . . . . . . . . . . . . . . . . . 82

4.3.17 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . 83

4.3.18 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 83

4.3.19 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.3.20 Detailed Performance Analysis . . . . . . . . . . . . . . . . . . 84

4.4 Contextual Dialogue Act Classification for Open-Domain Conversational

Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

4.4.1 Contextual Dialogue Act Classifier (CDAC) Model . . . . . . 89

4.4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 93

4.4.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 95

4.5 Knowledge-Aware Contextual Topic Suggestion for Open-Domain Conversational

Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

4.5.1 Conversational Topic Suggestion (CTS): Problem Definition . 103

4.5.2 CTS-Seq Approach . . . . . . . . . . . . . . . . . . . . . . . . 104

4.5.3 System State and User Profile Features . . . . . . . . . . . . 105

4.5.4 CTS-Seq: Models . . . . . . . . . . . . . . . . . . . . . . . . . 106

4.5.5 CRF Implementation of CTS-Seq: CTS-CRF . . . . . . . . . 106

4.5.6 Deep-learning based implementation of CTS-Seq: CTS-CNN

and CTS-RNN . . . . . . . . . . . . . . . . . . . . . . . . . . 107

4.5.7 Hybrid Sequential and Collaborative Filtering: CTS-Seq-CF . 111

4.5.8 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 113

4.5.9 Baseline 1: Popularity Method . . . . . . . . . . . . . . . . . . 113

4.5.10 Baseline 2: Collaborative Filtering (CF) . . . . . . . . . . . . 114

4.5.11 Baseline 3: Contextual Collaborative Filtering: Contextual-CF 115

4.5.12 Methods Compared . . . . . . . . . . . . . . . . . . . . . . . 116

4.5.13 Dataset: Amazon Alexa Prize 2018 . . . . . . . . . . . . . . . 117

4.5.14 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 117

4.5.15 Ground Truth Labels . . . . . . . . . . . . . . . . . . . . . . . 119

4.5.16 Training CTS-CRF Model . . . . . . . . . . . . . . . . . . . . 119

4.5.17 Results And Discussion . . . . . . . . . . . . . . . . . . . . . . 120

4.5.18 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

4.5.19 Feature Ablation on CTS . . . . . . . . . . . . . . . . . . . . 123

4.5.20 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

5 User Intent inference in E-commerce Search 127

5.1 JointMap: Joint Query Intent Understanding For Modeling Intent Hierarchies

in E-commerce Search . . . . . . . . . . . . . . . . . . . . . 128

5.1.1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 129

5.1.2 Joint-Learning of High-Level Intent Tasks . . . . . . . . . . . 131

5.1.3 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . 133

5.1.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 134

5.1.5 Main Results and Ablation Analysis . . . . . . . . . . . . . . 136

5.2 Label Representation for Product Category

Mapping in E-commerce Search . . . . . . . . . . . . . . . . . . . . . 138

5.3 DeepCAT: Model and Implementation . . . . . . . . . . . . . . . . . 138

5.3.1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . 139

5.3.2 Query Representation (Query2Vector Network) . . . . . . . . 140

5.3.3 Word-Category Representation . . . . . . . . . . . . . . . . . 140

5.3.4 Category-Category representation . . . . . . . . . . . . . . . . 141

5.3.5 Computation of Loss Function . . . . . . . . . . . . . . . . . . 141

5.4 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 142

5.4.1 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . 142

5.4.2 DeepCAT Experimental Design . . . . . . . . . . . . . . . . . 143

5.4.3 Parameter Setting . . . . . . . . . . . . . . . . . . . . . . . . 143

5.4.4 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 144

5.4.5 Methods Compared . . . . . . . . . . . . . . . . . . . . . . . . 144

5.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 144

5.5.1 Results on Minority Classes . . . . . . . . . . . . . . . . . . . 145

5.5.2 Results on Traffic Buckets . . . . . . . . . . . . . . . . . . . . 145

5.6 Ablation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

5.7 APRF-Net: Attentive Pseudo-Relevance Feedback Network for Query

Categorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146

5.7.1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 147

5.7.2 Initial Retrieval Step . . . . . . . . . . . . . . . . . . . . . . . 148

5.7.3 Representation Layers . . . . . . . . . . . . . . . . . . . . . . 148

5.7.4 Corpus-Aware Attention Network . . . . . . . . . . . . . . . . 150

5.7.5 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . 152

5.7.6 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . 153

5.7.7 Empirical Results and Discussion . . . . . . . . . . . . . . . . 155

5.7.8 Ablation Analysis . . . . . . . . . . . . . . . . . . . . . . . . 156

5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

6 Discussion 159

6.1 Strengths of Knowledge-aware Based Models: Addressing RQ1, RQ2,

and RQ3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

6.1.1 Addressing RQ1 . . . . . . . . . . . . . . . . . . . . . . . . . . 160

6.1.2 Addressing RQ2 . . . . . . . . . . . . . . . . . . . . . . . . . 161

6.1.3 Handling Data Imbalance: Addressing RQ3 . . . . . . . . . . 163

6.1.4 Handling tail and torso Queries: Addressing RQ3 . . . . . . . 164

6.1.5 Handling Overfitting: Addressing RQ3 . . . . . . . . . . . . . 166

6.2 Limitations of Using External

Sources of Information . . . . . . . . . . . . . . . . . . . . . . . . . . 167

6.2.1 Higher Latency . . . . . . . . . . . . . . . . . . . . . . . . . . 167

6.2.2 Aligning Knowledge Source with the Target Dataset . . . . . . 168

6.2.3 O✏ine vs. Online Evaluation . . . . . . . . . . . . . . . . . . 168

6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

7 Conclusions 170

7.1 Summary of the Results: . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.2 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . 173

7.3 Summary of the Dissertation . . . . . . . . . . . . . . . . . . . . . . . 177

Bibliography 180

About this Dissertation

Rights statement
  • Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.
School
Department
Degree
Submission
Language
  • English
Research Field
关键词
Committee Chair / Thesis Advisor
最新修改

Primary PDF

Supplemental Files