Knowledge-Aware User Intent Inference for Web Search and Conversational Agents Public
Ahmadvand, Ali (Fall 2021)
Published
Abstract
User intent inference is a critical step in designing intelligent information systems (e.g., conversational agents and e-commerce search engines). Accurate user intent inference improves user experience and satisfaction, but is a challenging task since user utterances or queries can be short, ambiguous, and contextually dependent. Moreover, in an e-commerce setting, the collected datasets are often labeled by weak supervision (e.g., click-through data), resulting in an imbalanced and sparse dataset. To address these problems, my dissertation proposes integrating entity knowledge-bases, conversation context, and user profile information to improve user intent inference for conversational agents. Additionally, I investigate joint learning, product taxonomies, and unlabeled domain-specific corpora (e.g., catalog) to improve query intent inference in e-commerce search.
To evaluate the proposed models, I examine the user intent inference for two main settings: 1) open-domain conversational agents and 2) e-commerce search engines. The conversational agent research is evaluated on conversations collected from real users as part of Amazon Alexa Prize competitions, and the e-commerce efforts use real query logs collected from The Home Depot's search engine. My dissertation shows that leveraging entity knowledge-base, conversation context, and user profile information accounts for most improvements for the conversational setting. The results demonstrate that the proposed models significantly enhance topic classification accuracy by 15% and dialogue act accuracy by 8% for conversational agents. For e-commerce search, the dissertation shows that joint-learning, product taxonomies, and unlabeled domain-specific corpora can significantly improve intent inference accuracy. The proposed models improve the performance of the top-1 retrieved documents by 6%-8% on standard metrics for e-commerce search. The results in both settings offer a significant improvement over state-of-the-art deep learning methods. The insights and findings in this dissertation suggest a promising direction for developing the user intent inference in both open-domain conversational agents and e-commerce search.
Table of Contents
1 Introduction and Motivation 1
1.1 User Intent Inference in Open-Domain
Conversational Agents . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Challenges for User Intent Inference in
Conversational Agents . . . . . . . . . . . . . . . . . . . . . . 4
1.2 User Intent Inference in Web Search Engines . . . . . . . . . . . . . . 6
1.2.1 Challenges for User Intent Inference in
E-commerce Search . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Dissertation Research Questions . . . . . . . . . . . . . . . . . . . . . 10
1.4 Contributions and Dissertation Structure . . . . . . . . . . . . . . . . 10
1.4.1 Contributions to Open-Domain
Conversational Agents . . . . . . . . . . . . . . . . . . . . . . 11
1.4.2 Contributions to E-commerce Search . . . . . . . . . . . . . . 12
1.5 Summary and Dissertation Structure . . . . . . . . . . . . . . . . . . 13
2 RelatedWork 14
2.1 User Intent Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 User Intent Inference in Open-Domain
Conversational Agents . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 Topic Classification: NLU . . . . . . . . . . . . . . . . . . . . 17
2.2.2 Dialogue Act Classification: NLU . . . . . . . . . . . . . . . . 19
2.2.3 Smart Topic Suggestion for Open-Domain Conversational Agents 21
2.3 User Intent Inference for Web Search . . . . . . . . . . . . . . . . . . 24
2.3.1 Joint Learning for User Intent Inference . . . . . . . . . . . . 26
2.3.2 Label Representation for User Intent Inference . . . . . . . . . 29
2.3.3 Pseudo-relevance feedback for User Intent Inference . . . . . . 31
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3 Integrating knowledge in User Intent Inference 33
3.1 Integrating Knowledge for Conversational Agents . . . . . . . . . . . 33
3.2 Integrating Knowledge for E-commerce Search . . . . . . . . . . . . . 38
3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4 User Intent inference in Conversational Agent 42
4.1 Emory IrisBot: An Open-Domain Conversational Bot for Personalized
Information Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.1 Language Understanding and Entity Recognition pipeline . . . 44
4.1.2 Domain-specific Components: . . . . . . . . . . . . . . . . . . 45
4.2 Topic and Intent Classification . . . . . . . . . . . . . . . . . . . . . . 46
4.2.1 Contextual Topic Classification . . . . . . . . . . . . . . . . . 46
4.2.2 Language Understanding and Entity Recognition Pipeline . . 48
4.2.3 Domain-specific Components . . . . . . . . . . . . . . . . . . 49
4.2.4 Topic and Intent Classification . . . . . . . . . . . . . . . . . . 49
4.2.5 Intent Classification . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2.6 Dialogue Manager and Response Ranking . . . . . . . . . . . 53
4.2.7 Dialogue Manager: Implementation . . . . . . . . . . . . . . . 54
4.2.8 Personalized Topic Suggestion . . . . . . . . . . . . . . . . . . 55
4.2.9 Personalized Sequential Topic Suggestion Model . . . . . . . . 56
4.2.10 Cross-Component topic suggestion . . . . . . . . . . . . . . . 58
4.2.11 Results and Discussion . . . . . . . . . . . . . . . . . . . . . 58
4.2.12 Topic and Intent Classifier: Internal Evaluation . . . . . . . . 59
4.2.13 Topic Suggestion Results . . . . . . . . . . . . . . . . . . . . . 60
4.2.14 E↵ects of Personalization on conversation behavior and ratings 61
4.3 ConCET: Entity-Aware Topic Classification for Open-Domain Conversational
Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.1 ConCET System Overview . . . . . . . . . . . . . . . . . . . . 63
4.3.2 Conversational Entity Linking . . . . . . . . . . . . . . . . . . 65
4.3.3 DBpedia Spotlight . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3.4 PMI-based Entity Linker (PMI-EL) . . . . . . . . . . . . . . . 66
4.3.5 ConCET: Concurrent Entity-Aware Topic Classifier . . . . . . 70
4.3.6 Textual Representation . . . . . . . . . . . . . . . . . . . . . . 70
4.3.7 Entity Representation . . . . . . . . . . . . . . . . . . . . . . 73
4.3.8 Merging and FeedForward Layer . . . . . . . . . . . . . . . . . 75
4.3.9 Conversational Dataset Overview . . . . . . . . . . . . . . . . 76
4.3.10 Amazon Alexa Prize 2018 . . . . . . . . . . . . . . . . . . . . 76
4.3.11 Obtaining True Labels for Alexa Data . . . . . . . . . . . . . 77
4.3.12 Self-Dialogue Dataset . . . . . . . . . . . . . . . . . . . . . . . 77
4.3.13 Synthetic Training Data Generation . . . . . . . . . . . . . . . 79
4.3.14 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 80
4.3.15 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.16 Training Parameters . . . . . . . . . . . . . . . . . . . . . . . 82
4.3.17 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.18 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 83
4.3.19 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.3.20 Detailed Performance Analysis . . . . . . . . . . . . . . . . . . 84
4.4 Contextual Dialogue Act Classification for Open-Domain Conversational
Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.4.1 Contextual Dialogue Act Classifier (CDAC) Model . . . . . . 89
4.4.2 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 93
4.4.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 95
4.5 Knowledge-Aware Contextual Topic Suggestion for Open-Domain Conversational
Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.5.1 Conversational Topic Suggestion (CTS): Problem Definition . 103
4.5.2 CTS-Seq Approach . . . . . . . . . . . . . . . . . . . . . . . . 104
4.5.3 System State and User Profile Features . . . . . . . . . . . . 105
4.5.4 CTS-Seq: Models . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.5.5 CRF Implementation of CTS-Seq: CTS-CRF . . . . . . . . . 106
4.5.6 Deep-learning based implementation of CTS-Seq: CTS-CNN
and CTS-RNN . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.5.7 Hybrid Sequential and Collaborative Filtering: CTS-Seq-CF . 111
4.5.8 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 113
4.5.9 Baseline 1: Popularity Method . . . . . . . . . . . . . . . . . . 113
4.5.10 Baseline 2: Collaborative Filtering (CF) . . . . . . . . . . . . 114
4.5.11 Baseline 3: Contextual Collaborative Filtering: Contextual-CF 115
4.5.12 Methods Compared . . . . . . . . . . . . . . . . . . . . . . . 116
4.5.13 Dataset: Amazon Alexa Prize 2018 . . . . . . . . . . . . . . . 117
4.5.14 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 117
4.5.15 Ground Truth Labels . . . . . . . . . . . . . . . . . . . . . . . 119
4.5.16 Training CTS-CRF Model . . . . . . . . . . . . . . . . . . . . 119
4.5.17 Results And Discussion . . . . . . . . . . . . . . . . . . . . . . 120
4.5.18 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.5.19 Feature Ablation on CTS . . . . . . . . . . . . . . . . . . . . 123
4.5.20 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5 User Intent inference in E-commerce Search 127
5.1 JointMap: Joint Query Intent Understanding For Modeling Intent Hierarchies
in E-commerce Search . . . . . . . . . . . . . . . . . . . . . 128
5.1.1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.1.2 Joint-Learning of High-Level Intent Tasks . . . . . . . . . . . 131
5.1.3 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.1.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . 134
5.1.5 Main Results and Ablation Analysis . . . . . . . . . . . . . . 136
5.2 Label Representation for Product Category
Mapping in E-commerce Search . . . . . . . . . . . . . . . . . . . . . 138
5.3 DeepCAT: Model and Implementation . . . . . . . . . . . . . . . . . 138
5.3.1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.3.2 Query Representation (Query2Vector Network) . . . . . . . . 140
5.3.3 Word-Category Representation . . . . . . . . . . . . . . . . . 140
5.3.4 Category-Category representation . . . . . . . . . . . . . . . . 141
5.3.5 Computation of Loss Function . . . . . . . . . . . . . . . . . . 141
5.4 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 142
5.4.1 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . 142
5.4.2 DeepCAT Experimental Design . . . . . . . . . . . . . . . . . 143
5.4.3 Parameter Setting . . . . . . . . . . . . . . . . . . . . . . . . 143
5.4.4 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . 144
5.4.5 Methods Compared . . . . . . . . . . . . . . . . . . . . . . . . 144
5.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 144
5.5.1 Results on Minority Classes . . . . . . . . . . . . . . . . . . . 145
5.5.2 Results on Traffic Buckets . . . . . . . . . . . . . . . . . . . . 145
5.6 Ablation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.7 APRF-Net: Attentive Pseudo-Relevance Feedback Network for Query
Categorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.7.1 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 147
5.7.2 Initial Retrieval Step . . . . . . . . . . . . . . . . . . . . . . . 148
5.7.3 Representation Layers . . . . . . . . . . . . . . . . . . . . . . 148
5.7.4 Corpus-Aware Attention Network . . . . . . . . . . . . . . . . 150
5.7.5 Dataset Overview . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.7.6 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . . 153
5.7.7 Empirical Results and Discussion . . . . . . . . . . . . . . . . 155
5.7.8 Ablation Analysis . . . . . . . . . . . . . . . . . . . . . . . . 156
5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
6 Discussion 159
6.1 Strengths of Knowledge-aware Based Models: Addressing RQ1, RQ2,
and RQ3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
6.1.1 Addressing RQ1 . . . . . . . . . . . . . . . . . . . . . . . . . . 160
6.1.2 Addressing RQ2 . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.1.3 Handling Data Imbalance: Addressing RQ3 . . . . . . . . . . 163
6.1.4 Handling tail and torso Queries: Addressing RQ3 . . . . . . . 164
6.1.5 Handling Overfitting: Addressing RQ3 . . . . . . . . . . . . . 166
6.2 Limitations of Using External
Sources of Information . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6.2.1 Higher Latency . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6.2.2 Aligning Knowledge Source with the Target Dataset . . . . . . 168
6.2.3 O✏ine vs. Online Evaluation . . . . . . . . . . . . . . . . . . 168
6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
7 Conclusions 170
7.1 Summary of the Results: . . . . . . . . . . . . . . . . . . . . . . . . . 170
7.2 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . 173
7.3 Summary of the Dissertation . . . . . . . . . . . . . . . . . . . . . . . 177
Bibliography 180
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Mot-clé | |
| Committee Chair / Thesis Advisor |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Knowledge-Aware User Intent Inference for Web Search and Conversational Agents () | 2021-12-08 22:37:11 -0500 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|