Modeling Rich Interactions for Web Search Intent Inference, Ranking and Evaluation Public

Abstract

Billions of people interact with Web search engines daily and their interactions provide valuable clues about their interests and preferences. While modeling search behavior, such as queries and clicks on results, has been found to be effective for various Web search applications, the effectiveness of the existing approaches are limited by ignoring what the searcher sees (examination) and does (context) before clicking a result. This thesis aims to address these limitations by modeling and interpreting a wider range of searcher interactions, including mouse cursor movement and scrolling behavior (or pinching, zooming and sliding with a touch screen), that could be served as a proxy of searcher examination, contextualized in a search session. The thesis focuses on improving three fundamental and interrelated areas of Web search, namely, intent inference, ranking and evaluation. To improve the first area, the thesis developed techniques to infer the immediate search goals in a search session, along multiple dimensions, including top-level general intent (e.g., navigational vs. informational), commercial intent (e.g., research vs. purchase) and advertising receptiveness (i.e., interest in search ads). To improve the second area, the thesis developed the Post-Click Behavior (PCB) relevance prediction model for estimating the "intrinsic" document relevance from the examination and interaction patterns on the viewed result documents. To improve the third area, the thesis developed techniques for predicting search success, which include a principled framework to study Web search success, and fine-grained interaction models that improve prediction accuracy for both desktop and mobile settings. As demonstrated with extensive empirical evaluation, the developed techniques outperform the state-of-the-art methods that only use query, click and time signals, enabling more intelligent Web search systems.

Table of Contents

Contents

1 Introduction 1

1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Background and Related Work 10
2.1 Inferring Search Intent . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1 General Search Intent . . . . . . . . . . . . . . . . . . . . . 10
2.1.2 Topical Search Intent . . . . . . . . . . . . . . . . . . . . . 12
2.1.3 Commercial Search Intent . . . . . . . . . . . . . . . . . . 12
2.2 Estimating Document Relevance . . . . . . . . . . . . . . . . . . . 14
2.2.1 Click-through . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.2 Dwell Time . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.3 Examination . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.4 Personalization . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.5 Search Context . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Evaluating Search Experience . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Query-level . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Session-level . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.3 Multi-engine level . . . . . . . . . . . . . . . . . . . . . . 24

3 Inferring Search Intent 27

3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.1.1 Inferring General Search Intent . . . . . . . . . . . . . . . 29
3.1.2 Inferring Commercial Search Intent . . . . . . . . . . . . . 30
3.1.3 Application: Search Advertising . . . . . . . . . . . . . . . 31
3.2 Search and User Model . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.1 Search Model: Tasks and Goals . . . . . . . . . . . . . . . 33
3.2.2 User Model: Goal-driven Search . . . . . . . . . . . . . . . 34
3.3 Infrastructure, Features and Algorithms . . . . . . . . . . . . . . . 36
3.3.1 Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.3 Classifier Implementation . . . . . . . . . . . . . . . . . . 40
3.4 Inferring General Search Intent . . . . . . . . . . . . . . . . . . . . 42
3.4.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4.2 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.3 Methods Compared . . . . . . . . . . . . . . . . . . . . . . 44
3.4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . 45
3.5 Inferring Commercial Search Intent . . . . . . . . . . . . . . . . . 51
3.5.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . 51
3.5.2 Methods Compared . . . . . . . . . . . . . . . . . . . . . . 52
3.5.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . 52
3.5.4 Ad Click-through on Real Search Data . . . . . . . . . . . 54
3.6 Inferring Advertising Receptiveness . . . . . . . . . . . . . . . . . 55
3.6.1 Methods Compared . . . . . . . . . . . . . . . . . . . . . . 56
3.6.2 Data and Evaluation Metrics . . . . . . . . . . . . . . . . . 57
3.6.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . 57
3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4 Estimating Document Relevance 61
4.1 Landing Page Examination . . . . . . . . . . . . . . . . . . . . . . 64
4.2 Post-Click Behavior (PCB) Features . . . . . . . . . . . . . . . . . 68

4.2.1 Dwell Time . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.2.2 Result Rank . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.2.3 Cursor Movements . . . . . . . . . . . . . . . . . . . . . . 70
4.2.4 Vertical Scrolling . . . . . . . . . . . . . . . . . . . . . . . 70
4.2.5 Interactions in the Areas of Interest (AOI) . . . . . . . . . . 70
4.2.6 Task/Session-level Context . . . . . . . . . . . . . . . . . . 71
4.2.7 User Normalization . . . . . . . . . . . . . . . . . . . . . . 71
4.3 Relevance Estimation Models . . . . . . . . . . . . . . . . . . . . . 72
4.3.1 Ridge Regression (RR) . . . . . . . . . . . . . . . . . . . . 72
4.3.2 Bagging with Regression Trees (BRT) . . . . . . . . . . . . 72
4.4 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.4.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.4.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 74
4.4.3 Methods Compared . . . . . . . . . . . . . . . . . . . . . . 75
4.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 76
4.5.1 Feature Association with Relevance . . . . . . . . . . . . . 77
4.5.2 Predicting Document Relevance . . . . . . . . . . . . . . . 80
4.5.3 Re-ranking . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

5 Evaluating Search Experience 87
5.1 Predicting Search Success with UFindIt . . . . . . . . . . . . . . . 90
5.1.1 Search Success Model . . . . . . . . . . . . . . . . . . . . 91
5.1.2 Acquiring Search Behavior Data . . . . . . . . . . . . . . . 93
5.1.3 Predicting Search Success . . . . . . . . . . . . . . . . . . 94
5.1.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . 98
5.1.5 Real World Success Prediction:A Log-based Study . . . . . 100
5.2 Predicting Search Success with FSB . . . . . . . . . . . . . . . . . 106
5.2.1 Fine-grained Session Behavior (FSB) Features . . . . . . . 106
5.2.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5.2.3 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . 111

5.2.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . 111
5.3 Predicting Success in Mobile Search . . . . . . . . . . . . . . . . . 118
5.3.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 118
5.3.2 Results and Discussion . . . . . . . . . . . . . . . . . . . . 120
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

6 Conclusions and Future Work 126
6.1 Summary of Findings . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.2 Integrating Intent Inference, Ranking and Evaluation . . . . . . . . 128
6.2.1 Integrating Intent Inference . . . . . . . . . . . . . . . . . . 129
6.2.2 Integrating Relevance Estimation . . . . . . . . . . . . . . 129
6.2.3 Integrating Automatic Evaluation . . . . . . . . . . . . . . 130
6.2.4 Pre-click and Post-click Instrumentation . . . . . . . . . . . 130
6.2.5 Infrastructures for Offline and Online Deployment . . . . . 131
6.2.6 Evaluating the Deployed System . . . . . . . . . . . . . . . 132
6.3 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . 134
Bibliography 137

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Math and Computer Science
Degree	PhD
Submission	Dissertation
Language	English
Research Field	Computer Science Artificial Intelligence Information Science
Mot-clé	user behavior modeling data mining Web search log mining mouse tracking post-click behaivor Web search ranking pre-click behavior evaluation Web search sessions user studies machine learning fine-grained search interactions implicit feedback learning to rank intent inference Web search examination eye tracking information retrieval success prediction
Committee Chair / Thesis Advisor	Agichtein, Eugene, Emory University
Committee Members	White, Ryen, Microsoft Research Lu, James, Emory University Hongyuan, Zha, Georgia Institute of Technology Xiong, Li, Emory University

Dernière modification

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	Modeling Rich Interactions for Web Search Intent Inference, Ranking and Evaluation ()	2018-08-28 14:37:01 -0400	Press to Select an action Download

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions