Training Search and Ranking Models with Minimal Supervision Öffentlichkeit
Chandradevan, Ramraj (Fall 2024)
Abstract
Searching for relevant or useful information in large document collections or the Web is challenging, particularly to support diverse search tasks, domains, or languages. The current State-of-the-Art methods use a Learning to Rank approach, typically using deep Neural Networks pretrained on large ranking datasets such as MS-MARCO. These methods have shown impressive performance without additional training (a.k.a in a zero-shot setting), since acquiring sufficiently large domain specific training data is often not feasible. However, as zero-shot ranking does not take advantage of the target domain information, there is a potential for improvement, especially for specialized domains and tasks. However, to fine- tune a large Neural Net-based ranker for a new domain requires large amounts of labeled training data. The central challenge of this thesis is whether state-of-the-art Neural rankers can be adapted to new domains, with minimal supervision. To solve this challenge, this thesis proposes several unsupervised and weakly-supervised approaches to learning to rank, aiming to fill a critical gap in prior literature, namely how to automatically train a (Neural) ranker with minimal supervision. My research focuses on three primary research questions: (1) Can adapting query representation with domain information improve ranking performance? (2) Can ranking models be effectively fine-tuned with minimal or no supervision? and (3) Can ranking models be further adapted for specific downstream tasks, such as Retrieval Augmented Generation (RAG), using weak or no supervision? To address these questions, this thesis investigates multiple techniques, including LLM self-referencing, pseudo-labeling, synthetic query generation, ensemble-prompting, contrastive fine-tuning, and query enrichment. These approaches leverage diverse target information such as documents, synthetic queries, and weak labels. Lastly, this thesis proposes a list of experiments to evaluate the proposed approaches on ranking benchmarks across multiple domains, such as BEIR, and across multiple languages, such as MIRACL and CLEF, and across retrieval tasks, such as CRAG, to demonstrate their effectiveness and robustness.
Table of Contents
1 Introduction ................................................................... 1
1.1 Overview of Thesis.............................................................. 3
2 Related Work ................................................................... 5
2.1 Transfer Learning.............................................................. 5
2.2 Domain Adaptation(DA).......................................................... 6
2.3 Source-Free Domain Adaptation(SFDA) ............................................ 6
2.4 Test Time Domain Adaptation(TTDA)................................................ 7
2.5 Learning-to-Rank Overview ..................................................... 7
2.6 Supervised Training Neural Rankers .............................................. 8
2.7 Weakly Supervised Training Neural Rankers......................................... 9
2.8 Unsupervised Training Neural Rankers............................................. 12
2.9 LLM based Neural Rankers......................................................... 13
3 Unsupervised Test Time Query Representation .................................... 18
3.1 Query Expansion(QE) ........................................................... 19
3.2 Pseudo Relevance Feedback(PRF).................................................. 19
3.3 Overview of Proposed Query Representation Approaches .............................. 20
3.4 Learning to Enrich Query Representation .......................................... 21
3.5 Unsupervised Ensemble Generative Prompting to Enrich Query Representation .... 28
4 Unsupervised Ranker Adaptation ................................................. 35
4.1 Overview of Proposed Ranker Adaptation Approaches ................................. 36
4.2 Preliminary Study on Domain Fine-tuning........................................... 37
4.3 Diversified Synthetic Query Generation for Ranker Fine-tuning ................ 41
4.4 Multilingual Synthetic Query Generation for Ranker Fine-tuning ............... 53
5 Task-Specific Neural Ranker Training ........................................... 63
5.1 Overview of Proposed Task Fine-tuning Approaches .................................. 65
5.2 Task-Specific Contrastive Fine-tuning .......................................... 65
5.3 Event-Aware TaskFine-tuning ................................................... 72
5.4 Task Fine-tuning Neural Ranker in a RAG System....................................... 79
6 Conclusions and Future Work .................................................... 99
6.1 Thesis Contributions .......................................................... 100
6.2 Summary of Results.............................................................. 100
6.3 Limitations .................................................................. 102
6.4 Future Work.................................................................... 103
A Appendix for TFT-RAG ........................................................... 105
A.1 Reranked References for TFT-RAG Prediction........................................ 105
Bibliography ..................................... ............................... 107
About this Dissertation
| School | |
|---|---|
| Department | |
| Degree | |
| Submission | |
| Language |
|
| Research Field | |
| Stichwort | |
| Committee Chair / Thesis Advisor |
Primary PDF
| Thumbnail | Title | Date Uploaded | Actions |
|---|---|---|---|
|
|
Training Search and Ranking Models with Minimal Supervision () | 2024-12-11 19:24:23 -0500 |
|
Supplemental Files
| Thumbnail | Title | Date Uploaded | Actions |
|---|