Automatic Synthesis of Application Analogs for High-Performance Computing Restricted; Files Only

Abstract

High-performance computing advances by measurement, and that measurement is costly. The field's response is to build an application analog: a proxy for a real HPC object that preserves the computational, communication, memory, I/O or performance behaviors relevant to a particular research or evaluation objective. Benchmarks and mini-apps are the familiar examples. Building analogs by hand is slow, which has motivated a decade of work on automatic synthesis, and that work judges its results by resemblance to the original. This dissertation demonstrates that statistical, generative, and large-language-model methods can automate analog synthesis, but that resemblance alone is not sufficient for acceptance. It also shows that analogs are no longer built only for applications: an HPC-specialized code model is an analog of parallelization expertise, and a code-LLM benchmark is an analog of an evaluation.

Three independent studies test that claim. The first compares seven generative models, across four families, on call-level MPI trace synthesis for four applications; the statistical baseline ranks second on graph-structural fidelity yet misses predicted communication time by 69.7%, while the variational autoencoder, third on structure, proves the most reliable in simulation. The second evaluates four open language models, in base and fine-tuned configurations, across thirteen computational motifs, and finds the value of task-specific adaptation inversely proportional to base competence (r = −0.973) while correctness and speedup are statistically independent (r = 0.27, p = 0.40). The third characterizes eighteen code-LLM versions across seven families and finds that what separates families is institutional and philosophical rather than architectural. A model's family cannot be recovered from the code it generates at better than chance, and the one model to score perfect correctness does so 34,575× slower than the sequential baseline. In each case, an evaluation restricted to resemblance leads to an unreliable recommendation. The studies establish that an analog must be validated against the decision it informs, not against its resemblance to what it replaces, and from this the dissertation derives a utility-first evaluation protocol.

Table of Contents

CONTENTS

1 Introduction — 1

1.1 Motivation: The Cost of Observation — 1

1.2 The Application Analog — 3

1.3 Research Questions — 6

1.4 Contributions — 7

1.5 The Case for One Dissertation — 9

1.6 Organization of the Dissertation — 10

2 Related Work: The Automatic Synthesis of Application Analogs — 12

    2.0.1 One construction, several names — 12

    2.0.2 Why the construction, and not the artifact — 14

    2.0.3 The empirical baseline — 15

2.1 Forty-Five Years of Hand-Built Analogs — 16

2.2 How the Field Judges an Analog — 19

    2.2.1 The five characteristics — 19

    2.2.2 Both kinds of evidence are the same kind of evidence — 20

    2.2.3 The measures, and what they have in common — 21

2.2.4 The acknowledged challenges, and the one that is not acknowl-

    edged — 23

2.3 Automatic Synthesis: Definition and Pipeline — 24

    2.3.1 Definition and pipeline — 24

2.3.2 Why automation makes the representation problem worse, not

    better — 26

2.4 Four Families of Synthesis Method — 27

    2.4.1 Rule-based and search-based synthesis — 27

    2.4.2 Evolutionary methods — 28

    2.4.3 Swarm intelligence — 29

    2.4.4 Machine learning — 30

    2.4.5 Large language models — 32

2.5 Analogs of Expertise and of Models — 34

2.6 Seven Challenges, Reread — 36

2.7 Gaps Addressed by This Dissertation — 38

2.8 Summary — 40

3 Synthetic Generation of Call-Level MPI Communication Traces: A Comparative Study Across Statistical, Adversarial, Variational, and

Language-Model Generators — 41

3.1 Introduction and Motivation — 41

    3.1.1 Contributions — 43

    3.1.2 Chapter Organization — 44

3.2 Background and Related Work — 44

    3.2.1 MPI Trace Collection and Characterization — 45

    3.2.2 Synthetic Trace Generation in HPC — 45

    3.2.3 Generative Trace Synthesis in the Networking Domain — 46

    3.2.4 Temporal-Graph Generative Models — 47

    3.2.5 Generative Model Families and Learning Paradigms — 48

3.3 Dataset and Representation — 48

    3.3.1 Application Suite — 48

    3.3.2 Tracing and Event Extraction — 49

    3.3.3 Temporal Weighted Directed Graph — 50

3.4 Generative Methods — 51

    3.4.1 Method 1 — Statistical Baseline (Gaussian Copula) — 52

    3.4.2 Method 2 — Generative Adversarial Network — 53

    3.4.3 Method 3 — Temporal Graph GAN — 53

    3.4.4 Method 4 — Transfer Learning — 54

    3.4.5 Method 5 — Meta-Learning (FOMAML) — 54

    3.4.6 Method 6 — Variational Autoencoder ( Neural Network) — 55

    3.4.7 Method 7 — Transformer Language Model — 55

3.5 Evaluation Framework — 56

    3.5.1 Structural Fidelity via Centrality Divergence — 56

    3.5.2 Communication-Intensity Fidelity — 56

    3.5.3 Temporal Fidelity — 57

    3.5.4 Simulation-Based Utility — 57

3.6 Results — 58

    3.6.1 Structural Fidelity — 58

    3.6.2 Communication-Intensity Fidelity — 60

    3.6.3 Temporal Fidelity — 65

    3.6.4 Simulation-Based Utility: The Decisive Test — 67

    3.6.5 Synthesis of Rankings — 70

3.7 Discussion — 70

    3.7.1 Structural Fidelity Is Necessary but Not Sufficient — 70

    3.7.2 Why the Likelihood-Based Generators Lead on Utility — 71

    3.7.3 Why Meta-Learning and the Temporal GAN Fail Here — 71

    3.7.4 Lessons on Architecture, Data, and Inductive Bias — 72

    3.7.5 Threats to Validity and Limitations — 72

3.8 Conclusion — 73

4 Large Language Models for OpenMP Pragma Completion — 75

4.1 Introduction — 75

4.2 Background and Motivation — 76

  4.2.1 OpenMP and the cost of parallelization expertise — 76

  4.2.2 Large language models for code — 77

  4.2.3 The task: OpenMP pragma completion — 77

  4.2.4 The ML×HPC gap — 79

4.3 Related Work — 79

  4.3.1 Language models for parallel and HPC code — 79

  4.3.2 Transformer-based parallelization advisors — 80

  4.3.3 Classical automatic parallelization — 80

  4.3.4 Evaluation of generated parallel code — 81

4.4 Research Questions — 81

4.5 Task Formulation — 82

  4.5.1 The completion task — 82

  4.5.2 Scope — 83

4.6 Dataset — 83

  4.6.1 Source corpus and collection — 83

  4.6.2 Preprocessing pipeline — 84

  4.6.3 Pragma extraction and structural annotation — 84

  4.6.4 Task-specific preparation for this study — 85

  4.6.5 Decontamination — 86

  4.6.6 Instruction formatting and the completion-only objective — 86

4.7 Models — 86

  4.7.1 Selection criteria — 86

  4.7.2 Fine-tunable lineup — 87

  4.7.3 Models excluded from the lineup — 88

  4.7.4 Base and fine-tuned configurations — 89

4.8 Fine-Tuning Methodology — 89

  4.8.1 Parameter-efficient adaptation — 89

  4.8.2 Completion-only objective — 89

  4.8.3 Hyperparameters and reproducibility — 90

  4.8.4 Compute environment — 90

4.9 Prompt construction: motifs and strategies — 91

  4.9.1 The thirteen computational motifs — 91

  4.9.2 Prompting strategies — 92

4.10 Evaluation Framework — 94

  4.10.1 Correctness — 95

  4.10.2 Performance — 98

  4.10.3 Comparison arms — 99

  4.10.4 Failure taxonomy — 99

  4.10.5 Problem suite and the difficulty gradient — 99

  4.10.6 Statistical protocol — 101

4.11 Experimental Setup — 101

4.12 Results: OpenMP pragma completion — 102

  4.12.1 Composite correctness and the effect of fine-tuning — 103

  4.12.2 Directive emission versus generation completeness — 104

  4.12.3 Motif difficulty: predicted versus measured — 108

  4.12.4 Prompting effects — 112

  4.12.5 Performance of correct generations — 113

  4.12.6 Generation-process efficiency — 124

  4.12.7 Which model performs best — 125

  4.12.8 Failure analysis — 126

4.13 Discussion — 127

4.14 Threats to Validity — 129

4.15 Summary — 131

4.16 Future Work — 133

5 A Feature-Mapping Framework for Predicting LLM Behavior on

HPC Coding Tasks — 135

5.1 Introduction — 135

    5.1.1 Research Questions — 136

    5.1.2 Scope — 137

5.2 Background and Motivation — 138

    5.2.1 Three Notions of Stability — 138

    5.2.2 The Stability Typology — 138

    5.2.3 The Lineage Bridge — 139

5.3 Related Work — 140

    5.3.1 Code LLM Evaluation — 140

    5.3.2 Parallel Code Generation — 140

    5.3.3 Parallel Computational Motifs — 141

    5.3.4 Architectural Lineage — 141

    5.3.5 Gap Addressed — 141

5.4 Methods — 142

    5.4.1 Stage 1: Feature Characterization — 142

    5.4.2 Stage 2: Behavioral Characterization — 148

    5.4.3 Stage 3: Performance Evaluation — 155

    5.4.4 Reproducibility — 159

5.5 Results — 159

    5.5.1 Stage 1 Results — 159

    5.5.2 Stage 2 Results — 168

    5.5.3 Stage 3 Results — 176

5.6 Discussion — 184

    5.6.1 Synthesis Across Stages — 184

    5.6.2 The Feature-to-Performance Link — 185

5.7 Limitations — 186

    5.7.1 Construct Validity — 186

    5.7.2 Internal Validity — 186

    5.7.3 External Validity — 187

    5.7.4 Conclusion Validity — 187

5.8 Summary — 188

6 Key Highlights: What the Three Studies Say Together — 190

6.1 Purpose of This Chapter — 190

6.2 Three Analogs, One Question — 190

6.3 Highlight 1 — Fidelity Does Not Entail Utility, and the Gap Is an

    Ordering — 191

6.4 Highlight 2 — The Analog’s Representation Sets a Ceiling the Method

    Cannot Exceed — 194

6.5 Highlight 3 — Adaptation Buys Most Where Competence Is Least — 195

6.6 Highlight 4 — Convergence and Shared Blind Spots — 197

6.7 Highlight 5 — Negative Results, and the Instrument Question — 198

6.8 Synthesis: A Utility-First Evaluation Protocol — 200

6.9 The Highlights in One Table — 201

6.10 What the Agreement Is Worth — 203

7 Conclusion — 204

7.1 What This Dissertation Established — 204

7.2 Answers to the Research Questions — 206

7.3 Limitations of the Whole — 207

7.4 Future Directions — 208

7.5 Closing — 211

A Per-Method Temporal Comparisons (Chapter 3) — 213

B Detailed pass@k breakdown (Chapter 4) — 216

B.1 Qwen3.6-27B, pass@k by motif — 216

B.2 HPC-Coder-V2-16B, pass@k by motif — 217

B.3 Relationship between pass@k and the composite — 218

C Exact motif interfaces and prompt templates (Chapter 4) — 219

C.1 Motif signatures and canonical kernels — 219

C.2 Exact prompt templates — 220

D Stage 1 Feature Dataset (Chapter 5) — 222

E Stage 2 Similarity Matrix (Chapter 5) — 225

E.1 Within-Family Pairs — 226

F Glossary of Terms (Chapter 5) — 227

Bibliography — 230

About this Dissertation

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Laney Graduate School
Department	Computer Science and Informatics
Degree	Ph.D.
Submission	Dissertation
Language	English
Research Field	Computer Science
Keyword	Performance evaluation High-performance computing Generative models Application analogs MPI communication traces Large language models for code
Committee Chair / Thesis Advisor	Simon Garcia de Gonzalo, Sandia National Lab Dorian Arnold , Emory University
Committee Members	Tony Pan, Emory University Liang Zhao, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions
	File download under embargo until 20 August 2027	2026-08-07 20:30:00 -0400	File download under embargo until 20 August 2027

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions