Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue Open Access

Abstract

Emotion recognition in conversation (ERC) has been widely studied, yet the

application of Large Language Models (LLMs) to continuous dimensional emotion

evaluation in multimodal dialogue remains largely unexplored. This thesis proposes

a multimodal LLM-based framework that performs two independent tasks on the

IEMOCAP dataset: discrete emotion recognition across six categories, and dimen-

sional emotion evaluation along the Valence–Arousal–Dominance (VAD) continuum.

Following the SpeechCueLLM approach, acoustic information is incorporated as

natural language descriptions of pitch, volume, and speaking rate, enabling LLMs to

access non-lexical cues without architectural modification. We evaluate five models

spanning the LLaMA and GPT families under zero-shot prompting, few-shot prompt-

ing, and LoRA parameter-efficient fine-tuning. Results show that LoRA fine-tuned

LLaMA models substantially outperform prompt-engineered GPT models on both

tasks, which is somewhat counterintuitive given the larger scale of GPT models, and

we attribute this performance gap to domain adaptation rather than model capacity.

Our best model achieves a Valence Concordance Correlation Coefficient (CCC) of

0.7822, establishing a new state-of-the-art on IEMOCAP for this dimension. The error

analysis reveals that GPT models have specifically high confusion rates for certain

emotions, showing its lack of adaptation to this domain. Meanwhile, the performance

asymmetry across VAD dimensions is explained by the annotator agreement on those

three dimensions: the reliability hierarchy in IEMOCAP annotations is mirrored

directly in the models’ performance on VAD dimensions.

Table of Contents

Introduction

Background

Approach

Experiments

Analysis

Conclusion

About this Honors Thesis

Rights statement

• Permission granted by the author to include this thesis or dissertation in this repository. All rights reserved by the author. Please contact the author for information regarding the reproduction and use of this thesis or dissertation.

School	Emory College
Department	Computer Science
Degree	B.S.
Submission	Honors Thesis
Language	English
Research Field	Computer Science
Keyword	Emotion Recognition Natural Language Processing Dialogue Large Language Models Emotion Evaluation
Committee Chair / Thesis Advisor	Jinho Choi, Emory University
Committee Members	Joyce Ho, Emory University Philip Wolff, Emory University

Last modified

Primary PDF

Thumbnail	Title	Date Uploaded	Actions

Supplemental Files

Thumbnail	Title	Date Uploaded	Actions
	/concern/parent/41687k115/file_sets/nz806135r ()	2026-04-14 21:39:38 -0400	Press to Select an action Download