Presentation
Prompt Phrase Ordering Using Large Language Models in HPC: Evaluating Prompt Sensitivity
DescriptionLarge language models (LLMs) often require well-designed prompts for effective responses, but optimizing prompts is challenging due to prompt sensitivity, where small changes can cause significant performance variations. This study evaluates prompt performance across all permutations of independent phrases to investigate prompt sensitivity and robustness. We used two datasets: GSM8k, for mathematical reasoning, and a custom prompt for summarizing database metadata. Performance was assessed using the llama3-instruct-7B model on Ollama and parallelized in a high-performance computing environment. We compared phrase indices in the best and worst prompts and used Hamming distance to measure performance changes between phrase orderings. Results show that prompt phrase ordering significantly affects LLM performance, with Hamming distance indicating that changes can dramatically alter scores, often by chance. This supports existing findings on prompt sensitivity. Our study highlights the challenges in prompt optimization, indicating that modifying phrases in a successful prompt does not guarantee another successful prompt.
Event Type
ACM Student Research Competition: Graduate Poster
ACM Student Research Competition: Undergraduate Poster
Doctoral Showcase
Posters
TimeTuesday, 19 November 202412pm - 5pm EST
LocationB302-B305
TP
XO/EX