Presentation
A Comparison Study of Open Source LLMs for HPC Ticket Answering
DescriptionWe are designing an automatic ticket answering service for computing centers such as Texas Advanced Computing Center (TACC), National Center for Supercomputing Applications (NCSA), and San Diego Supercomputer Center (SDSC). In this work, we investigate the capability and feasibility of open source language models (LLMs) on the ticket answering task. We compare four open source LLMs (OPT-6.7B, Falcon-7B, Llama 2-7B, and Llama 3.1-8B) by fine-tuning them with a curated dataset with over 110,000 historical question/answer pairs. Our results show that fine-tuned LLMs are capable of generating reasonable answers. Llama-7B has a lower validation loss and perplexity than OPT-6.7B and Falcon-7B. We also observe that fine-tuning with LoRA introduces non-trivial generalization loss compared with dense fine-tuning. We will design an evaluation dataset and perform quantitative evaluation for the three LLMs in the future.

Event Type
ACM Student Research Competition: Graduate Poster
ACM Student Research Competition: Undergraduate Poster
Doctoral Showcase
Posters
TimeTuesday, 19 November 202412pm - 5pm EST
LocationB302-B305
TP
XO/EX