BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20250626T234543Z
LOCATION:B302-B305
DTSTART;TZID=America/New_York:20241119T120000
DTEND;TZID=America/New_York:20241119T170000
UID:submissions.supercomputing.org_SC24_sess487_post261@linklings.com
SUMMARY:A Comparison Study of Open Source LLMs for HPC Ticket Answering​
DESCRIPTION:Fangru Linghu, Mingkai Zheng, and Zhao Zhang (Rutgers Universi
 ty)\n\nWe are designing an automatic ticket answering service for computin
 g centers such as Texas Advanced Computing Center (TACC), National Center 
 for Supercomputing Applications (NCSA), and San Diego Supercomputer Center
  (SDSC). In this work, we investigate the capability and feasibility of op
 en source language models (LLMs) on the ticket answering task. We compare 
 four open source LLMs (OPT-6.7B, Falcon-7B, Llama 2-7B, and Llama 3.1-8B) 
 by fine-tuning them with a curated dataset with over 110,000 historical qu
 estion/answer pairs. Our results show that fine-tuned LLMs are capable of 
 generating reasonable answers. Llama-7B has a lower validation loss and pe
 rplexity than OPT-6.7B and Falcon-7B. We also observe that fine-tuning wit
 h LoRA introduces non-trivial generalization loss compared with dense fine
 -tuning. We will design an evaluation dataset and perform quantitative eva
 luation for the three LLMs in the future.\n\nRegistration Category: Tech P
 rogram Reg Pass, Exhibits Reg Pass\n\nSession Chairs: Ayesha Afzal (Friedr
 ich-Alexander University, Erlangen-Nuremberg; Erlangen National High Perfo
 rmance Computing Center); Sally Ellingson (University of Kentucky); and Al
 an Sussman (University of Maryland)\n\n
END:VEVENT
END:VCALENDAR