BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260924T172458Z
LOCATION:B313
DTSTART;TZID=America/New_York:20241118T171000
DTEND;TZID=America/New_York:20241118T173000
UID:submissions.supercomputing.org_SC24_sess757_ws_ai4s120@linklings.com
SUMMARY:AstroMLab 2: Benchmarking Specialised LLMs for Astronomy
DESCRIPTION:Rui Pan (University of Illinois Urbana-Champaign), Tuan Dung N
 guyen (University of Pennsylvania), Alberto Accomazzi (Harvard University)
 , Tirthankar Ghosal (Oak Ridge National Laboratory (ORNL)), and Yuan-Sen T
 ing (The Ohio State University)\n\nContinual pretraining of large language
  models on domain-specific data has been proposed to enhance performance o
 n downstream tasks. In astronomy, the previous absence of astronomy-focuse
 d benchmarks has hindered objective evaluation of these specialized LLM mo
 dels. Leveraging a recent initiative to curate high-quality astronomical M
 CQs, this study aims to quantitatively assess specialized LLMs in astronom
 y. We find that the previously released AstroLLaMA series, based on LLaMA-
 2-7B, underperforms compared to the base model. We demonstrate that this p
 erformance degradation can be partially mitigated by utilizing high-qualit
 y data for continual pretraining, such as summarized text from arXiv. Desp
 ite the observed catastrophic forgetting in smaller models, our results in
 dicate that continual pretraining on the 70B model can yield improvements.
  However, the current supervised fine-tuning dataset still constrains the 
 performance of instruct models. In conjunction with this study, we introdu
 ce a new set of models, AstroLLaMA-3-8B and AstroLLaMA-2-70B, building upo
 n the previous AstroLLaMA series.\n\nTag: Artificial Intelligence/Machine 
 Learning\n\nRegistration Category: Workshop Reg Pass\n\nSession Chairs: Mu
 rali Emani (Argonne National Laboratory (ANL)); Gokcen Kestor (Barcelona S
 upercomputing Center (BSC); University of California, Merced); and Dong Li
  (University of California, Merced)\n\n
END:VEVENT
END:VCALENDAR