BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20250626T234541Z
LOCATION:B308
DTSTART;TZID=America/New_York:20241121T093000
DTEND;TZID=America/New_York:20241121T100000
UID:submissions.supercomputing.org_SC24_sess398_pap377@linklings.com
SUMMARY:Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LL
 Ms under Shadowy Sparsity
DESCRIPTION:Tuowei Wang (Tsinghua University, China); Kun Li (Microsoft Re
 search); Zixu Hao (Tsinghua University, China); Donglin Bai (Microsoft Res
 earch); Ju Ren and Yaoxue Zhang (Tsinghua University, China); and Ting Cao
  and Mao Yang (Microsoft Research)\n\nThe adaptation of pre-trained LLMs t
 o diverse downstream tasks through fine-tuning is essential for numerous a
 pplications. However, the inefficiency of parameter-efficient fine-tuning 
 (PEFT) techniques presents significant challenges regarding time investmen
 ts and operational costs. In this paper, we first introduce a nuanced form
  of sparsity, termed Shadowy Sparsity, which is distinctive in fine-tuning
  and has not been adequately addressed for acceleration. Under Shadowy Spa
 rsity, we propose Long Exposure, an efficient system to accelerate PEFT fo
 r LLMs. Long Exposure comprises three key components:  Shadowy-sparsity Ex
 poser employs a prolonged sensing range to capture more sparsity details u
 nder shadowy sparsity; Sequence-oriented Predictor provides efficient yet 
 accurate predictions to handle large-sequence inputs and constantly evolvi
 ng parameters; and Dynamic-aware Operator facilitates more structured comp
 utational patterns and coalesced memory accesses to address dynamic sparse
  operations. Comprehensive evaluations demonstrate that Long Exposure outp
 erforms state-of-the-arts with up to 2.49x speedup in end-to-end fine-tuni
 ng, offering promising advancements in PEFT acceleration.\n\nTag: Algorith
 ms, Artificial Intelligence/Machine Learning, Heterogeneous Computing, Per
 formance Optimization\n\nRegistration Category: Tech Program Reg Pass\n\nS
 ession Chair: Ramakrishnan Kannan (Georgia Institute of Technology)\n\n
END:VEVENT
END:VCALENDAR