Presentation
Reshaping High-Energy Physics Applications for Near-Interactive Execution Using TaskVine
DescriptionHigh energy physics experiments produce petabytes of data annually that must be reduced to gain insight into the laws of nature. Early-stage reduction executes long-running, high-throughput workflows across thousands of nodes spanning multiple facilities to produce shared datasets. Later stages are typically written by individuals or small groups and must be refined and re-run many times for correctness. Reducing iteration times of later stages is key to accelerating discovery. We demonstrate our experience reshaping late-stage analysis applications on thousands of nodes. It is not enough merely to increase scale: it is necessary to make changes throughout the stack, including storage systems, data management, task scheduling, and application design. We demonstrate these changes when applied to two analysis applications built on open source data analysis frameworks (Coffea, Dask, TaskVine). We evaluate the performance of the applications on opportunistic campus clusters, showing effective scaling up to 7200 cores, thus producing significant speedup.
Event Type
Paper
TimeWednesday, 20 November 20244pm - 4:30pm EST
LocationB311
Accelerators
Applications and Application Frameworks
Modeling and Simulation
Numerical Methods
Task Parallelism
TP
Archive
view