Presentation
Accelerating Communications in High-Performance Scientific Workflows
DescriptionAdvances in networks, accelerators, and cloud services encourage programmers to reconsider where to compute — such as when fast networks make it cost-effective to compute on remote accelerators despite added latency. Workflow and cloud-hosted serverless computing frameworks can manage multi-step computations spanning federated collections of cloud, high-performance computing, and edge systems, but passing data among computational steps remains a challenge when applications are a composition of multiple distinct software with differing communication and patterns.
This work introduces a new programming paradigm that decouples data flow from control flow by extending the pass-by-reference model to distributed applications. ProxyStore, developed here, implements this paradigm through object proxies that act as wide-area object references with just-in-time resolution. The proxy model enables producers to communicate data unilaterally, transparently, and efficiently to both local and remote consumers. This decoupling enables the dynamic selection of different data movement methods, depending on what data are moved, where data are moved, or when data are moved — a longstanding challenge in distributed applications.
The efficacy of the proxy paradigm is further understood through four high-level proxy-based programming patterns applied to real-world computational science applications. These high-level patterns — distributed futures, streaming, ownership, and stateful actors — make the power of the proxy paradigm accessible for more complex and dynamic distributed program structures. ProxyStore is evaluated through standardized benchmark suites, introduced here, and meaningful science applications, spanning bioinformatics, federated learning, and molecular design, in which substantial improvements in runtime, throughput, and memory usage are demonstrated.
This work introduces a new programming paradigm that decouples data flow from control flow by extending the pass-by-reference model to distributed applications. ProxyStore, developed here, implements this paradigm through object proxies that act as wide-area object references with just-in-time resolution. The proxy model enables producers to communicate data unilaterally, transparently, and efficiently to both local and remote consumers. This decoupling enables the dynamic selection of different data movement methods, depending on what data are moved, where data are moved, or when data are moved — a longstanding challenge in distributed applications.
The efficacy of the proxy paradigm is further understood through four high-level proxy-based programming patterns applied to real-world computational science applications. These high-level patterns — distributed futures, streaming, ownership, and stateful actors — make the power of the proxy paradigm accessible for more complex and dynamic distributed program structures. ProxyStore is evaluated through standardized benchmark suites, introduced here, and meaningful science applications, spanning bioinformatics, federated learning, and molecular design, in which substantial improvements in runtime, throughput, and memory usage are demonstrated.