Presentation
Establishing a High-Performance and Productive Ecosystem for Distributed Execution of Python Functions Using Globus Compute
DescriptionDemocratizing access to the research computing ecosystem is critical for accelerating research progress. However, the gap between a high-level workload, such as Python in a Jupyter notebook, and the resources exposed by HPC systems is significant. Users must securely authenticate, manage network connections, deploy and manage software, provision and configure nodes, and manage workload execution. Globus Compute reduces these barriers by providing a managed, fire-and-forget model that enables execution of Python functions across any resource to which a user has access. In this paper we describe enhancements to Globus Compute that further reduce barriers to use of the research computing ecosystem: an asynchronous, future-based executor interface for submitting and monitoring tasks, shell and MPI-based function types, and a multi-user endpoint that can be deployed by administrators and used by authorized users.
Author/Presenters
