Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
Commit to Compute: Leveraging CI/CD Tools to Automate HPC Tasks
DescriptionA recent case study from the Hartree Centre is presented. In this project, we developed a stack that integrates GitLab CI/CD tooling with a Slurm-enabled HPC environment (deployed in AWS ParallelCluster). The stack enables the triggering of computationally expensive tasks automatically from code commits, as well as the ability to pin task outputs to those commits via artifacts.
This case study showcases a template for architectures that enable engineers to work on designs/models independently with familiar tooling and share outputs, which may be dependencies for collaborators, automatically; all without needing to interact directly with an unfamiliar HPC environment. This presentation will focus on the challenges we faced in developing the architecture, and how we overcame those challenges. In particular, we will discuss our approach to defining "templated" computational tasks to run on HPC, and some pitfalls in creating client CLI interfaces for REST APIs.
This case study showcases a template for architectures that enable engineers to work on designs/models independently with familiar tooling and share outputs, which may be dependencies for collaborators, automatically; all without needing to interact directly with an unfamiliar HPC environment. This presentation will focus on the challenges we faced in developing the architecture, and how we overcame those challenges. In particular, we will discuss our approach to defining "templated" computational tasks to run on HPC, and some pitfalls in creating client CLI interfaces for REST APIs.
