Presentation
SIGN IN TO VIEW THIS PRESENTATION Sign In
Serverless Computing for Dynamic HPC Workflows
DescriptionContainers have become an important component for scientific workflows, enhancing reproducibility, portability, and isolation when coupled with workflow management systems. However, integrating containers with these systems can be complex, potentially hindering wider adoption. Serverless platforms offer a solution by providing a layer of abstraction over container orchestrators, simplifying management while introducing event-driven capabilities.
This paper presents a novel integration of serverless with workflow management systems to optimize scientific workflow execution. Our approach leverages serverless functions to dynamically provision containers for workflow tasks, resulting in up to 30\% faster execution. We found that performance can be further improved by reusing containers between multiple different tasks that were provisioned by the serverless platform. These findings demonstrate the utility of combining specialized container orchestration with established workflow management to streamline scientific computing, improve resource utilization, and accelerate time-to-results. Serverless' event-driven architecture enables efficient resource scaling, aligning with the dynamic nature of scientific workloads.
This paper presents a novel integration of serverless with workflow management systems to optimize scientific workflow execution. Our approach leverages serverless functions to dynamically provision containers for workflow tasks, resulting in up to 30\% faster execution. We found that performance can be further improved by reusing containers between multiple different tasks that were provisioned by the serverless platform. These findings demonstrate the utility of combining specialized container orchestration with established workflow management to streamline scientific computing, improve resource utilization, and accelerate time-to-results. Serverless' event-driven architecture enables efficient resource scaling, aligning with the dynamic nature of scientific workloads.
