Presentation
Optimising Workflow Lifecycle Management: Development, HPC-Ready Containers Deployment and Provenance
DescriptionWith Exaflop systems already here, the application communities are eager to leverage these large and complex systems. The complexity is further increased by the applications' need to combine different aspects beyond traditional HPC solvers and simulators, with artificial intelligence (AI) and data analytics (DA). The eFlows4HPC project proposed a software stack and the HPC Workflows-as-a-Service (HPCWaaS) methodology to provide tools to simplify the development, deployment, execution, and reuse of workflows. These results are leveraged in the DT-GEO and CAELESTIS projects. These tools also aim to support the reproducibility, portability and ease of use of complex workflows.
The tutorial will focus on a set of tools and methodologies for managing the whole application workflow lifecycle. In particular, the tutorial will cover aspects of developing computational workflows with PyCOMPSs and new extensions to better integrate with AI and DA with examples from DT-GEO and CAELESTIS projects. The tutorial will also describe how to automatically record workflow provenance with PyCOMPSs to share FAIR workflows in public repositories, enabling their reproducibility. Finally, we will explain how to generate specific containers that leverage HPC systems features and use them in the workflow deployment phase. The tutorial will include hands-on sessions on different aspects.
Notes for Attendees:
The tutorial hands-on will be performed on linux-based systems and it is convenient that the attendees have some background at user-level of linux.The tutorial will focus on a set of tools and methodologies for managing the whole application workflow lifecycle. In particular, the tutorial will cover aspects of developing computational workflows with PyCOMPSs and new extensions to better integrate with AI and DA with examples from DT-GEO and CAELESTIS projects. The tutorial will also describe how to automatically record workflow provenance with PyCOMPSs to share FAIR workflows in public repositories, enabling their reproducibility. Finally, we will explain how to generate specific containers that leverage HPC systems features and use them in the workflow deployment phase. The tutorial will include hands-on sessions on different aspects.
Attendees can look at this video if they are not familiar (there are others in YouTube).
Also, check this manual on how to connect to MN5.
If your laptop runs on Windows, Install a ssh client such as putty.
The hands-on will be based on examples in Python. If you are not familiar with Python, you can learn some basics here here.
You also need to create a WorkflowHub user account in advance:
- Create your WorkflowHub account:
- Open https://workflowhub.eu/
- Click “Register”
- “Log in using GitHub” or
- Register with your e-mail
- Mandatory: First name, Last name, e-mail. Recommended: ORCID
- Confirm registration with received e-mail
- Join “COMPSs Tutorials” team (“eFlows4HPC” Space):
- Join a Team”
- Search for ”COMPSs Tutorials”
- Organization:
- Search for your institution not only by acronym, also with full words
- Can try also: Browse -> Organizations -> Country (filter)
- If not found: Create -> Organization
Event Type
Tutorial
TimeMonday, 18 November 20248:30am - 12pm EST
LocationB209
Applications and Application Frameworks
Emerging Technologies
Parallel Programming Methods, Models, Languages and Environments
TUT
Links