Presentation
Enabling cross-Facility LLM pre-Training
DescriptionBig-tech companies pre-train SOTA LLMs on special-purpose, private HPCs, while public research centres lack the resources to compete. We advocate a new take on large model training, e.g., LLMs, called xFFL, which leverages federated learning as an enabling technique to exploit geographically distributed computing power to bridge such digital divide. This work introduces a proof-of-concept federated training of LLaMA-3 8B on three EuroHPC Top500 facilities, proving the viability of leveraging cross-facility publicly available computational power to sustain SOTA LLM workloads.