MACHINE LEARNING-DRIVEN RESOURCE ALLOCATION FOR DYNAMIC WORKLOADS IN HPC SYSTEMS

Abstract

High-performance computing (HPC) systems now support workloads that shift constantly in shape and intensity, from tightly coupled simulations to bursty AI training jobs. Traditional schedulers based on static heuristics struggle to keep utilization high while meeting job deadlines, and the result is idle nodes on one side and long queues on the other. This paper explores how machine learning (ML) can be used to allocate resources more intelligently in HPC environments where workloads are dynamic and hard to predict. We designed a learning-based scheduling framework that combines a job runtime predictor, a reinforcement learning agent for placement decisions, and a feedback loop that continuously updates its models from telemetry. Experiments were carried out on a simulated 512-node cluster running a mix of scientific and AI workloads drawn from public traces. Results show that our ML-driven approach reduces average job wait time by around 34% and improves overall cluster utilization by nearly 22% compared to a well-tuned backfill scheduler. Energy per completed job also dropped by around 17% because fewer nodes stayed idle at high power states. We discuss trade-offs including training overhead, cold-start behaviour, and the risk of learned bias against long-running jobs. Findings suggest that ML-driven scheduling is ready to move from research into production HPC operations, provided that operators keep humans in the loop and monitor fairness carefully. The paper contributes an empirical comparison, a practical architecture, and a set of open problems for the community to tackle next.