TKS x Microsoft Challenge

TKS, The Knowledge Society, is a ten month innovation program for teenagers that runs challenges with real companies, and I did this one through them. It was their short format: one week from brief to deck, five of us on a team. We took Microsoft data centre efficiency, specifically the GPUs that Azure rents out to run AI workloads. The research, the numbers and the proposal are all ours, put together as a TKS exercise rather than as work done with Microsoft.

The problem is that GPUs are partitioned at job start for worst-case demand and never rebalanced while the job runs. Real workloads fluctuate, so a large share of that reserved compute sits idle, typically leaving utilization somewhere between 45% and 67%. Idle capacity still costs almost the full amount: power delivery, cooling and capital depreciation do not care whether the chip is doing anything. It also means scaling AI means buying more GPUs rather than using the ones already racked.

Our recommendation was an operating system layer that first sorts each GPU into partitionable or passthrough based on how it is being used, then manages workloads across the fleet to hold utilization in an 80% to 90% band, pulling jobs off saturated GPUs and pushing them onto underused ones while they run.

The numbers were where most of the work went. By our modelling, holding that band lifts per-GPU utilization by roughly 57% on average, which turns into about 16% more revenue from the same hardware, or $2.9B+ a year, and $26M+ a year in electricity, about 1.4% of data centre power draw. For scale, a single Azure AI data centre with 20,000 GPUs is leaving around $84M of potential revenue on the table annually. We grounded all of it in NVIDIA Run:ai, academic work on fine-grained GPU sharing, and published large-cluster utilization studies.

We closed with a rollout rather than an idea: one month of design, three to four months of development with a team of 16 to 20 engineers, three to four months testing against selective GPU clusters, then a three to five month phased rollout starting in low-risk data centres. Ten to fifteen months end to end.