Cloud GPUs vs. On-Prem AI Clusters: The 25.7% Utilization Break-Even for an Eight-GPU H100 Node
A team can prove a model on rented accelerators and still be surprised by idle reservations, data movement, or a cluster that cannot be cooled. The decision is not simply “cloud or hardware.” It is a pipeline-placement problem: which stages need elastic capacity, which require predictable memory bandwidth, and how many billable GPU-hours can be kept productively busy.

U.S. East (N. Virginia) Linux on-demand snapshots checked November 16, 2026: AWS p5.48xlarge, eight H100 GPUs, $55.04 per instance-hour; p4d.24xlarge, eight A100 40 GB GPUs, $21.96 per hour. An eight-H100 HGX system is modeled at $300,000, the midpoint of a seller’s November 20 posted $295,000–$320,000 range. An eight-A100 40 GB used system uses a $59,100 indicative fair-market value, not a completed sale. Both are depreciated straight-line over three years with zero residual. Add $15,000 per node-year for facility, support, and network; energy uses 8 kW IT draw for H100 or 6 kW for A100, PUE 1.40, and EIA’s preliminary January–August 2026 U.S. industrial average of $0.0909/kWh. Taxes, egress, financing, extra fabric, and labor beyond that allowance are excluded. Replace each assumption with a quote or meter reading before a purchase.
Quick Decision Matrix
| Workload signal | Cloud starting point | Ownership signal |
|---|---|---|
| Bursty work or under 25.7% annual H100 utilization | Rent by the job and stop idle capacity | Do not buy an H100 node on this model alone |
| Stable work above 25.7% H100 or 21.5% A100 use | Retain on-demand capacity for overflow | A measured, shared cluster may reduce unit cost |
| Residency, latency, or capacity-control requirement | Keep burst pools and disaster recovery elastic | Buy only after power, cooling, fabric, and support are contracted |
Compare completed work, not instance labels
An instance-hour is a capacity unit, not a product. For inference, compare successful, quality-qualified output tokens at a target latency. For training, compare a validated checkpoint or a fixed number of examples processed to a target loss. Record end-to-end throughput on the exact model, precision, batch, sequence length, and software stack. A GPU with twice the hourly price can still be cheaper per accepted token if it finishes the same work more than twice as quickly; a faster accelerator can lose if memory limits force smaller batches or communication dominates.
Use the same accounting boundary on both sides. Cloud cost includes the full instance bill while allocated, attached storage, object-store requests, inter-zone or internet egress, orchestration, and idle reservations. Self-hosted cost includes hardware, warranty, spares, rack and power capacity, electricity, cooling overhead, fabric, storage, deployment labor, and idle capacity. For colocation, use the provider’s committed kilowatts, cross-connects, remote-hands schedule, and minimum term—not a residential electricity rate. Separate one-time procurement from recurring run cost so finance can see cash timing and steady-state economics.
The model is: annual node cost = purchase price ÷ three years + annual facility/support allowance + IT kilowatts × PUE × 8,760 × electricity price. Break-even busy fraction = annual node cost ÷ (comparable cloud rate × 8,760). Cloud capacity is assumed releasable between jobs; the owned node remains provisioned and incurs annual cost. Utilization means productive allocated node-hours divided by all available node-hours. It is a threshold for this configuration, not a universal rule.
Scenario A: bursty research and evaluation
Suppose a team runs prompt sweeps, retrieval evaluations, short fine-tunes, and weekly comparisons. Its equivalent H100 node would be busy only 15% of the year. On the stated snapshot, an on-demand p5 costs $55.04 for each allocated node-hour. Fifteen-percent use is 1,314 hours, about $72,297 a year before storage, network, or orchestration. The modeled owned H100 node costs about $123,918 a year even if the queue is quiet. Cloud is lower on compute at this demand, provided instances are terminated rather than left running.
Profile preprocessing, cleaning, tokenization, and evaluations separately. Put work that does not need H100 memory bandwidth on CPU or lower-cost instances. Keep model build, adapter training, and final validation on an accelerator only for active windows. Enforce time-to-live labels, stop abandoned workspaces, and publish GPU-hours per run. Spot capacity may lower the bill, but interruption handling, checkpoint frequency, and recovery time belong in effective cost. A cheap hour that destroys an eight-hour job is not cheap.
For A100, an indicative $59,100 value for a used eight-GPU 40 GB HGX node, the same three-year life and $15,000 annual allowance, and a six-kilowatt IT draw produce modeled annual cost of about $41,389. Against p4d at $21.96 per node-hour, the threshold is 21.5% utilization. Below it, interruptible rental or a smaller single-GPU pool is easier to justify. The used-system reference is a market estimate, not a seller quote; condition, warranty, included network cards, and storage can move acquisition cost substantially.
Scenario B: steady inference with a traffic floor
Consider production inference with a stable baseline, predictable peaks, and a model that fits on one eight-GPU node. Annualized H100 hardware is $100,000; the allowance is $15,000; modeled power is about $8,918. Total annual cost is roughly $123,918, or $14.15 per calendar node-hour before dividing by productive hours. It reaches parity with $55.04 cloud capacity at 25.7% busy utilization. At 65% busy time, owned cost is $21.76 per active node-hour versus $55.04 for an on-demand p5 hour. With equal throughput, the ratio is about 2.53 times as many output tokens per dollar before omitted costs.
To make the token arithmetic tangible without implying a benchmark, assume a replay measures 8,000 accepted output tokens per second on both deployments. This is an illustrative input, not a result measured for this article. One active hour emits 28.8 million tokens: cloud yields about 523,256 tokens per dollar; the owned node at 65% utilization yields about 1.32 million. At the 25.7% threshold, both are approximately 523,000 under the equal-throughput assumption. Replace 8,000 with the same reproducible workload replay on both platforms. If the cloud configuration is 1.5 times faster for your model, adjust its token numerator before deciding.
Batching, continuous batching, KV-cache pressure, quantization, and time-to-first-token determine whether that assumption holds. A service with strict tail latency may reserve spare replicas; a batch endpoint may consolidate requests and raise utilization. Track output tokens per second and successful tokens alongside p95/p99 latency, queue time, GPU memory, and errors. A device can appear busy while producing little useful work because it is waiting on host preprocessing, remote storage, synchronization, or an oversized context.
Scenario C: distributed training and the facility boundary
Distributed training changes the cost center. Eight GPUs inside one HGX chassis share a high-bandwidth NVLink domain. Beyond one chassis, the job needs a fabric, a topology that avoids oversubscribed links, compatible collective libraries, and storage able to feed every node. AWS p5 offers eight H100s per instance with high-speed Elastic Fabric Adapter networking; p4d offers eight A100 40 GB devices. The $55.04 and $21.96 hourly prices are not equivalent performance offers. P5 buys a newer accelerator and a different memory/interconnect profile. Select it only after validating model fit, time-to-train, and total bill for the actual job.
For the used A100 40 GB node, the same assumptions yield about $41,389 annual cost. The modeled crossover against p4d is 21.5%. At 60% busy time, ownership averages $7.88 per active node-hour versus $21.96 on p4d, or 2.79 times the token-per-dollar ratio if benchmark throughput is equal. That is a condition, not a claim that a used server will match cloud performance or availability. Check its memory configuration, warranty, adapters, storage, and repair path.
The H100 seller snapshot is $295,000–$320,000 for a configured eight-GPU system, with parts and networking confirmed to order. The modeled $300,000 midpoint is not a procurement offer. A server price may omit the multi-node fabric, optics, rack integration, burn-in, spares, or a service-level agreement. The $15,000 annual allowance must be replaced if colocation, remote hands, support, or networking costs more. At PUE 1.40, an 8 kW IT draw becomes 11.2 kW of facility load, before demand charges or site-specific cooling limits. Confirm the rack feed and thermal design with the operator.
Do not move every pipeline stage as one fleet. Keep elastic experiments and burst evaluation in cloud; retain a stable serving or fine-tuning pool only where utilization, residency, or latency justifies it. Preserve a cloud overflow path for seasonal peaks and maintenance. Store checkpoints in portable formats, pin driver and collective-library versions, and test recovery across environments. Apply the same identity, audit, secrets, and dataset controls on both sides; otherwise the cost model excludes the engineering needed to run hybrid infrastructure.
Migration and implementation checkpoints
- Instrument a baseline. For at least four representative weeks, capture allocated GPU-hours, busy time, memory high-water mark, power draw, queue delay, output tokens per second, accepted tokens, retries, and p95/p99 latency by model and stage. Separate maintenance from idle demand.
- Benchmark a fixed contract. Version the prompt mix, input/output lengths, batch, concurrency, quantization, quality check, and latency target. Compare both deployments with the same image and data. Record completion time and total cost, not a peak-performance claim.
- Price the whole path. Include data reads, checkpoints, snapshots, cross-zone traffic, egress, retrieval, load balancers, observability, power demand charges, colocation minimums, fabric ports, warranty, and spare-node policy. Never compare an all-in private cluster to an instance-only cloud line.
- Model the workload mix. Separate burst jobs, steady serving, overnight batches, and recovery capacity. Calculate utilization by pool and month; a high cluster average can hide one saturated model and one idle generation. Use a conservative demand forecast and residual value.
- Design placement and failure modes. Decide which stages may cross regions, how data is encrypted and retained, how checkpoints are validated, what happens when a node or link fails, and whether cloud overflow meets latency and privacy objectives.
- Prove operations before scaling. Test driver upgrades, node drain, canary rollout, power and cooling alarms, spare replacement, and clean checkpoint restore. Assign owners for firmware, scheduling, quota, incident response, and forecasting; these are recurring costs.
- Set explicit buy gates. Start with a reversible cloud pilot. Require utilization above the calculated threshold with margin, a stable forecast, a signed power/facility quote, a validated benchmark, and an approved support plan before purchase. Recompute when rates, hardware, energy, PUE, or throughput changes.
A monthly cohort report should show accepted output tokens, compute and facility spend, dollars per million accepted tokens, and stranded capacity by model. If an H100 node is busy 30% of the month, it may clear this simplified threshold, but volatility, maintenance, and demand growth determine whether the margin survives. A ramp that arrives late can erase a spreadsheet advantage; a committed traffic floor can make ownership rational even when a short pilot looks expensive.
For a workload-shape comparison outside AI, see our Lambda versus ECS Fargate RPS model. It makes the capacity floor explicit rather than treating a list price as a universal answer.
FAQ
Does 25.7% mean an H100 server is always cheaper beyond that point?
No. It is parity for the stated price, three-year depreciation, $15,000 annual allowance, energy inputs, and comparable hours. Financing, labor, taxes, fabric, downtime, spares, and performance can move it. Recalculate from an all-in quote and measured benchmark.
Why are p4d and p5 not apples-to-apples?
p4d has eight A100 40 GB GPUs; p5 has eight H100 GPUs. Their prices compare capacity offers, not equal performance. Test the real model and compare cost per quality-qualified token or completed training step.
Should idle-but-ready capacity count against utilization?
Yes, when it is held for production. Count busy hours against all hours provisioned, including failover replicas and deployment headroom. A research pool that shuts down must still include restart time and any facility charge that continues.
Could a cloud commitment move the break-even?
Yes. Discounts or interruptible capacity lower the rate, while a commitment can add minimum spend. Substitute the actual price and billable hours, then include interruption and commitment risk. These are dated on-demand snapshots, not a regional capacity promise.
What must a colocation quote specify?
Get the rack feed, usable kilowatts, cooling limit, PUE boundary, power and demand charges, network cross-connect, fabric quote, remote-hands coverage, warranty response, delivery, and minimum term in writing. Meter at the wall and replay a representative workload before approval.
Sources and calculation notes
AWS billing rules: EC2 On-Demand Pricing. November 16, 2026 p5 and p4d Linux snapshots, normalized from AWS price data: AW3 Host dated rate table; confirm current region and OS in AWS pricing. H100 system range: Gatewell USA, posted November 20, 2026. A100 used-node estimate: American Compute October 2026 Rack Report, indicative listings-based value, not realized sales. Electricity: EIA Electric Power Monthly, Table 5.6.A. AI data-center efficiency context: LBNL’s 2025 Update. Node-year totals and thresholds are calculations from the assumptions above. Token-rate examples are hypothetical, not benchmark results.
Comments
Post a Comment