H100 vs H200: LLM Benchmarks, Rental Costs, and Which to Choose

We served the same Qwen3-32B model in BF16 with the same vLLM build on one H100 SXM 80 GB and one H200 SXM 141 GB, and measured both. Decode ran 1.2 to 1.3 times faster on the H200 at low concurrency, but not enough to offset its 48% higher hourly rate: for short prompts and single-user or small-batch serving the H100 delivered more output tokens per dollar. The picture flipped once the KV cache filled. With 8,192-token prompts at eight concurrent requests the H100 could hold only two or three sequences at a time and queued the rest, so the H200 produced 1.65 to 2 times the throughput and cost 10 to 27% less per token, depending on how much cache the H100 server had left. At 16,384-token prompts the H200 was 2.2 times faster and 31% cheaper per token, and the H100 could not start the server at all for 32,768-token prompts. Rent the H100 for models that leave real cache headroom in 80 GB; rent the H200 when long contexts or concurrency push the KV cache past what 80 GB leaves free.

Quick recommendation

These numbers come from one model at one precision. Quantizing the model or shrinking the context moves the boundary toward the H100; larger models or longer contexts move it toward the H200.

What we measured (September 14, 2026)

Both GPUs were rented on Jarvislabs as dedicated single-GPU VMs in the same region (IN2, Noida), with no other GPU workloads. The load generator ran on the same VM, so the numbers describe the deployment on that machine rather than network latency.

Configuration differences: the H200 VM plan has 28 vCPUs and 295 GB RAM on an Intel Xeon Platinum 8580 host, against 24 vCPUs and 196 GB on an Intel Xeon Platinum 8480C host for the H100 VM. The load generator never approached either limit, but the load generator and server share the host, so the CPU difference is a confound we could not remove. A background telemetry poller (nvidia-smi plus a metrics scrape once per second) ran on both machines for the whole experiment, and from the second cell onward two such pollers ran instead of one on both machines; this was identical on the two GPUs.

At the 0.90 memory setting the model weights took 61 GiB on both GPUs. vLLM reported 5.6 to 7.2 GiB of KV cache on the H100 (23,040 to 29,344 tokens across restarts) and 60 to 62 GiB on the H200 (246,512 to 252,816 tokens). That ten-fold difference in cache capacity, not compute, drives every result below.

Speed and latency: same model on both GPUs

Four workload shapes, 32 requests per repetition. TTFT is time to first token, TPOT is time per output token, E2E is end-to-end request latency. p50 values are medians across the three repetitions.

Three patterns stand out.

Decode speed scaled by roughly 1.3× on the H200 whenever both GPUs held every request in memory: 24.9 ms versus 18.6 ms per output token at concurrency 1, and 302 versus 401 tokens per second at concurrency 8. Time to first token on the 8,192-token prompt was the same on both (about 0.94 s). We did not profile the kernels, so we describe this as the observed pattern for this model rather than a claim about which stage is bandwidth-bound.

The H100 ran out of cache at 8,192 tokens × 8 requests. Only three sequences were resident at once; the other five waited in the scheduler queue, which is why median time to first token jumped to 12.5 s while the H200 kept all eight resident at 3.7 s. Nothing failed, no preemptions were logged, and every request completed. The cost was throughput: 60 versus 100 tokens per second.

Short-prompt batching worked identically on both. At 512 × 512 and concurrency 8 both GPUs kept all eight sequences resident and the H200's edge was the same 1.3× seen at concurrency 1.

Throughput varied by less than 1% across repetitions in every cell. Median latencies varied by less than 3%, with one exception: time to first token in the H200 512 × 512 concurrency-8 cell ranged from 335 to 409 ms across the three runs. With 32 requests per repetition, p95 values are descriptive; they are not a production SLA.

Long context: where the H200's memory changes the answer

Same model and precision, 128 output tokens, 16 requests per probe, one server restart per context length with max_model_len set to input + 256 on both GPUs. "Resident" is the peak number of sequences the scheduler held in the KV cache at once; requests above that number queued rather than failing.

Times are median end-to-end latency per request. Concurrency 8 cells were repeated three times; the other probes ran once, so treat their numbers as indicative. The H100's 8,192 × 8 row shows 49 tokens per second here against 60 in the speed table above because this server (max_model_len 8,448) was left 5.6 GiB of cache by vLLM's memory profiling against 7.2 GiB for the speed-test server (max_model_len 9,216), so it held two sequences at a time instead of three.

The H100's throughput flattened at exactly the point where resident sequences stopped growing: 49 tokens per second for every 8k probe from concurrency 2 upward, 24 tokens per second for every 16k probe. Added concurrency only lengthened the queue. The H200 kept scaling until 32,768 × 8, where it held seven of eight sequences with the cache 93% full; that is the first cache boundary we observed on the H200 for this model.

The H100 startup failure at 32,768 tokens was a configuration limit, not a crash under load. vLLM refused to start because serving even one 33,024-token request needs 8.06 GiB of KV cache and 5.63 GiB was available. Raising --gpu-memory-utilization, quantizing the KV cache or the weights, or running two H100s with tensor parallelism would each change that outcome, and each is a different experiment.

Cost: is the H200 premium worth paying?

GPU cost for a run is the hourly rate times the measured wall time of that run; cost per million output tokens divides that by the tokens actually generated. Rates are the Jarvislabs on-demand prices on September 14, 2026: $2.69 per hour for the H100 and $3.99 per hour for the H200, storage extra. At those rates the H200 has to deliver 1.48× the throughput to match the H100's cost per token. This table shows where it did and where it did not.

"Speed" rows use the speed-test server (32 requests per run, three runs, H100 cache 29,344 tokens). "Long context" rows use the long-context servers (16 requests per run, one run except at concurrency 8, H100 cache 23,040 tokens). The two 8,192 × 8 rows are the same workload on those two server configurations; the smaller cache is why the H100 drops from 60 to 49 tokens per second and the H200's advantage grows from 10% to 27%. "Cheaper by" is the saving relative to the more expensive GPU's cost.

Per thousand completed requests the same pattern holds: $1.27 (H100) versus $1.42 (H200) for the short-prompt batch, $1.58 versus $1.42 for 8k prompts at concurrency 8, and $4.04 versus $2.77 for 16k prompts at concurrency 8.

These are GPU-only costs of a benchmark that kept the GPU saturated. They are not what a lightly used endpoint costs per token, and they exclude storage, model download, server startup and idle time. The whole experiment, including several setup iterations, billed roughly $12 of GPU time across both instances during the two hours of active work. The two VMs were then left running idle for about nine and a half hours before being paused, which added about $62 and brought the total bill to $73.87. That idle spend is a lesson about pausing instances, not a cost of the benchmark.

Read cost next to latency. In the 512 × 512 single-request cell the H100 is 10% cheaper per token, but each response takes 12.8 s instead of 9.6 s. If your product needs the faster response, the H100's saving is not available to you.

When the H100 is the better rental

• The model and its cache fit with headroom. A 32B BF16 model left only 5 to 7 GiB of cache on the H100, and that was the constraint in every cell where the H100 fell behind. A 7B to 14B BF16 model, or a quantized 32B or 70B, leaves far more room and would move the boundary out.
• Short prompts, or long prompts at concurrency 1 to 2. The H100 was 10 to 20% cheaper per output token in every cell where both GPUs kept all requests resident.
• Throughput per dollar matters more than latency. In those same cells the H200 was faster but never 1.48× faster.
• Compute-heavy training. Both GPUs have the same Tensor Core specification. Our experiments measured inference only and say nothing about training or fine-tuning speed.

When the H200 is worth the premium

• Long prompts at concurrency. From 8k tokens × 4 requests and 16k tokens × 2 requests upward the H200 was 1.6 to 2.2× faster and 7 to 31% cheaper per token, because the H100 had already run out of cache.
• 32k contexts with a 32B BF16 model. The H100 could not start the server at this configuration; the H200 served eight concurrent 32k requests with the cache 93% full.
• Latency targets. The H200 cut end-to-end latency by 15 to 54% in every matched cell. Per-token decode time was 25% lower when both GPUs held every request. In the cache-bound cells the H200's time per output token was higher, not lower, because it was decoding two to four times as many sequences at once; each request still finished sooner because it never waited in the queue.
• Avoiding tensor parallelism. One H200 held what would otherwise need two H100s for these long-context cells. We did not run the two-H100 comparison; total cost of two H100s ($5.38 per hour) is above one H200 ($3.99 per hour), but the throughput of that pair is unmeasured here.

What this benchmark does not show

• Answer quality. Prompts were random tokens with forced output lengths, sent to the raw completions endpoint. This measures hardware and serving behaviour, not chat or reasoning quality, and nothing here reflects Qwen3's thinking mode.
• Production steady state. Each cell sent a fixed batch of 16 or 32 requests. Client concurrency caps in-flight requests; it is not the number of sequences the GPU held, which is why we report resident sequences separately.
• Other precisions and models. A quantized KV cache would roughly double the tokens the same cache holds. FP8 weights would free about 30 GiB on the H100, several times its current cache. Either shifts every boundary above. A 70B model in BF16 does not fit either GPU alone and is a different study.
• The cause at the hardware level. We observed a consistent 1.2 to 1.3× decode speedup on the H200 and identical prefill latency, but we did not profile kernels, so we do not claim which stages are bandwidth-bound.

H100 vs H200: Specs Comparison

These specs are for SXM variants. H100 NVL (94GB, 3.9 TB/s, 350-400W) and H200 NVL (141GB, 4.8 TB/s, up to 600W) have different power profiles but the same memory/bandwidth characteristics for H200.

The H200 provides 76% more memory and 43% higher bandwidth than H100 SXM while sharing the same Hopper architecture and Tensor Core configuration. In our measurements the usable difference was larger than the headline 76%: because the 32B model's weights consume the same 61 GiB on both, the H200 had roughly ten times the H100's free memory for KV cache.

NVIDIA H100 Overview

NVIDIA announced the H100 at GTC 2022 as the flagship Hopper architecture GPU. It represented a significant jump from the A100 in both compute and memory bandwidth.

The H100 SXM delivers 3.35 TB/s of HBM3 bandwidth. Compared to the A100 family, that's about 2.1x the bandwidth of A100 40GB (1,555 GB/s) and about 1.6x the bandwidth of A100 80GB (2,039 GB/s).

Key H100 features:

• Fourth-generation Tensor Cores
• Transformer Engine that dynamically manages FP8/FP16 precision
• Native FP8 support for efficient inference
• 80 billion transistors on TSMC 4N process

NVIDIA reports up to 4x faster GPT-3 (175B) training versus A100 in its published benchmarks. That is a manufacturer figure for training; our measurements above cover inference only.

NVIDIA H200 Overview

NVIDIA announced the H200 at Supercomputing 2023. It's not a new architecture but an enhanced H100 with significantly more memory. The H200 uses HBM3e instead of HBM3, pushing capacity to 141GB and bandwidth to 4.8 TB/s.

Key H200 characteristics:

• Same Hopper architecture as H100
• Same Tensor Core configuration and compute specs
• 141GB HBM3e (76% more than H100)
• 4.8 TB/s bandwidth (43% higher than H100)
• Available in SXM and NVL form factors

NVIDIA's own benchmarks report about 1.9x faster inference on Llama 2 70B compared to H100. That is a manufacturer-reported result for a larger model than we tested. Our measured Qwen3-32B speedups ranged from 1.2× when both GPUs had cache headroom to 2.2× when the H100 was cache-bound.

Power and Cooling

NVIDIA lists the same max TDP (up to 700W) for both H100 SXM and H200 SXM, and both of our rented VMs reported a 700 W power limit.

The NVL variants differ: H100 NVL runs at 350-400W while H200 NVL goes up to 600W. If you're planning self-hosted deployments, verify the specific SKU's power requirements.

Pricing and Availability

Jarvislabs on-demand rates on September 14, 2026 were $2.69 per GPU-hour for the H100 and $3.99 per GPU-hour for the H200, storage extra. Rates change as availability shifts, so check those pages before deciding. Both GPUs are available as templates and as GPU VMs.

For teams evaluating both, the safest path is still to start on the H100 and measure your own cache headroom with your model, context length and concurrency. The software stack is identical, so moving to the H200 is a configuration change, not a migration.

Reproducing these results

Everything behind the tables above is public in the Jarvislabs benchmarks repository: the pinned environment, the server and client scripts, the raw vLLM benchmark JSON for every cell, per-second GPU telemetry, server logs, the spend log and the analysis script. The scripts create no account-specific state. Point them at any two Hopper VMs, run scripts/run_all.sh h100 and scripts/run_all.sh h200, then scripts/analyze.py regenerates every table in this article from the raw files. If a number here disagrees with what the script produces, open an issue there.

What Comes After H200?

NVIDIA's Blackwell architecture (B100, B200, GB200) is the next generation after Hopper. Blackwell brings architectural improvements beyond memory upgrades.

H100 and H200 will remain solid choices for years given their mature software ecosystem and broad deployment base. The Hopper architecture is well-optimized across frameworks and continues to receive driver and library updates.

FAQ

What's the main difference between H100 and H200?

Memory. H200 has 141GB HBM3e versus H100's 80GB HBM3, with 43% higher bandwidth (4.8 TB/s vs 3.35 TB/s). Both share the same Hopper architecture and identical compute specs. In our Qwen3-32B test the extra memory translated into ten times the KV cache capacity, because the model weights took the same 61 GiB on both.

Is H200 faster than H100?

Yes, in every inference cell we measured, by 1.2 to 1.3× when both GPUs held all requests in memory and by up to 2.2× when the H100 ran out of KV cache. NVIDIA reports about 1.9× on Llama 2 70B. The H200 is not a new architecture; the speedup comes from memory bandwidth and, at higher load, from memory capacity.

Can I run the same models on both GPUs?

Yes. Both support the same CUDA ecosystem, frameworks, and libraries. We used the identical vLLM build and flags on both. The one exception is capacity: a configuration that needs more KV cache than the H100 has free, like our 32k-context Qwen3-32B server, will refuse to start on the H100 and start normally on the H200.

Which GPU is better for LLM inference?

Depends on how much of the 80 GB your model leaves free. For our 32B BF16 model the H100 was 10 to 20% cheaper per output token with short prompts or low concurrency, and the H200 was 7 to 31% cheaper per token once prompts reached 8k tokens at concurrency 4 or 16k tokens at concurrency 2. Quantized or smaller models shift that boundary toward the H100; larger models and longer contexts shift it toward the H200.

Does fitting the model in 80 GB mean the H100 will perform the same?

No. Qwen3-32B fit on both GPUs with room to spare for the weights, yet the H100 was slower in every cell and fell behind by 2× once concurrent long prompts filled its cache. Model fit, memory bandwidth, cache capacity and concurrent serving are separate constraints. Fit tells you the server will start; cache headroom tells you how it will scale.

Does H200 use more power than H100?

No, for SXM variants. Both H100 SXM and H200 SXM have the same 700W power envelope, and both of our rented VMs reported a 700 W limit. NVL variants differ (H100 NVL: 350-400W, H200 NVL: up to 600W).

Should I start on H200 or use H100 now?

Start on the H100 if your model leaves real headroom in 80 GB and your prompts are short or your concurrency is low; it was cheaper per token in those cells. Move to the H200 when you serve long contexts at concurrency, need lower latency, or your server cannot start on 80 GB. Both are available on demand, so measuring your own workload on each for an hour costs less than guessing.