The Temperature nvidia-smi Won't Show You
My 3090 sits 3 mm above a 4090. Under sustained load its core read a comfortable 70°C — while the memory junction, invisible to standard tooling, ran 94. Getting that number took reverse-engineering a monitoring app's shared memory.
The riser saga ended with my RTX 3090 mounted ~3 mm above the 4090’s backplate — a spacing that looks like a mistake and needed evidence it wasn’t. So before committing real workloads, I instrumented everything and soaked it: about four and a half hours of sustained inference inside a six-hour instrumented window.
The catch: on GDDR6X cards, the sensor that matters most is the one standard tooling
hides. nvidia-smi reports the 3090’s memory-junction temperature as N/A — while that
junction throttles at ~110°C and, on a card breathing through a 3 mm gap, is guaranteed
to be the first thing cooking.
Getting the hidden number
HWiNFO can see the junction sensor, but I needed it in scripts, logs, and alerting — not
a GUI. HWiNFO publishes all sensors to a shared-memory block, whose layout I ended up
mapping empirically because my copy didn’t match the historical SDK structures: dump the
header, find the section offsets (readings at +8280, 460 bytes each — not the documented
316), then locate fields inside an element by scanning for printable runs. Labels at +12
and +140, unit at +268, value as a double at +284, with a UTF-8 duplicate section
explaining the size growth. Forty lines of PowerShell later, every 3090 sensor streams
into the same CSV as nvidia-smi’s.
Worth knowing before you try this at home: the free tier disables shared memory 12 hours after each launch (your monitoring dies on a schedule), and Ada cards report thermals in a different dialect than Ampere — the 4090 exposes “T.Limit” headroom values where the 3090 gives absolute temperatures, so a naive cross-card dashboard compares incommensurable numbers.
The verdict, and the doctrine
The sandwich passes: at a 300 W cap, the sustained hours held the junction at 90–94° — a real but stable 16° margin — with core at 66–70, fans at ~80% with headroom, zero throttle events, and a flat equilibrium after the first forty minutes. I’d re-check it in summer and after any power-cap change, and the alerting now watches junction thresholds (96 warn, 102 critical), not core.
The general doctrine costs nothing and would have saved me an earlier incident (a 450 W-era junction spike that almost certainly took the whole machine offline mid-run): find the sensor that throttles first on your specific silicon, and alert on that one. Core temperature on a GDDR6X card is a lagging, flattering indicator — my “70° card” was a 94° card all along, and only one of those numbers gets a vote at 110.