Why model outputs NaN for all surface channels during inference?
shape: (85, 721, 1440), latlon: (90.00~-90.00) x (0.00~359.75)
z50: 179490.2344 ~ 205233.6719
z100: 142126.3906 ~ 164670.5312
z150: 119570.5781 ~ 141401.7031
z200: 102894.1875 ~ 123601.8828
z250: 89555.5625 ~ 108837.4531
z300: 78473.3750 ~ 96189.2500
z400: 60391.3867 ~ 75184.0547
z500: 45823.1562 ~ 58143.5703
z600: 33477.7305 ~ 43983.3320
z700: 22609.5586 ~ 31959.0059
z850: 8538.8535 ~ 16424.0020
z925: 2222.9902 ~ 9518.2061
z1000: -3867.4587 ~ 3131.0154
t50: 185.7772 ~ 229.3458
t100: 186.8497 ~ 231.4670
t150: 191.4461 ~ 232.9318
t200: 192.2884 ~ 235.9922
t250: 197.8167 ~ 240.6715
t300: 203.8592 ~ 250.1586
t400: 215.1154 ~ 264.2981
t500: 222.8104 ~ 275.3831
t600: 211.3748 ~ 285.3513
t700: 219.9240 ~ 291.4831
t850: 228.2802 ~ 304.5688
t925: 232.0518 ~ 310.2230
t1000: 235.5404 ~ 315.3259
u50: -22.7820 ~ 77.4903
u100: -36.7628 ~ 65.7464
u150: -33.9436 ~ 66.9999
u200: -36.8218 ~ 81.5968
u250: -44.3387 ~ 90.9427
u300: -46.0766 ~ 85.5679
u400: -43.5785 ~ 68.9162
u500: -35.5019 ~ 62.6256
u600: -29.6463 ~ 52.8383
u700: -29.8201 ~ 42.5945
u850: -32.4083 ~ 39.4811
u925: -35.1361 ~ 35.3792
u1000: -21.6389 ~ 22.3071
v50: -30.0476 ~ 48.4844
v100: -31.0799 ~ 43.1312
v150: -36.8724 ~ 43.3916
v200: -60.6146 ~ 58.6688
v250: -58.7458 ~ 60.7002
v300: -67.0607 ~ 61.1306
v400: -61.1582 ~ 54.6420
v500: -54.5159 ~ 40.7864
v600: -46.4449 ~ 34.4637
v700: -35.2874 ~ 33.8496
v850: -40.1182 ~ 36.1816
v925: -38.3045 ~ 38.9277
v1000: -21.7216 ~ 23.6245
q50: 0.0007 ~ 0.0033
q100: 0.0006 ~ 0.0063
q150: -0.0008 ~ 0.0393
q200: -0.0064 ~ 0.2281
q250: -0.0416 ~ 0.7505
q300: -0.0832 ~ 1.6893
q400: -0.2897 ~ 4.3518
q500: -0.3494 ~ 6.9992
q600: -1.1464 ~ 10.9991
q700: -1.3624 ~ 13.9666
q850: -0.9039 ~ 18.5888
q925: -1.0911 ~ 20.6932
q1000: -0.8573 ~ 24.6604
msl: nan ~ nan
t2m: nan ~ nan
d2m: nan ~ nan
sst: nan ~ nan
ws10m: nan ~ nan
ws100m: nan ~ nan
u10m: nan ~ nan
v10m: nan ~ nan
u100m: nan ~ nan
v100m: nan ~ nan
lcc: nan ~ nan
mcc: nan ~ nan
hcc: nan ~ nan
tcc: nan ~ nan
ssr: nan ~ nan
ssrd: nan ~ nan
fdir: nan ~ nan
ttr: nan ~ nan
tcw: nan ~ nan
tp: nan ~ nan
I have print the mean vertical profile of q (specific humidity) from 50 hPa down to 1000 hPa using your official 'mean.nc'. The values I obtained are:
[2.74e-3, 2.74e-3, 5.53e-3, 2.03e-2, 6.02e-2, 1.33e-1, 4.08e-1, 8.92e-1, 1.58e+0, 2.49e+0, 4.65e+0, 6.09e+0, 7.08e+0] (unit as per your documentation: kg·kg⁻¹).
If the unit is truly kg·kg⁻¹, the near‑surface value of ~7 g/kg would actually be 7.08 kg/kg, which is thermodynamically impossible (exceeding the saturation limit). The numbers appear to be in g/kg instead. Could you please confirm whether the output is in g/kg or if a scaling factor is applied?
Sorry for the unit confusion. You're correct — q is in g/kg, not kg/kg. ERA5 specific humidity was scaled ×1000 before normalization during data preparation. We've updated the README and variables.py to fix the unit documentation. Thanks for catching this!
Re: NaN for surface channels
I'm unable to reproduce this issue. I've uploaded infer_log.txt from an inference run on an H100 with PyTorch 2.11.0 / CUDA 12.8 — all 40 steps completed with valid values across all 85 channels, no NaN. Since the model was exported on this same machine/environment, the issue may be environment-related. Could you share your PyTorch version and GPU setup?
Re: Input/output space
To clarify:
input.nc: pre-normalized (z-score normalized space)- Output
.ncfiles: denormalized to physical units viaoutput = output * std + mean tp: additionally reversed from log1p space viaexpm1, clipped to ≥ 0