Iman Soltanzadeh

·Iman Soltanzadeh

Pangu-Weather in Nature: an AI model that beats ECMWF's operational forecast on reanalysis

Pangu-Weather beat ECMWF's operational forecast on every variable tested against reanalysis, over 10,000 times faster. The Nature paper, with its caveats.

AI weather models · Machine learning · Meteorology · Research notes

Paper card for Bi et al., Nature 2023, with a bar chart of five-day 500 hPa geopotential error: 296.7 for Pangu-Weather against 333.7 for ECMWF's operational IFS.

TL;DR

  • Pangu-Weather, a deep-learning model from Huawei Cloud, produced more accurate global forecasts up to seven days ahead than ECMWF’s operational system for every variable tested, verified against reanalysis for 2018.
  • A 24-hour global forecast takes 1.4 seconds on a single GPU; the authors put the model at more than 10,000 times faster than the operational system.
  • The authors are clear about the limits: the model was trained and tested on reanalysis, whereas real forecast systems work from observational data; it does not forecast precipitation; and its smooth forecasts may underestimate extremes.

What was published

On 5 July 2023, Nature published Pangu-Weather by Kaifeng Bi and colleagues at Huawei Cloud. The authors compare it with the Integrated Forecasting System (IFS) of the European Centre for Medium-Range Weather Forecasts (ECMWF), which they describe as the world’s best numerical weather prediction system. They cite a 2021 review that saw great potential in AI but judged that fundamental breakthroughs were still needed before it could beat numerical prediction, and suggest that those breakthroughs seem to be arriving earlier than expected.

How the model works

Pangu-Weather learns how the atmosphere evolves from ERA5, ECMWF’s reanalysis of past weather, at 0.25° resolution. It was trained on 39 years (1979–2017), validated on 2019 and tested on 2018. It forecasts five upper-air variables on 13 pressure levels and four surface variables, including 2 m temperature and 10 m wind.

Two ideas carry the paper:

  • The atmosphere as one volume. Pressure levels are treated as a third dimension, and a transformer network with Earth-specific position information learns how the levels interact. Earlier models such as FourCastNet treated the data in two dimensions.
  • Fewer steps. Four networks forecast 1, 3, 6 and 24 hours ahead, and a forecast always uses the longest step available. Fewer steps leave less room for errors to accumulate.

Each network has about 64 million parameters and took around 16 days to train on 192 GPUs.

The results

Verified against ERA5 for 2018:

  • Lower error on every variable tested. Errors (RMSE) were typically about 10 % lower than those of the operational IFS, and the advantage held at all lead times from 1 hour to 7 days.
  • Time gained. Pangu-Weather typically reached the same accuracy as the IFS 10 to 15 hours further ahead, and more than 24 hours further ahead for specific humidity.
  • Tropical cyclones. For 88 named cyclones in 2018, track errors were smaller than those of ECMWF’s high-resolution forecast (chart below). The authors call this comparison somewhat unfair, because ECMWF’s forecast started from its own operational initial conditions and Pangu-Weather from reanalysis.
  • Ensembles. With a preliminary perturbation method, the mean of a 100-member ensemble beat the single forecast at five to seven days but was slightly worse at one day, and its spread was somewhat too narrow.

Slope chart of mean tropical cyclone track error for 88 named cyclones in 2018, from three to five days ahead: Pangu-Weather rises from 120 to 196 km, ECMWF’s high-resolution forecast from 162 to 272 km.

Mean position error of cyclone centres, 88 named tropical cyclones in 2018. Values from Bi et al. (2023); chart drawn for this post.

The authors’ caveats

  • Training and testing used reanalysis; real forecasting systems start from observations, so performance in operations needs further study.
  • Precipitation was not included.
  • Forecasts are smoother than the real atmosphere, which risks underestimating the strength of extreme events. In the authors’ tests both Pangu-Weather and the IFS underestimated extremes, Pangu-Weather increasingly so with lead time.
  • Combining models with different step lengths can make forecasts inconsistent in time.

My view

The speed is what stands out to me for energy forecasting. The authors note that computing cost limits the number of ensemble members in numerical prediction, and with it the accuracy of probabilistic forecasts; a model this fast eases that limit, which I see as the most promising point for wind and solar. The outputs include 10 m wind and 2 m temperature but not hub-height wind or solar radiation, so energy applications would still need further work. The authors expect AI and numerical methods to be combined in future, and I share that expectation.

References

  • Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X. and Tian, Q. (2023). Accurate medium-range global weather forecasting with 3D neural networks. Nature, published online 5 July 2023. doi:10.1038/s41586-023-06185-3. Open access under CC BY 4.0.
  • Trained models and inference code: github.com/198808xc/Pangu-Weather (doi:10.5281/zenodo.7678849), as stated in the paper.
  • Schultz, M. G. et al. (2021). Can deep learning beat numerical weather prediction? Philosophical Transactions of the Royal Society A, 379, 20200097. doi:10.1098/rsta.2020.0097.
  • Pathak, J. et al. (2022). FourCastNet: a global data-driven high-resolution weather model using adaptive Fourier neural operators. Preprint, arXiv:2202.11214.
  • Hersbach, H. et al. (2020). The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146, 1999–2049.

Charts are drawn for this post from numbers stated in the paper’s text; they are not reproductions of the paper’s figures.

The views in this article are my own and do not represent those of my employer or of any organisation I am associated with.