Object-based Verification of Atmospheric River Predictions in the Northeast Pacific

July 27, 2021

Laurel DeHaan, an applications programmer at CW3E, recently published a paper (DeHaan et al., 2021) in Weather and Forecasting, along with co-authors from CW3E (Rachel Weihs, Luca Delle Monache, and F. Martin Ralph) and Portland State University (Andrew Martin), titled “Object-based Verification of Atmospheric River Predictions in the Northeast Pacific.” This study contributes to the goals of CW3E’s 2019-2024 Strategic Plan to support Atmospheric River (AR) Research and Applications by advancing the metrics and methods used in AR forecast verification. Verification metrics of AR forecasts are an important component of assessing forecast informed reservoir operations (FIRO) viability and enable other end users to objectively assess the quality of a specific forecast or the performance of a numerical weather prediction model in general.

In this study, a new verification framework is proposed leveraging freely available software and metrics used successfully for other applications. Specifically, AR detection and statistics are computed for the first time using the Method for Object-based Diagnostic Evaluation (MODE). In addition, the measure of effectiveness (MoE) is introduced as a new metric for understanding AR forecast skill in terms of size and location. The MoE provides a quantitative measure of the position of an entire forecasted AR compared to observation, regardless of whether the AR is making landfall. In addition, the MoE can provide qualitative information about the evolution of a forecast by lead time with implications about the predictability of an AR. For example, three different cases illustrate how the MoE can identify errors in AR size and show variability with lead time in Figure 1. This study also analyzes AR forecast verification and skill using 11 years of cold season forecasts from two numerical weather prediction models, one global and one regional. Four different thresholds of integrated vapor transport (IVT) are used in the verification revealing differences in forecast skill based on the strength of an AR (Figure 2). In addition to MoE, AR forecast skill is also addressed in terms of intensity error, landfall position error, and contingency table metrics. These metrics show that the models had more difficulty in predicting the existence, size, and location of higher intensity ARs compared to their ability to predict those characteristics for lower intensity ARs. However, when the models correctly predicted the existence of a higher intensity AR, there was more forecast skill in terms of relative intensity error and landfall position error than lower threshold ARs.

Figure 1: Measure of Effectiveness for forecasts valid Feb 6, 2015 (left), Feb 10, 2016 (middle), and Feb 12, 2017 (right). The Feb 6 case demonstrates somewhat consistent AR forecasts where the area of the forecasted AR is generally too large. The Feb 10 case demonstrates large shifts in the AR forecast by lead time. The Feb 10 case demonstrates 2 different AR forecast regimes: one for the 24–72 hour forecasts, the other for the 96-168 hour forecasts. (Figure 3 from DeHaan et al., 2021).

Figure 2: 2007-2017 average MoE for West-WRF (stars) and GEFS (circles) for thresholds of 250 (top left), 500 (top right), 750 (bottom left) and 1000 (bottom right) kg m^–1 s^-1. Diameter is a function of the standard deviation of the MoE values. Both models produce ARs that are the correct size (on the diagonal) on average in most cases, with a slight bias towards ARs that are too large at the 24-hour lead time. Increasing difficulty in forecasting the correct AR location with higher thresholds can be seen by the dots moving farther away from the upper right-hand corner as the threshold increases. (Figure 7 from DeHaan et al., 2021).

DeHaan, L. L., Martin, A. C., Weihs, R. R., Delle Monache, L., & Ralph, F. M. (2021). Object-based Verification of Atmospheric River Predictions in the Northeast Pacific Weather and Forecasting (published online ahead of print 2021), https://doi.org/10.1175/WAF-D-20-0236.1