In this document I will explore how to create the first part of the evaluation system I proposed. The working title of this is the “Forecast-Hour Evaluation.” The idea here is that we are looking at the performance of the model by looking at how it performed with different start times (using the most recent 00-hr forecast as input).
## Warning in rgl.init(initValue, onlyNULL): RGL: unable to open X11 display
## Warning: 'rgl.init' failed, running with 'rgl.useNULL = TRUE'.
For this evaluation system we need to look at three different output folders. Here we use the folders named, forecast_day_minus_0
, forecast_day_minus_1
, forecast_day_minus_2
. The contents of each of these folders will be similar: wrfout files for 86 forecast hours and time-series data for different locations of interest. Here we will first read the forecast data.
Now we will read the observation data from the ASOS stations. The script that downloads the data is in ./obs_station_day_minus_0/dl_ny_asos.py
. The lines for the dates to download need to be changed before running it. Once the files are download, the lines below reads the data and adds column names.
Model and observation data do not share the same units for the same variable. For temperature, WRF is in Kelvin and ASOS is in degreesF. For winds, WRF is in m/s and ASOS is in knots. The formulas used to convert the numbers to a common system is shown here. For temperature I will use Kelvin, and m/s for wind speeds.
Now we have one data frame for all the observations, and three (3) data frames of the WRF data (one data frame per forecast init time). The lines below provide a visual of the data frames.
## Date.Time year mon day hour min sec Temperature Mixing.Ratio
## 1 2020-04-26 00:00:05 2020 4 26 0 0 5.0004 283.1914 0.00451
## 2 2020-04-26 00:00:10 2020 4 26 0 0 10.0008 283.2469 0.00451
## 3 2020-04-26 00:00:15 2020 4 26 0 0 15.0012 283.2955 0.00451
## 4 2020-04-26 00:00:20 2020 4 26 0 0 20.0016 283.3401 0.00451
## 5 2020-04-26 00:00:24 2020 4 26 0 0 24.9984 283.3811 0.00451
## 6 2020-04-26 00:00:29 2020 4 26 0 0 29.9988 283.4196 0.00451
## U_WIND V_WIND Wind.Speed Wind.Direction Station
## 1 -1.39266 3.33559 3.614646 157.3387 JFK
## 2 -1.39505 3.24363 3.530906 156.7280 JFK
## 3 -1.40042 3.16880 3.464458 156.1574 JFK
## 4 -1.39328 3.10170 3.400261 155.8104 JFK
## 5 -1.38812 3.04252 3.344220 155.4756 JFK
## 6 -1.38458 2.99194 3.296781 155.1667 JFK
## Date.Time year mon day hour min sec Temperature Mixing.Ratio
## 1 2020-04-25 00:00:05 2020 4 25 0 0 5.0004 282.3018 0.00654
## 2 2020-04-25 00:00:10 2020 4 25 0 0 10.0008 282.3226 0.00654
## 3 2020-04-25 00:00:15 2020 4 25 0 0 15.0012 282.3431 0.00655
## 4 2020-04-25 00:00:20 2020 4 25 0 0 20.0016 282.3626 0.00655
## 5 2020-04-25 00:00:24 2020 4 25 0 0 24.9984 282.3819 0.00655
## 6 2020-04-25 00:00:29 2020 4 25 0 0 29.9988 282.4017 0.00655
## U_WIND V_WIND Wind.Speed Wind.Direction Station
## 1 -0.64592 -2.57242 2.652274 14.09524 JFK
## 2 -0.62324 -2.49231 2.569054 14.03976 JFK
## 3 -0.59594 -2.42148 2.493734 13.82605 JFK
## 4 -0.57400 -2.36096 2.429734 13.66472 JFK
## 5 -0.55576 -2.30793 2.373902 13.53931 JFK
## 6 -0.54057 -2.25693 2.320765 13.46949 JFK
## Date.Time year mon day hour min sec Temperature Mixing.Ratio
## 1 2020-04-24 00:00:05 2020 4 24 0 0 5.0004 280.4555 0.00353
## 2 2020-04-24 00:00:10 2020 4 24 0 0 10.0008 280.5028 0.00354
## 3 2020-04-24 00:00:15 2020 4 24 0 0 15.0012 280.5467 0.00354
## 4 2020-04-24 00:00:20 2020 4 24 0 0 20.0016 280.5885 0.00354
## 5 2020-04-24 00:00:24 2020 4 24 0 0 24.9984 280.6281 0.00355
## 6 2020-04-24 00:00:29 2020 4 24 0 0 29.9988 280.6665 0.00355
## U_WIND V_WIND Wind.Speed Wind.Direction Station
## 1 -1.80493 1.52667 2.363999 130.2257 JFK
## 2 -1.78325 1.50036 2.330464 130.0760 JFK
## 3 -1.77260 1.48220 2.310634 129.9015 JFK
## 4 -1.75523 1.45845 2.282084 129.7238 JFK
## 5 -1.74314 1.43608 2.258509 129.4833 JFK
## 6 -1.72932 1.41137 2.232154 129.2193 JFK
## Station Date.Time Temperature Relative.Humidity Wind.Direction
## 1 JFK 2020-04-25 00:00:00 NaN NaN 10
## 2 JFK 2020-04-25 00:05:00 NaN NaN 30
## 3 JFK 2020-04-25 00:10:00 NaN NaN 20
## 4 JFK 2020-04-25 00:15:00 NaN NaN 20
## 5 JFK 2020-04-25 00:20:00 NaN NaN 30
## 6 JFK 2020-04-25 00:25:00 NaN NaN 40
## Wind.Speed year mon day hour min sec
## 1 2.572016 2020 4 25 0 0 0
## 2 4.115226 2020 4 25 0 5 0
## 3 4.629630 2020 4 25 0 10 0
## 4 4.629630 2020 4 25 0 15 0
## 5 5.144033 2020 4 25 0 20 0
## 6 2.572016 2020 4 25 0 25 0
Time-matching is performed using a routine that can be found in Analysis01-Time_Matching_Problem.Rmd
. The time matching will be done per variable. For the Forecast-Hour Evaluation product, we will focus on the temperature, wind speed and wind direction variables. Also, now that we have read all the TS data and ASOS data, we need to extract the day of interest, or doi
for the time-series.
Note that for this product the “day of interest” will always be the UTC date of the day before.
We now have filtered data frames for the observations and model data for the day of interest.
Next, we will select only the temperature data for comparing the model and observations. This needs to be done on a per station basis. Note that we use the function drop_na()
to drop rows which contain NaN or NA data. Since each variable is measured at different intervals, not all variables will have data available at every time step in the ASOS data. The functions may be too sensitive to missing data and thus we take care to remvove it here from the observations, after we have isolated a particular variable.
For the temperature data I will use Bias, RMSE and MAE for the comparison statistics
Forecast.Init | BIAS | RMSE | MAE | |
1 | WRF D-0 | 0.252 | 0.897 | 0.793 |
2 | WRF D-1 | -0.046 | 0.959 | 0.830 |
3 | WRF D-2 | 0.428 | 0.887 | 0.697 |
Forecast.Init | BIAS | RMSE | MAE | |
1 | WRF D-0 | -1.104 | 1.747 | 1.347 |
2 | WRF D-1 | -1.448 | 1.852 | 1.514 |
3 | WRF D-2 | -1.068 | 1.588 | 1.277 |
Forecast.Init | RMSE | MAE | |
1 | WRF D-0 | 24.619 | 18.602 |
2 | WRF D-1 | 18.787 | 14.249 |
3 | WRF D-2 | 30.770 | 24.700 |
Forecast.Init | BIAS | RMSE | MAE | |
1 | WRF D-0 | 0.032 | 0.442 | 0.342 |
2 | WRF D-1 | 0.036 | 1.528 | 1.228 |
3 | WRF D-2 | 0.668 | 1.299 | 1.008 |
Forecast.Init | BIAS | RMSE | MAE | |
1 | WRF D-0 | -3.963 | 4.676 | 3.973 |
2 | WRF D-1 | -4.037 | 4.602 | 4.043 |
3 | WRF D-2 | -4.002 | 4.631 | 4.009 |
Forecast.Init | RMSE | MAE | |
1 | WRF D-0 | 24.703 | 18.205 |
2 | WRF D-1 | 17.596 | 13.866 |
3 | WRF D-2 | 23.233 | 18.415 |
Forecast.Init | BIAS | RMSE | MAE | |
1 | WRF D-0 | 0.563 | 0.825 | 0.758 |
2 | WRF D-1 | 0.430 | 1.208 | 0.902 |
3 | WRF D-2 | 1.149 | 1.447 | 1.218 |
Forecast.Init | BIAS | RMSE | MAE | |
1 | WRF D-0 | -3.692 | 4.385 | 3.786 |
2 | WRF D-1 | -3.639 | 4.300 | 3.723 |
3 | WRF D-2 | -3.635 | 4.302 | 3.719 |
Forecast.Init | RMSE | MAE | |
1 | WRF D-0 | 43.039 | 25.491 |
2 | WRF D-1 | 45.470 | 28.715 |
3 | WRF D-2 | 39.608 | 25.699 |