Q&A about Precipitation Verification at NCEP/EMC

Last update: 24 Apr 2008


What is VSDB? What is FVS?
Where can I find documentations about VSDB and FVS?
What do the numbers in the verification files mean?
What scores can be calculated from the "F,H,O" numbers?
An example of score calculation
What are bias-adjusted scores?
What do the subregions look like?
How do I plot FVS stats for a subregion of ConUS?
Where can I find the raw daily gauge data used in the old verification?
What is the verifying analysis?
What is the difference between the "old" and the "new" verification systems?
Where can I find the VSDB files generated at NCEP?
How do I generate my own verification statistics?
How are the scores computed for multiple subregions/time periods?
Why doesn't the monthly/month-to-date scores page include more score plots?
Info on precip SL1L2 scores?
Does smoothing a model's QPF fields give it a better ETS score?


What is VSDB? What is FVS?

There are two components to the NCEP/EMC precipitation verification system. The first is the generation of the precipitation verification statistics by comparing model precipitation forecasts against verifying analyses for the same time periods (e.g. 12Z $daym1 to 12Z $day, for 24h verifications, or 06-09Z, for 3-hourly verifications), for various verifying regions. These numbers are stored in ascii files with the format of model_yyyymmdd.vsdb. 'vsdb' stands for 'verifications database', further information about it can be found in the on-line paper, EMC Verification Database and the NCEP Verification System User Guide.

The second component is to compute and display various precipitation scores from the vsdb files. This is handled by the Forecast Verification System (FVS), which pulls relevant data from the numbers in vsdb files, computes scores requested by the user, and (if requested) displays the scores using a GEMPAK-based user interface. Further information can be found in FVS - Forecast Verification System.

The VSDB and FVS packages are used by both the precipitation verification and non-precipitation near-surface/upperair verification systems.

You can compute the scores from the vsdb files yourself, without using the FVS.

Return to the Q&A index


Where can I find documentations about VSDB and FVS?

What do the numbers in the verification files mean?

There is one QPF verification statistics for each model, each day. The file is in the format of model_yyyymmdd.vsdb. The suffix 'vsdb' stands for 'verification database' (for more information, see this eta_20021002.vsdb contains verification statistics for Eta precipitation forecasts verified for the 24h period ending at 12Z 2 Oct 2002 under the new system. Model forecast verified for this 24h period include the following:

    Forecast length     model run cycle
       00-24h            2002100112
       06-30h            2002100106
       12-36h            2002100100
       18-42h            2002093018
       24-48h            2002093012
       30-54h            2002093006
       36-60h            2002093000
       42-66h            2002092918
       48-72h            2002092912
       54-78h            2002092906
       60-84h            2002092900
The file contains lines such as

    V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>.01 APCP/24 SFC =  1362  .35242  .27827  .44053
    V01 ETA 24 2002100212 MC_PCP G211/NWC FHO>.01 APCP/24 SFC =    60  .00000  .00000  .05000
    V01 ETA 24 2002100212 MC_PCP G211/SWC FHO>.01 APCP/24 SFC =    62  .08065  .06452  .20968
Meaning of each column (use the first line above as an example):
  1. V01 Version 1 of the NCEP FVS (Forecast Verification System)
  2. ETA Model name
  3. 24 Model forecast hour. Other forecast hours in this file are 36, 48, 60, etc.
  4. 2002100212Valid time of verification. When forecast hour is 24 and precipitation accumulation period is also 24h (see col. 8 below) this means we are verifying the 24h forecast from the 2002100112 model cycle against the observed precipitation accumulation ending at 2002100212.
  5. MC_PCP This identifier tells the FVS search program that this is the 'new precipitation verification statistics'. The identifier for the 'old precipitation verification statistics' is MB_PCP.
  6. G211/RFC Grid/region for the verification. G211 is the 80km Lambert conformal grid. 'RFC' means the verification region is the ConUS.
  7. FHO>.01 The last three numbers in this line are the number of grid points that has (normalized by total number of valid analysis points in the domain) Forecast, 'Hit' (i.e. both forecast and observed) and Observed precipitation greater than 0.01"/day. The precipitation thresholds are 0.01", 0.1", 0.25", 0.5", 0.75", 1.0", 1.5", 2.0", 3.0"/day.
  8. APCP/24 Precipitation accumulation period is 24h.
  9. SFC = Verification is done at the surface. This doesn't mean much in precipitation verifcation. It's included here because this is part of the NCEP FVS which includes upper-air verification of other forecast fields.
  10. 1362 The number of valid verification grid points in the verification domain. This is the number of grid points in the verifying analysis (i.e. the 'observation') that have valid values (i.e. non-missing).
  11. .35242 This is the F/T, the fraction of grid points that contain forecast precipitation exceeding the threshold given in col. 6. In this case, there are 480 points (0.35242*1362) that has 24h forecast precipitation exceeding 0.01"/day.
  12. .27827 This is the H/T, the fraction of grid points that have both forecast and observed precipitation exceeding the threshold given in col. 6. In this case, there are 379 points (0.27827*1362) that has 24h forecast and observed precipitation exceeding 0.01"/day.
  13. .44053 This is the O/T, the fraction of grid points that have observed precipitation exceeding the threshold given in col. 6. In this case, there are 600 points (0.44053*1362) that has 24h observed precipitation exceeding 0.01"/day.

Return to the Q&A index


What scores can be calculated from the "F,H,O" numbers?

Thanks to the efforts of Keith Brill (NCEP/HPC), 31 scores (28 unique ones) can be calculated from the "F,H,O" statistics contained in the statistics files using the FVS (Forecast Verification System). Here is a one-day example of 0-24h precipitation forecasts from 3 models:

24h Precipitation Accumulation (mm) Ending 12Z 2 Oct 2002
AWIP 211 Grid (80km Lambert Conformal)

A list of some common scores that can be computed from the "F,H,O" statistics are described in the table below. The symbols are defined as:

T: Total number of grid points to be verified
F: Number of forecast points above a threshold
O: Number of observed points above a threshold
H: Number of correctly forecasted points ('hits')

Note that this is not a comprehensive list of all scores that might be computed from F/H/O statistics. For further information, please consult http://www.emc.ncep.noaa.gov/mmb/papers/index.html (look for papers with "FVS" or "verification" in the title).

Click on the number in the first column for a plot of the score:
Code Type of Score Definition* Name in FVS Graph
1001 Bias F/O BIAS_SCORE
1002 Equitable Threat (H-CH)/(F+O-H-CH), where CH=F*O/T EQ_THT_SCORE
1003 Critical Success Index Same as 1008 CSI=THRTSC
1004 False Alarm Ratio 1.0-H/F FLS_ALRM_RATIO
1005 Postagreement H/F POSTAGREMT
1006 Prefigurance Same as 1007 PREFIG=POD
1007 Probability of Detection H/O PROB_OF_DET
1008 Threat Score H/(O+F-H) CSI=THRTSC
1009 Cumulative Forecast Prcp Vol % vs Thrshld FOR_CMLTV_%TOTVL
1010 Cumulative Correctly Frcst Prcp Vol % vs Thrshld VRF_CMLTV_%TOTVL
1011 Cumulative Observed Prcp Vol % vs Thrshld OBS_CMLTV_%TOTVL
1012 Forecast Prcp Vol % of Ttl vs Thrshld FOR_%TOTAL_VOL
1013 Correctly Forecast Prcp Vol % of Ttl vs Thrshld VRF_%TOTAL_VOL
1014 Observed Prcp Vol % of Ttl vs Thrshld OBS_%TOTAL_VOL
1015 Forecast Prcp # % of Ttl # vs Thrshld FOR_%TOTAL_CNT
1016 Correctly Forecast Prcp # % of Ttl # vs Thrshld VRF_%TOTAL_CNT
1017 Observed Prcp # % of Ttl # vs Thrshld OBS_%TOTAL_CNT
1018 Forecast # % of Ttl <= Thrshld vs Thrshld FOR_%TTL#
1019 Correctly Frcst # % of Ttl <= Thrshld vs Thrshld VRF_%TTL#
1020 Observed # % of Ttl <= Thrshld # vs Thrshld OBS_%TTL#
1021 False Alarm Rate FLS_ALRM_RATE
1022 Conditional Miss Rate COND_MSS_RATE
1023 Heidke Skill Score HEIDKE_SKL_SCR
1024 Peirce-Hanssen-Kuipers Skill Score P-H-K_SKL_SCR
1025 ODDS RATIO ODDS_RATIO
1026 Log of ODDS RATIO LOG_ODDS_RATIO
1027 Standard error of log of ODDS RATIO STDERR_LOG(OR)
1028 ODDS RATIO Skill Score ODDS_RTO_SKL_SCR
1029 ODDS RATIO Skill Score Cubed ODDS_RTO_SKSC**3
1030 log (ODDS RATIO) / STD ERROR of same LN_OR/STDV_LN_OR
1031 Area between ODDS RATIO ROC & chance ROC curves ROC_AREA-CHANCE
* Definitions are only given here for scores 1001-1008. A good source to look up the definitions of many scores is the paper by Stephenson, David B. 2000: Use of the "Odds Ratio" for Diagnosing Forecast Skill. Weather and Forecasting: Vol. 15, No. 2, pp. 221-232. You can also dig into the source code that performs the calculations: ftp://ftp.emc.ncep.noaa.gov:/mmb/precip/vsdb_prcp/docs/scorecalc_sorc.tar (start from rdtd/rdtdtrc.f, and go from there). A necessary Disclaimer: the code is provided ONLY for information purposes. Anyone who copies the code with the intention of using it for system development must proceed without expectation of assistance from NCEP. Questions about the design of the software would have to be answered solely by inspection of the code itself (the author of the source code no longer works for EMC).

Return to the Q&A index


An example of score calculation

This is an example of how the numbers for a one-day Eta 24h equitable threat score is computed from that day's "vsdb" file from the new database. The 00-24h Eta precipitation forecast equitable threat score (red line) is shown below:

And here are the relevant lines in the "vsdb" file:

 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>.01 APCP/24 SFC =  1362  .35242  .27827  .44053
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>.10 APCP/24 SFC =  1362  .21072  .14537  .19677
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>.25 APCP/24 SFC =  1362  .11968  .07416  .11380
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>.50 APCP/24 SFC =  1362  .05507  .03671  .05066
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>.75 APCP/24 SFC =  1362  .02129  .01542  .03084
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>1.0 APCP/24 SFC =  1362  .00514  .00294  .01689
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>1.5 APCP/24 SFC =  1362  .00000  .00000  .00294
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>2.0 APCP/24 SFC =  1362  .00000  .00000  .00000
 V01 ETA 24 2002100212 MC_PCP G211/RFC FHO>3.0 APCP/24 SFC =  1362  .00000  .00000  .00000
From the above, we arrive at the table below:

Threshold
(in/day)
F H O Eq. Threat
(H-CH)/(F+O-H-CH)
where CH=F*O/T
0.01 478 379 600 0.345
0.1 287 198 268 0.471
0.25 163 101 155 0.415
0.5 75 50 69 0.512
0.75 29 21 42 0.409
1.0 7 4 23 0.150
1.5 0 0 4 0.0
2.0 0 0 0 0.0
3.0 0 0 0 0.0

Return to the Q&A index


What are bias-adjusted scores?

From the FVS versions and change history guide (K. Brill):

      fvs version = 2003.07

          The ability to compute bias adjusted performance measures
          and skill scores computed from FHO statistical values was
          added.  Two bias adjustment methods are supported.  Bias
          adjustment is triggered by prefixing the 4-digit computa-
          tional code number with the bias adjustment code number.
          The following two bias adjustment code numbers are valid:

          101 -> Apply Fedor Mesinger's assumption that dH/dF=a(O-H)
                 to compute the adjusted value of H for bias = 1,
                 i.e. F=O.

          102 -> Apply conservation of the "Odds Ratio" (Stephenson
                 2000: Use of the "Odds Ratio" for Diagnosing
                 Forecast Skill, Wea. & For., 15, 221--232) to
                 compute the adjusted value of H for bias = 1.

          Example:  To compute the bias adjust equitable threat
                    score based on conservation of the Odds Ratio,
                    use the following computational code: 1021002.
A paper by Mesinger and Brill, Bias Normalized Precipitation Scores, provides more details about the two bias adjustment methods.

Return to the Q&A index


What do the subregions look like?

Precipitation verification is done on ConUS (dubbed 'RFC') and its subregions. Maps for the current subregions are below:

In order to evaluate the performance of high-resolution window NMM and ARW runs, NAM, GFS and the hi-res runs are also verified on the western and eastern ConUS domains, starting 11 Sep 2007:

Note the overlap between the Western/Eastern regions above. They are NOT the Western/Eastern halves of ConUS.

Legacy masks in older VSDB files:

Return to the Q&A index


How do I plot FVS stats for a subregion of ConUS?

You can plot statistics for the entire ConUS ('RFC'), for a single subregion (e.g. 'NWC'), or a combination of subregions. To plot the western and eastern halves of the ConUS as in these two web pages, if you are using a script to do your FVS plotting and have 'RFC' as an argument for the plot region, substitute RFC with "NWC GRB NMT NPL SWC SWD SMT SPL" (in double quotes) for western region and "MDW APL NEC LMV GMC SEC" for eastern region.

Note that you cannot simply use "WEST" or "EAST" as arguments to your script if you want to plot statistics for the western/eastern halves of ConUS. Some VSDB files contain statistics that has WEST/EAST in their 'region' field, but those refer to the ConUS areas covered by the West/East HiRes Windows runs. The two areas overlap, and they are not the western/easten HALVES of ConUS.

Return to the Q&A index


Where can I find the raw daily gauge data used in the (old) verification?

For the current month:
ftp.emc.ncep.noaa.gov:/mmb/gcp/precip/katz/usa-dlyprcp-yyyymmdd

Within the current year, monthly tar files:
ftp.emc.ncep.noaa.gov:/mmb/gcp/precip/esop/usa-boulder-monyyyy (mon=jan,feb,...)

Annual big tar files for the previous couple of years might also be available:
ftp.emc.ncep.noaa.gov:/mmb/gcp/precip/esop/usa-boulder-yyyy

Contact person for the raw gauge data: Sid.Katz@noaa.gov

Return to the Q&A index


What is the verifying analysis?
For 3-hourly verification, verifying analysis is the NCEP Stage II.
From Jan 2002 - 22 Jan 2012, NCEP/CPC's 1/8 degree daily gauge analysis was used for 24h (12Z-12Z) verification. Starting Dec 2007, when the CPC analysis was unavailable/bad, the NCEP Stage IV was used in its stead.
Starting 23 Jan 2012, when the CPC analysis became unavailable, the 24h accumulation of the NCEP Stage IV is used for 24h verification.


What is the difference between the "old" and the "new" verification systems?

On 1 Sep 2002, a new verification package was put into place to verify ETA, GFS, NGM, various ETA parallels and mesoscale nests. The old system continued to run for several years, to provide historical continuity. Between Jan 2005 and Oct 2006, only the NAM and NGM were verified on the old system. The old system was completely discontinued on Oct 2006.

In the old system, the 7,000-8,000 daily rain gauge reports are box-averaged to the verification grid. At higher grid resolutions (40km and up), this leads to many "holes" in the analysis domain in the data-sparse regions. In the new system, the verifying analysis is mapped to the verifying grid from the daily, 0.125-degree analysis generated by NCEP/CPC, which is much better at higher resolutions. When the CPC analysis became unavailable in Jan 2012, it was replaced by the NCEP Stage IV. The grid-to-grid mapping is done using NCEP/EMC's 'copygb' utility. An example of one day's rain gauge reports, the analysis on various resolutions under the new and the old system is available here.

The following is an outline of the main differences between the old and the new systems:

Old System New System
Verifying analysis Raw gauge data box averaged to grid CPC 0.125-deg analysis mapped to grid
VSDB file on tempest (NCEP/EMC users) /mmb/wd22yl/vsdb_prcp24.old /mmb/wd22yl/vsdb_prcp24
Indices of tar'd VSDB files http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/vsdblist/mb24/ http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/vsdblist/mc24/
(pre2008) and
http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/vsdblist/$yyyy
(from Jan 2008)
VSDB data identifier MB_PCP MC_PCP
Verifying grids G104(90km)1, G211 (80km), G212 (40km) G104(90km)1, G211 (80km), G212 (40km), G218(12km)2
Starting Date3 1 Jun 1992 1 Sep 2002
1 G104 verification done for NGM only.
2 G218 verification done for ETA, GFS and various mesoscale nests at NCEP/EMC.
3 Not all models have been verified from the starting dates. Check indices (links in table above) for details.

Return to the Q&A index


Where can I find the VSDB files generated at NCEP?

Monthly archive (if you do not have account on NCEP CCS and/or workstations):

Daily tar files for the past week (for users outside of NCEP) ftp://ftpprd.ncep.noaa.gov/pub/data1/nccf/com/verf/prod/precip.yyyymmdd

VSDB archive on tempest (for users within NCEP/EMC): If the vsdb files you are looking for is not on tempest:
Since the uncompressed VSDB files take up a lot of space, the policy is not to keep any VSDB files (other than that for NAM) on line for more than a year or two. Older files are removed from tempest after being archived (see HPSS archive below). If you want to comparing NAM against, say, MODELX for a time period from several years ago, and MODELX's vsdb files are no longer on tempest, do the following:

VSDB files for operational models, on NCEP's CCS:

Monthly archive on NCEP IBM's HPSS storage: Information below (on this subject) needs further update:
The contents of the above files are listed in here. If you are looking for a vsdb file for, say, a parallel run you made back in 1995, and do not find them in the monthly/annual tar files above, check out this list for the tar file saved in HSM:/hsmuser/g01/wx22yl/vsdb_prcp/misc/vsdbmb_inactive.20010411.Z. It was made on 11 Apr 2001, when I removed a bunch of vsdb files in directories that had long been dormant in order to free up some disk space.

There are two other tar files in HSM:/hsmuser/g01/wx22yl/vsdb_prcp/misc: avn.20010101-20020831.tar.Z contains the original 'AVN' vsdb files for the 20 months, before I renamed files/entries for that period to 'GFS'. eta.19920601-20020912.vsdb.original.tar.Z contains the original ETA vsdb files, before I changed all G90 verification to G190, and deleted the 'native grid' verifications.

Return to the Q&A index


How do I generate my own scores plot from the VSDB files?

Graphics such as those in http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/ are generated with NCEP's FVS (Forecast Verification System) (though FVS can do a lot more). You are encouraged to find out more about the system from the documentations.

If you have access to "tempest" and the IBM mainframe at NCEP, you can also use a canned script (tempest:/export/lnx168/wd22yl/pscore/general/pscore) by following these steps:

  1. Check tempest:/mmb/wd22yl/vsdb_prcp24 to see if vsdb files for the model(s)/time period you wish to plot is available there. If they are, you can use the "pscore" script directly.
  2. If the vsdb files you need is not in the above directory (say if you generated your own vsdb files, or if the vsdb files are no longer there - I delete older vsdb files to free up disk space), create your own vsdbdirectory/sub-directories for them (vsdbdirectory=vsdb_prcp24 or whatever you prefer to call them; sub-directory=model name). If the vsdb files you need have been deleted from my directory, look for them in HPSS: /hpssuser/g01/wx22yl/verif.new/vsdb_prcp/mc for the monthly vsdb tar files. Change the "/raid/usr1/wd22yl/vsdb_prcp24" in "pscore" into $yourpath/$vsdbdirectory. Note that the script expects all needed vsdb files to be under the same $vsdbdirectory, so if (say) you are comparing a model run against the Eta for the same time period, the Eta vsdb files might be on line (I keep them around for a long time) but the other model's vsdb files might be gone, and you need to create and populate both eta and $othermodel (model names need to be in lower-case for the directories) under $vsdbdirectory.
  3. There are ten arguments to "pscore":
    1. Output type. I use "ps|plot.ps|8.5;11|c" (in quotes) for colored, letter-sized postscript file in portrait mode, and convert the plot.ps to a gif file later (outputting directly to gif might be possible if you are knowledgeable about GEMPAK)
    2. Model name(s). If there are more than one model, put the names in quotes. Note that the model names need to be upper-case
    3. Verifying grid (211/212/218)
    4. Forecast hours to be verified. If there are more than one, put them in quotes, e.g. "24 36 48 60".
    5. Beginning $yyyymmdd for verification
    6. Ending $yyyymmdd for verification
    7. Verifying region ('RFC' means ConUS) or subregion(s). If combining more than one subregions, put them in quotes.
    8. Type of scores (optional): the default plot has equitable threat score on the top and bias on the bottom. If you wish to plot a pair of other scores, enter there 4-digit codes. For example, "1007 1004" would give you probability of detection/false alarm ratio.
    9. and x.: line and marker options. Have not tried this before, they might not work. If you wish to set your own colors and markers inside the "pscore" script, modify 'lineopt' and 'markopt' (see GEMPAK user manual).
  4. You can also look into tempest:/export/lnx168/wd22yl/pscore/general/README for examples of how to use the script.

If you don't have access to tempest but have access to NCEP's IBM mainframe, you can run the FVS there. NCEP's Mary Hart has created a documentation on how to do this, though the documentation is not online yet (ask Mary for the documentation).

Return to the Q&A index


How do I generate my own verification statistics?
The precipitation package is available via SubVersion at https://svn.ncep.noaa.gov/emc/verif/g2g_precip/. Currently this is only accessible by NCEP people. We hope that in the future a read-only version of the SVN can be made available to the public.

Here is the User Guide for the verification package.

Return to the Q&A index


How are the scores computed for multiple sub-regions/time periods?

The VSDB files contain the F,H,O,T numbers (actually, they are F/T,H/T,O/T and T) for each precipitation threshold/verification subregion/verification time period. To compute a score for a combination of subregions (e.g. eastern U.S.) or time periods (e.g. 24h forecasts for an entire month), the normalized F/H/O values are reconstituted by multiplying by T, then added together to form a "total F/H/O" for the entire combined region and/or combined time periods before the scores are computed.

Return to the Q&A index


Why doesn't the monthly/month-to-date page include more scores plots?

The monthly/month-to-date page provides selected plots that are most pertinent to the monitoring and development of NAM. I try to check on most of the plots each day to make sure that things are OK (it's not generated by a production job, so if one of the machines or one of disks involved was down, or there was a network communication problem, the scores might not have been updated). Due to time constraints, the number of plots included in the web page has to be limited to make routine monitoring feasible.

For additional plots and much more flexibility, you might wish to try the interactive web tool, http://cerberusdev.ncep.noaa.gov/EMC_VSDB_verif/.

Return to the Q&A index


Info on precip SL1L2 scores?

In addition to F-H-O stats, scalar L1L2 stats began to be computed for the precipitation verification VSDB files at the end of Apr 2006 (27 Apr for 3-hourly stats, 28 Apr for daily stats) on the G218 verification grid, for ConUS, the 14 sub-regions of ConUS, and the 3 "nested regions".

A sample script plotting model vs. obs area-averaged 3-hourly precipitation is in /export/lnx168/wd22yl/pscore/avgpcp/pscore_multicyc. For example, to plot a sum of all 00Z cycles in Jul 2006, for NAM and GFS, on ConUS:

pscore_multicyc "ps|plot.ps|11;8.5|c" "NAM GFS" 00 20060701 20060731 RFC 0.6

Arguments for the script:

  1. Graphic output file/format
  2. Model(s) to be plotted. Have not tried this for more than 3 models.
  3. Model cycle (00, 06, 12 or 18)
  4. Starting day of verification
  5. Ending day of verification
  6. Region/subregion name. 'RFC' means ConUS. See here for additional info on subregions
  7. Optional argument: maximum value on y-axis (unit: mm). Without this argument the top value on the vertical axis would depend on the maximum model/obs average precip value. If you are making multiple plots (say for different cycles or verifying periods) and want to have the same vertical scale, set this argument (that is, after making plots without this argument and finding out what the appropriate maximum value should be)

A sample script to plot area-averaged precipitation for a single model cycle is in /export/lnx168/wd22yl/pscore/avgpcp/pscore_1cyc. For example, to plot ConUS-averaged precipitation for NAM and GFS for the cycle of 00Z 20070410:

pscore_1cyc "ps|plot.ps|11;8.5|c" "NAM GFS" 2007041006 RFC 0.5

Arguments for the script:

  1. Graphic output file/format
  2. Model(s) to be plotted. Have not tried this for more than 2 models.
  3. Model cycle (yyyymmddhh
  4. Region/subregion name. 'RFC' means ConUS. See here for additional info on subregions
  5. Optional argument: maximum value on y-axis (unit: mm). Without this argument the top value on the vertical axis would depend on the maximum model/obs average precip value.

A note on the two sample FVS scripts: The sample script is provided for your convenience. While it is correct to my best knowledge, I am not an FVS expert and you are advised to consult, always, the official FVS document and FVS help pages, and construct your trace.ctl file and run scripts based on these documents (tip: plotting precip SL1L2 is essentially the same as plotting other scalar L1L2 statistics, i.e. 2m temperature or RH). Should there be a conflict between the sample scripts here and the official FVS document/FVS help pages, the language of the official FVS document/FVS help pages will control.

Return to the Q&A index


Does smoothing a model's QPF fields give it a better ETS score?

In re whether a model can score a higher ETS than its competitors by mapping its QPF output to a coarser grid: not if the comparison is done on a common verifying grid that is no finer than the coarsest of all the model output to be compared (yes, to some degree, if the verifying grid is finer).

Performance measures are sensitive to the spatial scale - generally a performance measure based on point-by-point comparison (such as the ETS) yields a lower score if model output and verifying analysis are at a higher resolution. If we are comparing a model forecast with more details (i.e. at higher resolution) with another that is smoother (whether because the model itself is of lower resolution or because the original model output has been mapped to a lower resolution grid), it is important that the comparison is done on a common verifying grid that is no finer than the coarser of the two model grids.

If both models are mapped to a coarser grid, does the "smoother model" (or the model output that has been mapped to a coarser grid) still has an advantage? To explore this, I did a 2-week experiment with the NAM's output at four different resolutions (12km native grid; 40 and 80km Lambert Conformal; 190.1km polar stereographic). The verification for all four were done on the three verifying grids normally used for our QPF verification: the 12/40/80km Lambert Conformal grids (G218/212/211).

As seen in P1-3 of this ppt, when the verifying grid is finer than the coarser of the model fields, the coarser fields tend to have an advantage at lower thresholds (at higher thresholds, coarser model output tend to lose out due to the smoothing out of forecast maxima). This advantage disappears when the verifying grid is no finer than the coarser model fields.

My conclusion is, while on a high-resolution verifying grid, model output of a finer resolution are likely to be penalized when compared to model output of coarser resolution, this factor is removed if both models are mapped to a common coarser grid, and more smoothing (i.e. additional mapping before the final mapping to the verifying grid) by itself does not give a model an advantage.

There are three verifying grids in our precipitation verification: the 12/40/80km Lambert Conformal grids (G218/212/211). Most NCEP models (opnl and parallel) are verified on all three grids. P4 of the ppt shows a comparison of NAM and GFS scores on each of the three grids - as you can see, the same model output tends to have lower scores when mapped to higher resolution grids.

International models are verified on the 40/80km grids. As of Jan 2013, the QPF files ECMWF send to us for the purpose of precipitation verification is on a 0.5 deg grid (and cropped to 24-50N, 125-65W). The coarsest model output resolution is for the CMC global model, at 0.9 x 0.7 deg (lat x lon), so the fairest comparison would be at the 80km G211, such as in http://www.emc.ncep.noaa.gov/mmb/ylin/pcpverif/scores/2012/201210/intl_glb.201210.gif

1 Feb 2013

Return to the Q&A index


yingDOTlinATnoaaDOTgov

Back to
NCEP/EMC Model Precipitation Scores Page

NCEP/EMC Mesoscale Modeling Branch