The New GRID2OBS Verification System:

User Guide

 

Last Modified: 24 Oct 2008

Julia Zhu

Email:  yuqiu.zhu@noaa.gov

 

This is a document on how to run the gridtobs verification using this new version of scripts and codes only, for more information on how this verification system works, please refer to Perry Shafran’s document http://www.emc.ncep.noaa.gov/mmb/papers/shafran/gridtob.pdf

 

This version of codes and scripts will become an official version to be implemented in NCEP’s production suite. It contains the gridtobs verifications on NAM, NDAS, DGEX, GFS, GDAS, HIRESW, NAMHW, RTMA, RSM, RUC, SREF (deterministic), SREFX (bias-corrected deterministic), SREFENS (the SREF ensemble), SREFENSX (bias-corrected SREF ensemble), OZONE, PM (Particular Matters), OZONEMAX, PMAVE and PMMAX.

 

1.               Flow Structure of this system

 

./sms/jverf_gridtobs_nam_00.sms (the SMS script, an example only)

 

   ./jobs/JVERF_GRIDTOBS_FITS.sms.dev (J-Job script, one script shared by all the models)

 

         ./scripts/exverf_gridtobs_fits.sh.sms (Ex-script, one script shared by all the models) 

 

               ./ush/verf_gridtobs_fits.sh  (Main script that executes the EDITBUFR, PREPFITS and GRIDTOBS steps)

  

                     ./ush/verf_gridtobs_getbufr.sh (This script decides which “prepbufr” file to be copied over)

                     ./ush/verf_gridtobs_getbufr_aqm.sh (This script is special to the AQM model only)

                     ./ush/verf_gridtobs_getmodinput.sh (This script obtains the model input grib files)

 

SMS Scripts

 

These are the job submission scripts. We should have one SMS script for each model on each cycle. In production, the gridtobs verification are to be divided and run in 8 cycles (00, 03, 06, 09…21). There is only one submission script that will process the EDITBUFR, PREPFITS and GRIDTOBS all together in one job. These scripts I have in the subversion do not assume to the jobs to be submitted through an SMS server, but you will still need to modify the following variables in order to run in your own account:

 

# @ output = /ptmp/wx22yz/com/output/prod/today/verf_gridtobs_gfs_12.o$(jobid)

# @ error = /ptmp/wx22yz/com/output/prod/today//verf_gridtobs_gfs_12.o$(jobid)

# @ initialdir = /ptmp/wx22yz/tmpnwprd

 

/meso/save/wx22yz/verif/nwprod/jobs/JVERF_GRIDTOBS_FITS.sms.$envir

 

You can just substitute “wx22yz” with your own account or change to whatever directory you would like to save those files. You can also change the “class” and “group” if necessary.

 

J-Job Scripts: JVERF_GRIDTOBS_FITS.sms.dev

 

This is a generic script to set up various environment variables. At the beginning of this script, I put a “key-switch” variable called “RUN_ENVIR” to decide whether this job is to be run in a “development” environment (EMC) or a production environment (NCO). Depending on the input for this environmental variable, the job then reads in a parameter file called “verf_gridtobs_config” which actually defines the working directory, the jlogfile, the locations of scripts, parameter files and executables, the input/output directories…and so on. You will need to review this parm file and make necessary modifications.

 

EX-Scripts: exverf_gridtobs_fits.sh.sms

 

This is a generic driver script. It reads in the parameter file called “verf_gridtobs_modelinfo”, which specifies the “regions” of the verification and the verification length for a certain model. For example, for the NAM model, we will verify 4 regions including “NAM”, “NAM40”, “NAMAK” and “NAM40AK”, and the verification length is 84 hours. For the SREF model, the “regions” will be the 21 members plus the ensemble run (totally 22 “regions”).

 

USH-Scripts:

 

These are called the utility scripts, they are the scripts called by the driver scripts and do the real work.

 

1)      verf_gridtobs_fits.sh

This is the main script called by exverf_gridtobs_fits.sh.sms to process the EDITBUFR, PREPFITS and GRIDTOBS steps.

 

2)      verf_gridtobs_getbufr.sh and verf_gridtobs_getbufr_aqm.sh

They are called by verf_gridtobs_ fits.sh to obtain the appropriate prepbufr file (the OBS file) for the verification.

 

3)      verf_gridtobs_getmodinput.sh

This script determines and retrieves the necessary model output files to be verified against the observations.

 

4)      verf_gridtobs_ens_sref.sh

This is the driver script to calculate the VSDB records for the ensemble verification (GRIDTOBS_ENS) for the SREF model only. It decides the number of pieces for the job to run on POE(currently I divide it into 33 pieces for the 00/12Z cycles, and 27 pieces for all  other cycles , which means this job will need 33 (00/12Z) and 27 (other cycles)  tasks to run. It is called by the J-Job script JVERF_GRIDTOBS_FITS.sms.dev.  This script is specific to the “SREFENS” and “SREFENSX” runs only.

 

5)      verf_gridtobs_srefens.sh

This script is a sub-script to verf_gridtobs_ens_sref.sh. It is the main script that calculates the VSDB records for each “piece” of the ensemble run. This script is specific to the “SREFENS” and “SREFENSX” runs only.

 

6)      verf_gridtobs_aqmmax.sh

This script generates the VSDB records for the “max” and “ave” verifications for the OZONE and PM runs. It is called by the J-Job script JVERF_GRIDTOBS_FITS.sms.dev. This script is specific to the “OZONEMAX”, “PMMAX” and “PMAVE” verifications only.

 

PARM Files:

 

1)      verf_gridtobs_config  (common to all models)

This file is used in the J-Job script. It defines the locations of the scripts, parameter files and executables, the working directory, the input/output directories, the jlogfile and the values for the several variables such as SENDCOM, SENDDBN, SENDSMS. The usage of this parameter file will make it possible for the developers and the production to share the same J-Job script with only a change in a key-switch variable called “RUN_ENVIR”. For the development run, we should use “RUN_ENVIR=dev”.

 

2)      verf_gridtobs_modelinfo (common to all models)

This file is used in the EX-script. It defines the “regions” or “grids” to be verified for a specific model and the verification length. To add a new model (e.g. namx, namy) to be verified, just add it to the end of this file with the same format as the existing models shown below:

 

MODEL|REGIONS|VERIFICATION LENGTH|FORECAST RANGE|

aqm|ozone pm|48|48|

gfs|gfs gfs40 gfs40ak|84|84|

gdas|gdas|09|09|

ngm|ngm|48|48|

nam|nam nam40 namak nam40ak|84|84|

ndas|ndas|03|03|

ruc|ruc ruc40|12|12|

rtma|rtma|00|00|

dgex|dgexco dgexak|192|192|

gfse|gfseco gfseak|192|192|

hiresw|eastnmm eastarw westnmm westarw aknmm akarw hinmm hiarw prnmm prarw|48|48|

rsm|rsmh|48|48|

namhw|nameasthw namwesthw namprhw namhihw namakhw|48|48|

sref|ebmjc ebmjp ebmjn ekfcc ekfcp ekfcn rsasc rsasp rsasn rrasp rrasn emc emp1 emn1 emp2 emn2 nmmc nmmp1 nmmn1 \

     nmmp2 nmmn2 srmean ebmjcak ebmjpak ebmjnak ekfccak ekfcpak ekfcnak rsascak rsaspak rsasnak rraspak \

     rrasnak emcak emp1ak emn1ak emp2ak emn2ak nmmcak nmmp1ak nmmn1ak nmmp2ak nmmn2ak srmeanak|87|87|

srefx|ebmjcx ebmjpx ebmjnx ekfccx ekfcpx ekfcnx rsascx rsaspx rsasnx rraspx rrasnx emcx emp1x emn1x emp2x emn2x \

      nmmcx nmmp1x nmmn1x nmmp2x nmmn2x srmeanx|87|87|

 

3)      verf_gridtobs_cycles (common to all models)

This parameter file defines the cycles in which the model output data are available. The cycles are used for the “verf_gridtobs_getmodinput.sh” script to determine the model input files to be added to the verification.

 

4)      verf_gridtobs_domains (common to all models)

This parameter file defines several parameters for the verification of a specific model “region”, where:

 

1st column:    the model region, e.g. for NAM, the regions are “nam” (grid 218), “nam40” (grid 212), “namak”(grid 242) and “nam40ak”(grid 216)

2nd column:   the NET variable. Regions/Models are grouped into several “NET” which share the same input parm cards to the PREPFITS and GRIDTOBS steps.

3rd column:    the RUN variable. This variable is used to determine the input model file name, e.g. for the GDAS model, the file name will be “gdas1.tCCz.pgrbfHH”, and the model file name should be  gdas1”.

4th column:    the model name in capital case to be used in the GRIDTOBS input parm card.

5th column:    the model name in capital case to be used in the PREPFITS input parm card.

6th column:    the starting forecast hour of the model output.

7th column:    the end forecast hour of the model output.

8th column:    the model output increment.

9th column:    the first segment of model input file name

10th column:  the second segment of the model input file name

11th column:  the third segment of the model input file name

12th column:  the extension (.tm00) of the file name

13th column:  whether COPYGB is needed to convert the input file into different grib file.

14th column:  the grid number for the input files to be converted to using COPYGB

15th column:  the verification grid

 

Here is a portion of the file(for the NAM and GFS models). If you need to add a new model to the verification, just add it to the end of this file with the same format as the existing models.

 

#REGION||NET|RUN|MODEL|PLL3|SFHR|EFHR|INC|FILNAM1|FILNAM2|FILNAM3|TMKK|DOCYGB|CGRID|GRDNAM|

### GFS ###

gfs|global|gfs|GFS|GFS|00|84|06|pgrbf| | | |NO| |3|

gfs40|global40|gfs|GFS|GFS|00|84|06|master.grbf| | | |YES|212|212|

gfsak|globalak|gfs|GFS|GFS|00|84|06|master.grbf| | | |YES|242|242|

gfs40ak|globalak|gfs|GFS|GFS|00|84|06|master.grbf| | | |YES|216|216|

 

### NAM ###

nam|meso|nam|NAM|NAM|00|84|06|awphys|awip12|awip32|.tm00|NO| |218|

nam40|meso|nam|NAM|NAM40|00|84|06|awip3d|awip3d| |.tm00|NO| |212|

namak|mesoak|nam|NAM|NAM|00|84|06|awak3d|awak3d| |.tm00|NO| |242|

nam40ak|mesoak|nam|NAM|NAM40|00|84|06|awipak|awipak| |.tm00|NO| |216|

 

5)      verf_gridtobs_grid104 (common to all models, in GRIDTOBS step)

There are two files “regions” and “grid104” that are used to divide grid 104 into different sub-regions so that statistics may also be performed within each of these sub-regions. This file
“grid104” assigns each grid point in the NCEP grid 104 with a unique sub-region number that is consistent with the definition in the file regions. Currently, 29 sub-regions are defined within NCEP grid 104. This is a fix file that you DO NOT need to modify when adding a new model.

 

6)      verf_gridtobs_regions (common to all models, in GRIDTOBS step)

This file defines the two-digit number and three-letter abbreviation associated with each sub-region. This is also a fix file you DO NOT need to change when adding a new model.

 

7)      verf_gridtobs_ keeplist (common to all models, in EDITBUFR step)

This is an input control file to be used in the EDITBUFR step, which allows specification of time window, areal extent and observation type to be included in the output file in order to filter out unnecessary observations. If you want to add a new model to the verification, please first check and see if your “NET” (in the 2nd column of the “verf_gridtobs_domains” file) has been defined in this file. If not, you can add it to the end of this file using the same format as in the existing NET.

 

8)      verf_gridtobs_levcat (common to all models, in the PREPFITS step)

This file specifies the number of levels to be read from the GRIB files and the logical flag of the data categories. There are 10 data categories. If you need to add a new model to the verification, please add the “regions” to the end of the file using the same format as in the existing file.

9)      verf_gridtobs_ prepfits.tab (common to all models, in the PREPFITS step)

This is a bufr table defining BUFR mnemonics. For the OZONE and PM runs, they use their own “prepfits.tab” files, which are “prepfits.tab_ozone” and “prepfits.tab_pm” respectively.

 

10)  verf_gridtobs.${NETWORK}  (each NETWORK has one specific GRIDTOBS control file)

The control file to be used in the GRIDTOBS step, it contains essentially the bounding parameters over which the partial sums are to be accumulated.  This bounding parameter info also facilitates generation of the 9 headers of the VSDB record. Please refer to Perry’s document http://www.emc.ncep.noaa.gov/mmb/papers/shafran/gridtob.pdf for a detailed description of each element in this control file. The control files for all the verification models are grouped into several NETWORK according to the verification OB types, verification areas (grids), verification variable types and verification levels. Here is an example of the control file (verf_gridtobs.meso), the “MODNAM”, “GRDNAM” and the number of hours will be generated interactively in the script and inserted into this control file to form a complete input control file to the GRIDTOBS step. If you want to add a new model and do not see a control file for your model (NETWORK) in the directory, please create one for your own by referring to this file.

 

V01   10

    1  MODNAM/GRDNAM

    1  19

    1  ADPUPA

    4  G236

  G245

  G246

  G247

    2  SL1L2

  FHO

   11  Z

  T

  Q

  RH

  PBL  6 500 1000 1500 2000 2500 3000

  CAPE 6 500 1000 1500 2000 3000 4000

  CIN

  LI 5 -10 -8 -5 -2 0

  TROP

  PWO

  VWND

   11  P1000

  P850

  P700

  P500

  P400

  P300

  P250

  P200

  P150

  P100

  P50

V01   10

    1  MODNAM/GRDNAM

    1  19

    3  ANYAIR

  PROFLR

  VADWND

    4  G236

  G245

  G246

  G247

    1  SL1L2

    2  T

  VWND

    9  P1000-850

  P850-700

  P700-550

  P550-400

  P400-300

  P300-250

  P250-200

  P200-150

  P150-50

V01   10

    1  MODNAM/GRDNAM

    1  19

    3  ANYSFC

  ONLYSF

  GPSIPW

   14  G104/NWC

  G104/SWC

  G104/NMT

  G104/SMT

  G104/GRB

  G104/SWD

  G104/NPL

  G104/SPL

  G104/MDW

  G104/LMV

  G104/GMC

  G104/NEC

  G104/SEC

  G104/APL

    2  SL1L2

  FHO

   14  SLP

  T

  TMAX

  TMIN

  RH 12 10 20 30 40 50 60 70 80 85 90 95 100

  DPT 7 0 5 10 15 20 25 30

  VIS 7 400 800 1600 3200 6400 12000 15000

  TCLD 12 10 20 30 40 50 60 70 80 85 90 95 100

  GUST 4 10 20 30 40

  HEAT 4 80 90 100 110

  CHILL 5 -20 -10 0 10 20

  PWO

  CLDBT

  VWND

    1  SFC

V01   10

    1  MODNAM/GRDNAM

    1  19

    3  ANYSFC

  ONLYSF

  GPSIPW

   13  G104/HWI

  G104/NPO

  G104/SPO

  G104/NAO

  G104/SAO

  G104/GLF

  G104/PRI

  G104/CAR

  G104/CAM

  G104/NSA

  G104/WCA

  G104/ECA

  G104/MEX

    2  SL1L2

  FHO

   14  SLP

  T

  TMAX

  TMIN

  RH 12 10 20 30 40 50 60 70 80 85 90 95 100

  DPT 7 0 5 10 15 20 25 30

  VIS 7 400 800 1600 3200 6400 12000 15000

  TCLD 12 10 20 30 40 50 60 70 80 85 90 95 100

  GUST 4 10 20 30 40

  HEAT 4 80 90 100 110

  CHILL 5 -20 -10 0 10 20

  PWO

  CLDBT

  VWND

    1  SFC

V01   10

    1  MODNAM/GRDNAM

    1  19

    3  ANYSFC

  ONLYSF

  GPSIPW

    1  G236

    2  SL1L2

  FHO

   14  SLP

  T

  TMAX

  TMIN

  RH 12 10 20 30 40 50 60 70 80 85 90 95 100

  DPT 7 0 5 10 15 20 25 30

  VIS 7 400 800 1600 3200 6400 12000 15000

  TCLD 12 10 20 30 40 50 60 70 80 85 90 95 100

  GUST 4 10 20 30 40

  HEAT 4 80 90 100 110

  CHILL 5 -20 -10 0 10 20

  PWO

  CLDBT

  VWND

    1  SFC

 

 

1.               Where to get the scripts and Codes

 

You can check out the GRIDTOBS verification package from EMC’s Subversion Server on

 

https://svn.ncep.noaa.gov/emc/verif/grid2obs/gridtobs_new

 

2.               What you need to change to make your own verification run

 

1)                  The SMS scripts

 

Currently what we have in running are only the operational models, including:

 

AQM (including OZONE and PM)

AQMMAX (including OZONE and PM)

DGEX

GDAS

GFS

HIRESW

NAM

NAMHW

NDAS

NGM

RSM

RTMA

RUC

SREF

SREFX

SREFENS

SREFENSX

 

Except for the AQM and DGEX (runs 4 times per day), all the other models run every 3-hourly (00Z, 03Z, 06Z, ….21Z). Each model for each cycle will need to have a separate SMS script. So if you want to run your verification 3-hourly, you will need to create 8 SMS scripts for it.

 

2)                  The PARM files

 

The Parm files you will need to edit to add your new model verification are:

 

verf_gridtobs_config  (need to define or modify the INDIR variable)

verf_gridtobs.modelinfo

verf_gridtobs.cycles

verf_gridtobs.keeplist   (grouped by NETWORK)

verf_gridtobs.levcat

verf_gridtobs.domains

 

And you will also need to create your own the GRIDTOBS control file (to generate the VSDB records) if it does not exist. Currently the control files (grouped by NETWORK) we have are:

 

DGEX:             verf_gridtobs.dgex    verf_gridtobs.dgexak

GFS/GDAS:     verf_gridtobs.global  verf_gridtobs.globalak   verf_gridtobs.global40

NAM/NDAS:  verf_gridtobs.meso  verf_gridtobs.mesoak

NGM:              verf_gridtobs.ngm

RUC:               verf_gridtobs.ruc

RTMA:                        verf_gridtobs.rtma

HIRESW/NAMHW:    verf_gridtobs.hrwwest  verf_gridtobs.hrweast  verf_gridtobs.hrwhi        verf_gridtobs.hrwpr  verf_gridtobs.hrwak

SREF/SREFX: verf_gridtobs.sref    verf_gridtobs.srefak

OZONE/PM:   verf_gridtobs.ozone      verf_gridtobs.pm

SREFENS/SREFENSX:          verf_gridtobs.srefens   verf_gridtobs.srefensx

 

Specifically, if the model you want to add to the verification is not an operational model (running at NCO), you will need to define or modify the INDIR variable to point to the model input file directory in the verf_gridtobs_config file. If it is not defined previously, it will take the default value set up in the J-Job script as:

 

Export INDIR=${INDIR:-/com/${NET}/prod/${RUN}}

 

3)                  The USH script

 

Usually you might not need to modify any of the scripts to add a new model into the verification system. But if the naming structure of the input files is specific(like the NAM, NDAS, RUC, RTMA, SREF, SREFX), you will need to check the verf_gridtobs_getmodinput.sh script to see if you need to add a block there or not (in the “case” statement).

 

 

3.               How to run the verification

 

To run the GRIDTOBS verification, simply submit the jobs to the loadlevelor using “llsubmit”, e.g. to submit the NAM model verification, you will do:

 

llsubmit  verf_gridtobs_nam_00.sms  (for the 00Z cycle)