Please take a moment to read the Disclaimer for this non-operational web page.  

The dumping of observational data is the first step in each NCEP network production suite.  At the appropriate network data cutoff time, up to three separate jobs are executed simultaneously - two dump jobs and a tropical cyclone processing job (the latter job is now actually submitted 5 minutes prior to the two dump jobs).  Once the two dump jobs have completed, a separate dump post processing job is initiated.

A .  Copying Files For Later Use By Analyses

In the Global Forecast System (GFS) and Global Data Assimilation System (GDAS) network runs, GRIB files containing current analyses of snow depth, ice distribution, and sea-surface temperature from NESDIS are copied from the NCEP IBM Central Computer System (IBM-CCS) /dcom database into network-specific (“/com”) directories.  These fields will be read later by the Global  Gridpoint Statistical Interpolation (GSI) analysis.  (Note: If the current files are not available, then one-day old files are copied.)

In the North American Model (NAM) and North American Data Assimilation System (NDAS) network runs, GRIB files containing current 16'th mesh  (24 km) analyses of snow/sea ice coverage produced by the NOAA/NESDIS Interactive Multisensor Snow and Ice Mapping System (IMS) are copied from the NCEP IBM-CCS /dcom database into network-specific /com directories.  These fields will be read later by the Regional Gridpoint Statistical Interpolation (GSI) analysis.  (Note: If the current files are not available, then one-day old files are copied.)

In the full-cycle Rapid Refresh (RAP) network, GRIB files containing current 96'th mesh (4 km) analyses of snow/sea ice coverage produced by the NOAA/NESDIS Interactive Multisensor Snow and Ice Mapping System (IMS) are copied from the NCEP IBM-CCS /dcom database into network-specific /com directories.  These fields will be read later by the RAP Gridpoint Statistical Interpolation (GSI) analysis.  (Note: If the current files are not available, then one-day old files are copied.) 

B.   Dumping of BUFR Observational Data (excluding WSR-88D Level II radial wind and reflectivity - see Dump Job 2 for the dumping of these data)

The process of accessing the observational database and retrieving a select set of observational data is accomplished in several stages by a number of FORTRAN codes.  This retrieval process is run in all of the operational networks many times a day to assemble “dump” data for model assimilation.   The script that manages the retrieval of observations provides users with a wide range of options.  These include observational date/time windows, specification of geographic regions for filtering (via either a lat/lon box, a center point lat/lon and radius, or a lat/lon grid point mask), data specification and combination, duplicate checking and bulletin “part” merging, and parallel processing.

The primary retrieval software performs the initial stage of all data dumping by retrieving subsets of the NCEP IBM-CCS BUFR /dcom data base that contain all of the data base messages valid for the data type, geographical filter and time window requested by a user. (Recall that the /dcom data base is continuously updated with new data as the data GTS decoder and satellite ingest jobs run.)  The retrieval software looks only at the date in Section One of the BUFR message to determine which messages to copy for a particular data type.  This results in an observing set containing possibly more data than was requested, but allows the software to function very efficiently.

The second stage of the process performs a final 'winnowing' of the data to an observing set with the exact time window requested¹.  This is done within the codes which remove exact- or near-duplicate reports (the nature of which is data type dependent) and merge bulletin parts for upper-air reports.

¹Normally, the six-hour cycle GFS, GDAS, and Climate Data Assimilation System (CDAS) network runs dump BUFR data globally over a six-hour time window centered on the analysis time.  The six-hour cycle NAM and NDAS network runs normally dump data within the expanded WRF-NMM-model domain over a six-hour time window centered on the analysis time (the NDAS assimilates data and updates every three-hours).  The one-hour full-cycle RAP, partial-cycle RAP (RAP_PCYC) and RTMA network runs normally dump BUFR data within the expanded WRF-NMM-model domain (a superset of the RAP and RTMA domains) over a one-hour time window centered on the analysis time.  The RTMA dump domain includes Guam and surrounding waters.  The one-hour cycle surface RUC network runs normally dump BUFR data within the expanded RUC domain over a one-hour time window centered on the analysis time.

The final stage of the process is the application of manual quality marks to the data extracted.  The quality marks are provided by personnel in two groups: the NCEP/NCO Systems Integration Branch (SIB) and the NCEP/Ocean Prediction Center (OPC).  The NCEP/NCO/SIB Senior Duty Meteorologists (SDMs) can apply quality markers to individual variables in many observational data types such as rawinsonde, dropwinsonde, PIBAL, aircraft, satellite wind, surface land, surface marine, wind profiler/SODAR and Vertical Azimuth Display (VAD) wind reports.  These markers either ensure that the datum marked will be assimilated by the particular analysis regardless of any subsequent quality control on it (called a "keep" flag), or ensure that it will NOT be assimilated (called a "purge" flag).  The SDMs use an interactive program on the IBM-CCS which initiates the off line execution of automated quality control programs run in the subsequent PREPBUFR processing steps and then review the programs’ decisions before making assessment decisions.  The SDMs use satellite pictures, meteorological graphics, continuity of data, input from reporting stations, past station performance and horizontal data comparisons (buddy checks) to decide whether or not to override quality control flags from the automated programs.  All flags are stored in an ASCII file on the IBM-SP for use during this data retrieval process.  The NCEP/NCO/SIB also maintains a list of data that should be rejected based on, among other things, monthly statistics provided from the NCEP and other international centers, and feedback from data producers.  All rejected data receive either a "reject" or "purge" flag here.  The flags are appended to the same ASCII file used for storing the SDM quality marks.  NCEP/OPC personnel perform real-time interactive quality control of global surface marine meteorological data and sea surface temperature using a graphical interactive program called CREWSS (Collect, Review, and Edit Weather data from the Sea Surface).  CREWSS provides an evaluation of the quality of the marine surface data provided by ships, buoys (drifting and moored), Coastal Marine Automated Network (CMAN) stations, and tide gauge stations by comparing the observations to GFS model first guess fields for all four synoptic periods. Data that differ from the first guess fields by more than certain amounts are then examined via techniques that involve buddy checks versus neighboring platforms, the platform’s track, and a one week history for each platform.  The NCEP/OPC personnel can either mark these data according to their quality, here applying either a "keep" or "purge" flag, or they can correct obvious errors in the data, such as incorrect hemisphere, misplaced decimal, etc. (corrected data receive a "good" quality mark in the subsequent PREPBUFR processing steps.)   Upon completion of interactive quality control, an ASCII text file containing all quality control decisions and corrections is then uploaded to the IBM-CCS for use during this data retrieval process.

Each data type selected for dumping is associated with a unique mnemonic string which represents a particular BUFR type and subtype in the /dcom database.  The complete list of BUFR data types is shown in Table 1.a.  This includes obsolete data types, future data types, and current data types which are currently not dumped in any network job.  In order to limit the number of output dump files in the operational network jobs, like data types are grouped together and represented by sequence or group mnemonics.  The data group mnemonics used to generate dump files in the various NCEP networks (including obsolete types)  are read by either the subsequent PREPBUFR processing steps , by the subsequent analysis codes, or by neither according to network.  See Table 1.b for a listing of data group mnemonic dumps read by the PREPBUFR processing steps and Table 1.c for a listing of data group mnemonic dumps read by the analysis codes. 

C.   Re-processing of BUFR Observational Data Dump Files

Some of the BUFR data dump files are re-processed into new BUFR files such that they can be used properly by the subsequent PREPBUFR processing or analysis programs.

 1.  SSM/I data - all network runs (NOTE: The SSM/I data went bad in November 2009 resulting in no data being processed.  All processing here was permanently turned off in October 2010):  The “reports” in the SSM/I products BUFR dump files (group mnemonics “ssmip” or “ssmipn”, see Table 1.b) consist of orbital scans, each of which contain 64 retrieval footprints of one or more products.  The program PREPOBS_PREPSSMI unpacks selected products out of the scans, superobs them onto a one-degree latitude/longitude grid (optional in some network runs) then encodes them as individual “reports” in the output, re-processed, BUFR file which contains only those data needed for subsequent PREPBUFR processing.  The output filename contains the qualifier “spssmi” (see Table 1.b, key for superscript 2 in “NET” column).  The GDAS, GFS and CDAS network runs superob the “operational” rainfall rate product generated at FNMOC, and the surface ocean wind speed and total column precipitable water products generated using a Neural-Net 3 algorithm (OMBNN3) developed by the Marine Modeling Branch of NCEP/EMC.  The NAM and NDAS network runs superob the “operational” surface ocean wind speed and total column precipitable water products generated at FNMOC.  The upper-air RUC network run processes the same products as the NAM and NDAS network runs but it does not superob the data.

 2. QuikSCAT data - NAM, NDAS, GFS, GDAS and CDAS network runs (NOTE: The QuikSCAT data went bad in November 2009 resulting in no data being processed.  All processing here was permanently turned off in October 2010): Each “report” in the QuikSCAT BUFR dump file (group mnemonic “qkscat”, see Table 1.b) consists of four sets of nudged wind vectors and other raw scatterometer information.  The program WAVE_DCODQUIKSCAT unpacks each report checking the report date for realism, selecting the proper nudged wind vector, and excluding reports over land, reports with missing nudged wind vector, reports with missing model wind direction and speed, reports with probability of rain greater than 10%, and reports at the edges of the orbital swath.  Reports passing checks are then superobed onto a one-half degree lat/lon grid according to satellite id and encoded into the output, re-processed BUFR file which contains only those data needed for subsequent PREPBUFR processing.  The output filename contains the qualifier “qkswnd” (see Table 1.b, key for superscript 1 in “NET” column).

 3. TRMM TMI data - GFS, GDAS and CDAS network runs: Each “report” in the TRMM TMI BUFR dump file (group mnemonic “trmm”, see Table 1.c) is at full footprint resolution.  The program BUFR_SUPERTMI unpacks each report checking the validity of the satellite id, observation date and total precipitation observation.  Reports passing checks are then superobed onto a one-degree lat/lon grid according to satellite id and encoded into the output, re-processed BUFR file.  The output filename contains the qualifier “sptrmm” (see Table 1.c, key for superscript 1 in “NET” column).  The Global GSI analysis (GFS and GDAS network runs only) reads the superobed data directly from the reprocessed "sptrmm" BUFR dump file (these data do not pass through the PREPBUFR processing steps). 

===> Dump Job 2, running simultaneously with Dump Job 1, performs the following single step:

Dumping of WSR-88D Level II radial wind and reflectivity BUFR Data

This currently runs in only the NAM and NDAS networks.  The processing is identical to that described in Dump Job 1, Step B above.  The dumping of WSR-88D Level II radial wind and reflectivity data is performed in a separate job from the dumping of all other data in the NAM and NDAS networks in order to save computation time since it takes almost as long to dump Level II data here as it takes to dump all other observational data in Dump Job 1.