Data Processing at NCEP
Dennis Keyser - NOAA/NWS/NCEP/EMC
(Last Revised 12/11/2009)
Please take a moment to read the Disclaimer for this non-operational web page.
Most of the observational data at NCEP are stored in WMO BUFR format. This format is an international standard and provides an efficient means for transferring data. In addition it allows for great flexibility for adding new observation elements.
There are a series of tables associated with BUFR. “Table A” defines the data category associated with a particular BUFR message containing report data. “Table B” classifies and defines data elements, or descriptors, according to scale, reference value, number of bits and units. “Table C” defines data description operators. “Table D” defines the list of common sequences. In addition, there are BUFR code and flag tables as well as code tables common to BUFR and other binary and alphanumeric codes. The need for external tables can make the process for BUFR data encoding and decoding quite cumbersome for a typical user.
As such, a special application has been designed at NCEP which provides user-friendly access to the BUFR files through a series of FORTRAN and C subroutines in a machine independent BUFR library (called BUFRLIB). These routines allow one to encode or decode data into BUFR using mnemonics to represent the data. The mnemonics are associated with BUFR descriptors in a special version of the Tables A, B, C and D. When a BUFR file is created, the mnemonic table is read in from an external location and is itself encoded into BUFR messages at the top of the output file. These messages have Table A data category (message type) 11 (BUFR tables). This allows each BUFR file to be “self defined”. No external tables are needed to decode data out of the file.
NCEP Central Operations has written a BUFRLIB software user guide which provides a detailed explanation of the NCEP BUFRLIB subroutines along with other useful information on BUFR as it is used at NCEP. Click here to view a sample Fortran code which uses BUFRLIB routines to encode reports into a BUFR file.
Next is a brief outline on the current method for processing
that arrive at NCEP. Its main function is to provide links to web pages
which discuss each item in detail.
here to see a flow chart depicting the data procesing system
1. NCEP receives the majority of its data from the Global Telecommunications System (GTS) and the National Environmental Satellite, Data, and Information Service (NESDIS).
a. The GTS and aviation circuit bulletins are transferred from the NWS Telecommunication Operations Center (TOC/NWSTG) to NCEP's Central Operations (NCO) and networked to one of two interactive nodes on the NCEP IBM Central Computer System (IBM-CCS) (one production, one development) using LDM (Local Data Manager) and DBNet (Distributive Brokered Networking) software packages. These data are then decoded from their native format and encoded into WMO BUFR format using decoder software designed by NCO's System's Integration Branch.
b. Most of the satellite data are processed in batch mode as they become available from the various NESDIS servers. Regularly scheduled jobs on the IBM-CCS transfer "new" files from these servers and encode the data into WMO BUFR format. See http://www.emc.ncep.noaa.gov/data_processing/satellite_ingest.doc/document.htm for more information.
c. NCEP receives "Level 3" radial wind data from 158 NEXRAD radar stations via the radar multicast (NIDS format). At eight minutes past each hour, a script runs to gather up the most recent radar data into eight pairs of files containing the raw data and a station list. At thirty-five minutes past each hour these raw data are superobed and encoded into BUFR format.
d. NCEP receives on-site superobed Level 2 (which we call "Level 2.5") radial wind data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Open Systems Radar Product Generator (ORPG) stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Twice each hour at five and thirty-five minutes past the hour, all newly arrived files are first uncompressed and then encoded into WMO BUFR format.
e. NCEP receives the full complement of Level 2 radial wind and reflectivity data from 158 NEXRAD radar stations via the NOAA/Radar Operations Center (ROC) Radar Data Acquisition stream from the NWS Telecommunication Operations Center (TOC/NWSTG) . Raw data files arriving at NCEP are placed in a holding area. Four times each hour at ten, twenty-five, forty and fifty-five minutes past the hour, all newly arrived files are first uncompressed, then quality controlled (both radial wind and reflectivity) and then encoded into WMO BUFR format.
All of the encoded BUFR data are then appended to the
in the data base. The files are organized by the WMO BUFR
local subtype and contain information in 24 hour blocks (based on
time). Observational files remain on-line for up to 10 days
migration to off line cartridges. This allows late arriving
to be accumulated. While on line, there is open access to them
only for operations, but also for research and study.
2. The various NCEP networks access the observational data
base at a
set time each day (i.e., the data cutoff time) and perform a
dump of requested observations. Observations of a similar
satellite-derived winds ("satwnd"), surface land reports ("adpsfc")]
dumped into individual BUFR files which maintain the original structure
of reports, although some interactive quality control is
duplicate reports are removed, and upper-air report "parts" are
for more information.
3. The final step in preparing most of the observational data
by the analysis involves the execution of a series of programs which
in the observations from the various dump files, add forecast ("first
background and observation error information, perform automated quality
control, and finally output the observations in a monolithic BUFR file
known as "PREPBUFR". See http://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/document.htm
for more information.
4. The PREPBUFR file is read by the Global
Statistical Interpolation (GSI) analysis which runs in the Global
System (GFS) and Global Data Assimilation System (GDAS), the
which runs in the regional North American Model (NAM) and North
Data Assimilation System (NDAS) which currently use the WRF-NMM model,
the 3DVAR which
in the Rapid
Cycle (RUC) system.
Here are some other links to my web sites devoted to data processing:BUFR Table D Descriptors and Mnemonics in NCEP Observational Database
Satellite Historical Documentation
Non-Satellite Data Historical Documentation
Format of Tropical Cyclone Vital Statistics Records (“tcvitals”)
NCEP BUFR File Structure [with emphasis on embedded BUFR table (A, B, D) messages]