Q&A about NCEP Stage II/Stage IV


For both Stage II/Stage IV:


For Stage II:


For Stage IV:


Q&A For both Stage II/Stage IV


What do the names (Stage I/II/III/IV, RMPA) mean?

The nomenclature is somewhat confusing. Here is an attempt to clarify:

Stage 1/Stage I: Digital Precipitation Arrays (DPAs) derived from radar reflectivities, on 4km 131x131 polarstereographic grids centered on individual radar sites.

Stage 2/Stage II: merged radar+gauges product on the local 131x131 grid (done at each of the 12 CONUS RFCs for Stage 3 in the past, and at NCEP for the 'NCEP Stage 2'). Also used to refer to the NCEP product after we mosaic the 'local (131x131) Stage 2' into a national product.

Stage 3/Stage III/RMPA: In the past, "Stage III" refers to 1-hour and 6-hour analyses from the RFCs that were mosaicked (non-NCEP) Stage 2 over each RFC domain, with manual QC. Currently the RFCs have transitioned to other algorithms - MPE (multisensor precipitation estimator) for most RFCs, Mountain Mapper for the Western RFCs and P1 for ABRFC. We shall hereby refer to the regional 1-hour/6-hour analyses we receive from the RFCs as the RMPAs (for Regional Multi-Sensor Precipitation Analysis).

Stage 4/Stage IV: RMPAs mosaicked/quilted into a national product at NCEP

In the past (prior to 2000 or so) we sometimes used the names Stage II/Stage IV interchangably to refer to the NCEP hourly multi-sensor precipitation analysis. Since the RFCs started routinely producing/transmitting the manually QC'd RMPAs over their respective domains, and NCEP has started to mosaick/quilt the MPE/Stage IIIs into a national 'Stage IV' product, we are now calling the non-QC'd multi-sensor product generated at NCEP (directly from the radar and gauge data, as opposed to the mosaic of the RMPAs) the 'NCEP Stage II'. For more information regarding the MPE algorithm/products at the RFCs, please see a presentation by Rich Fulton of NWS/OHD: http://www.nws.noaa.gov/oh/hrl/papers/wsr88d/tac_nov02.pdf. Additional information about the MPE and WSR-88D rainfall algorithm can be found in http://www.nws.noaa.gov/oh/hrl/papers/papers.htm#wsr88d.

Return to the Q&A index


What is the grid specification?

For the 4km grid (applicable to Stage IV1 and Stage II archived since 1 Jan 20022), the NWP HRAP (Hydrologic Rainfall Analysis Project) grid:

For the 15km grid (applicable to current Stage II):

All the grid lat/lon information is considered valid at the center of the grid box. If you compare the above specifications for the 4km grid against that given for the Grid 240 (the HRAP grid) in the NCEP GRIB manual, you might have noticed that the lat/lon coordinates for the corner points and the pole point seem to be shifted by half of a grid space. This is because the hydrologists who first created the HRAP grid (the national HRAP grid defined as above as well as the local HRAP grids the RFCs use for their Stage III/MPE data) defined a grid's coordinate using the 'lower-left' corner of the grid box, while meteorologists usually define a grid box's coordinate using the center of the box. In early 2000, when Brett MacDonald (then at NCEP/HPC) first plotted the regional analyses from the RFCs using GEMPAK (which considers a grid box's coordinate as the center of the box), he noticed that all the precipitation plots seemed to be 1/2-grid off. So he and I decided that, when mosaicking the local MPE products into the national Stage IV analysis, we should receive the MPE data as they are, mosaic them into Stage IV on the national HRAP grid, but provide our own grid definition for the HRAP for the atmospheric people (i.e. it is the same physical grid, but we define the grid coordinates using the center of the grid box, rather than the lower-left corner).

1The grid for Stage IV data from 1 Jan 2002 to 10 May 2004 was on a slightly shifted grid - the (1,1) point was mistakenly set to (23.117N,119.017W). Please see this page for further details.

2For the old 4km grid (applicable to the archived Stage II analysis prior to 1 Jan 2002), please see the old vs. new Stage II analyses comparison page.

For a more detailed discussion of the HRAP grid, please see the NWS/OHD documentation at http://www.nws.noaa.gov/oh/hrl/nwsrfs/users_manual/part2/_pdf/21hrapgrid.pdf.

Return to the Q&A index


What is the unit of the precipitation accumulation?

The unit is kg/m**2, which is equivalent to depth in mm.

Return to the Q&A index


How to find the lat/lon location of a grid point?

Two subroutines, w3fb06.f and w3fb07.f, can be used to find the nearest grid-coordinate for a given lat/lon location, and to find the lat/lon of a given grid point, respectively. Here are two examples of how to use the subroutines (please read the DISCLAIMER):

To find the grid coordinate of a location given by (rlat,rlon):
    call w3fb06(rlat,rlon,alat1,alon1,dx,alonv,xi,xj)
    where    dx=4762.5
             alonv=255.
    For the new (official NWP) HRAP grid (applicable to Stage IV analysis 
      since 11 May 2004, and Stage II analysis after 1 Jan 2002):
             alat1=23.117
             alon1=240.977 (=360.-119.023)
    For Stage IV analysis between 1 Jan 2002 - 10 May 2004, where the grid
      was slightly shifted:
             alat1=23.117
             alon1=240.983 (=360.-119.017)
    For the old HRAP grid (Stage II analysis archive, 1 May 1996 - 31 Dec 2001):
             alat1=22.7736
             alon1=239.624 (=360.-120.376)
    The return argument, (xi,xj), gives the location of point (rlat,rlon) on
    the 4km HRAP grid. (nint(xi),nint(xj)) is the grid point closest to
    (rlat,rlon).

To find the lat/lon location of a grid point (i,j):
    xi=float(i)
    xj=float(j)
    call w3fb07(xi,xj,alat1,alon1,dx,alonv,alat,alon)
    (alat,alon) is the lat/lon of the grid point (i,j)
For more information, please see the documentation block of these two subroutines.

Return to the Q&A index


Is there a land/sea mask for the HRAP grid?

Beth Ebert of the Bureau of Meteorology Research Centre kindly provided two land-sea masks on the HRAP grid created using IDL. The hi-res version has more spatial detail and the lo-res version has less. These are simple text files with 0's for water and 1's for land. The grid dimensions are 1121x881 and the grid origin is in the southwest.

Return to the Q&A index


How do I decode these files?

The format of the files is GRIB1. The files are compressed using the UNIX "compress" command and "uncompress" must be used before decoding (this refers to the Stage II/IV files. If you are referred to this section from some other EMC pages, then the file your are trying to decode might not have been compressed).

We regret that due to severe resource limitations, EMC cannot provide individual support on decoding these GRIB files or on GRIB decoding software. Please consult technical support department in your organization about GRIB/GRIB decoding. If you do not have access to other, non-NCEP decoders, you might want to check out the list below. Contact information for these software are listed in their web sites, but it seems that these are all more or less DIY programs, with the expectation that the user will be able to figure out answers to any questions through inspection of the documentation and the code.

1) wgrib, an excellent GRIB decoding, inventory, and manipulation package that is easy to compile on nearly any platform

2) NCEP GRIB Documentation and software (for GRIB enthusiasts who require no technical support)

Return to the Q&A index


How do I plot these grids?

There are numerous grid plotting packages, many of which are "GRIB-friendly". The following are suggestions on how to plot with GEMPAK, NCARGRAPHICS, and GRADS. Caveat: the suggestions are provided only for information purposes. A user will have to make any necessary changes to the code, scripts etc. and resolve any questions on his own, without assistance from NCEP.

For GEMPAK:

You will need to run nagrib, version 5.3 or higher. The 4km grid is over 1,000,000 points, so in order to use nagrib on these grids, a recompile is necessary to expand the size of the maximum allowable grid. The 15km grid, available for Stage II analyses for the past 24h or so, is about 110,000 points, and should be decodable with nagrib.

You will need to subset the 15km grids so that other GEMPAK programs (GDPLOT, etc.) will be able to handle them. In nagrib:

where GAREA is the location of the subset you are interesting in, it can be nearly as large as the entire grid, but must be less than 100,000 points.

For NCARGRAPHICS:

Mike Baldwin (the original developer of the NCEP Stage II) had made up a sample decoding and plotting code using NCARGRAPHICS and W3LIB GRIB decoding routines. It is available for downloading here. This code requires the W3LIB GRIB decoding routines described earlier. This code will plot the 4km and 15km grids and produces a map similar to those found on the Stage II/IV web pages.

For GrADS:

Refer to these instructions, provided by Jamie Kousky of OM. Note that in this set of instructions the grid specs in the *.ctl file is for the old HRAP grid, and for the data on the new HRAP grid you'll want to modify the array dimension, pole point etc. for the *.ctl.

Return to the Q&A index


Is there an archive? Where should I go to get the data?

The Past 10 days' data are placed on line by NCEP Central Operations as soon as the Stage II/Stage IV jobs are done. This is the quickest, most reliable place to get the current data. Starting from 3 Dec 2013, the retention time of these data files are increased from 1-2 days to 10 days.

The daily tar files are provided on a space-available bases. Caveat: if we find we are running low on space, these files might be deleted with no advance notice. Location and format of the tar files might be changed with no advance notice. Tar files might be missing or empty due to machine or connection problems, and missing tar files will not be available from NCEP. The daily tar file archive was first set up years ago when the NCAR Archive was still new and not as easy to use as today.

The NCAR CODIAC is the best place to get archived data. Here are the links to the data:

Note that each month's data appear in the NCAR archive around mid-month the next month , so the lag time for the archive can be up to 45 days or so. Since the availability of the daily tar files for the past few months from the NCEP ftp site (see above) is not guaranteed and missing files will not be available from NCEP, if you think you might need the data before they become available at the NCAR archive, your best bet is to download them each day from the past 10 days' data site.

Images at NCAR CODIAC:

NCEP users with access to the NCEP CCS and HPSS:
Past 5 days' data at CCS:/com/hourly/prod/nam_pcpn_anal.yyyymmdd
HPSS tape archive: /NCEPPROD/hpssprod/runhistory/rh${yyyy}/$yyyymm/$yyyymmdd/com_hourly_prod_nam_pcpn_anal.$yyyymmdd.tar

Please note that NCEP has very little resource to provide user support for the Stage II/IV product. If a product is not available on line, users will have to rely on the NCAR CODIAC archive for the data. Please send technical questions/issues with the CODIAC archive at codiac@ucar.edu. If you wish to obtain many months of the data all at once, I hear that CODIAC has a way to give you the data other than from the above link. Please contact codiac@ucar.edu for details.

Return to the Q&A index


How is the observation time reflected in the file name?

The yyyymmddhh in the file name is the ending hour of the accumulation period. If yyyymmddhh is followed by xxh, then you are looking at xxh accumulation ending at yyyymmddhh.

For the Stage II hourly analysis files, only the multi-hour (6h, 24h) analyses contain the indicator xxh.

The time stamp in the grib headings of the 6-hourly Stage IV analysis follows that used in the 6-hourly regional analyses produced by the River Forecast Centers (RFCs). The 6-hourly Stage IV analysis is the mosaic of 6-hourly MPE analyses from the RFCs (the RFCs perform 6-hourly analyses separately from their hourly analyses, consequently the 6-hourly Stage IV is not simply the sum of 6 consecutive hourly analyses). The 24h-hour Stage IV totals are the sum of four 6-hourly Stage IV analyses. For their 6-hourly analyses, the RFCs use the previous 12Z as the "reference" time in the GRIB heading (grib PDS octets 13-17), and the accumulation beginning/ending times are given in octets 19-20. When we mosaic the 6-hourly MPEs into the 6-hourly Stage IV, we follow this convention, (and when we add up the 6-hourlies into 24h totals, the time stamp is similar to that in the 6-hourlies). Because of this, the time stamps in the 6-hourly/24h-hourly GRIB headers look like that of forecasts rather than that of analyses, though we do give the resulting Stage IV files names more consistent with its analysis nature, with the accumulating time period and ending hour of accumulation. Here are how things go:

    Stage IV             Actual accum        Oct 13-16      Oct 19   Oct 20
    file name            ending time       ref yymmddhh      P1       P2
    ST4.2002032618.06h   2002032618          02032612         0        6
    ST4.2002032700.06h   2002032700          02032612         6       12
    ST4.2002032706.06h   2002032706          02032612        12       18
    ST4.2002032712.06h   2002032712          02032612        18       24
    ST4.2002032712.24h   2002032712          02032612        00       24
Note: analyses ending at 00Z (ST4.2002032700.01h, ST4.2002032700.06h etc.) are placed in that day's (i.e. 20020327) directory/tar file, even though their accumulation periods fell into the previous day (20020326).

All time in the Stage II/IV is UTC.

Return to the Q&A index


What causes the occassional "reflectivity rings" in the analyses?

Tim O'Bannon of the WSR-88D Radar Operations Center explained that this is an automatic gain control (AGC) problem that occasionally show up at some radar sites, and the crew at the WSR-88D Operations Center works with the sites to fix the problems as they arise. An example of this is in the 1-hour NCEP Stage II and Stage IV analysis ending at 2002032106, which is caused by the 200203210600 digital precipitation array (DPA) from TLH (Tallahassee, FL). We see this problem more often with the NCEP Stage II's than the Stage IV's, because at the RFCs where the regional MPE analyses are generated, an analyst is likely to notice the "ring" in the earlier, automated version of the analysis and re-run the later, QC'd version of the same analysis without the problem radar. 2002032106 is a rare example of the "reflectivity ring" showing up in the Stage IV hourly analysis. For this hour, we received only the automated version of the MPE analysis from SERFC. We did not receive the later, manually QC'd version from SERFC for this hour, but the problem was evidently noticed and corrected by SERFC during the QC, since the ring did not show up in the subsequent 6-hourly MPE analysis we received from them. We see the "reflectivity rings" more often in the NCEP Stage II since it does not have manual QC.

Return to the Q&A index


Which should I use, Stage II or Stage IV?

Since Stage IV benefits from manual QC at the local RFCs, it is generally better than Stage II, which has no manual QC. There are a number of other issues for your consideration:

1. Coverage period: Stage II began to be archived at NCAR on 1 May 1996. Stage IV did not began to be archived until 1 Jan 2002.

2. Do you need hourly or 6-hourly analysis? For 6-hourlies, Stage IV is the one to use (if you don't need the pre-2002 data). Stage IV hourlies have some coverage problem since we do not receive the local analyses from all the RFCs for all hours.

3. How soon do you need the analysis to be available? Stage II hourly analysis is made at 35 minutes after the top of the hour, then re-made twice at 6 hours and 18 hours later. Check out timeliness for Stage IV here.

4. If I choose Stage II, which type should I use? We recommend the multi-sensor analysis with the exception of early spring - early Aug 2000, when the radar coverage was bad. You might want to use the gauge-only analysis for that time period.

Return to the Q&A index


How were the data files compressed?

Data files up until 24 Jul 2013 have a suffix of ".Z" ('compress/uncompress'). Starting 25 Jul 2013, we transitioned to a new computer system and 'compress/uncompress' are no longer available, and the data files are gzip'd, with a suffix of ".gz". Both *.Z and *.gz files can be uncompressed using 'gunzip'.

Return to the Q&A index


Why is this file not called FAQ?

I do not know if any of these questions would be asked frequently. I thought about calling this 'F/IFAQ', but settled on Q&A.

Return to the Q&A index


Why are there missing/incomplete/unreadable hours? Can they be re-made?

Missing hours and hours with incomplete data coverage are usually caused by outage of input data streams (radar/gauge data for Stage II, regional analysis for Stage IV). Files that cannot be uncompressed or read might be the result of data transmission error somewhere along the way from production machine to workstation, then to NCAR CODIAC. Due to resource limitations (see the next item in the Q&A) we cannot go back and re-process a problem hour far in the past. In real time, when we encounter a problem, we try to resolve it quickly (by notifying agencies supplying the data, or by making a quick fix/reroute on our end), then try to salvage as much "recent" data as we can; after that we have to move on.

I have started lists of problem hours for Stage II and for Stage IV. If you encounter a problem hour that is not listed, please contact me at Ying[dot]Lin[at]noaa[dot]gov.

Return to the Q&A index


Why can't I get more user support?

Lack of resources - the development/maintenance/user support for the entire Stage II/Stage IV operation is supposed to be done by a fraction (about 25%) of one person's time. We became a de facto data provider accidentally - the Stage II/IV analyses were originally produced for our own data assimilation use in our model, so it was a side project for one person who was also doing a number of other things. Over time the Stage II/IV are used by many more people, but there has been no additional time allocated to their support. Given this time constraint, the only way to sustain the product and to best serve NCEP and outside users is to have most user questions answered by on-line documentation such as this Q&A file and to severely limit hands-on individual user support. This means, for example, that users are asked to obtain the data and decode the files by themselves, or obtain assistance from co-workers or from technical support department in their own organizations. We apologize for this inconvience, but we really do not have an alternative to keep this going.

Return to the Q&A index



Q&A For Stage II


What is the NCEP Stage II?

NCEP 'Stage II' refers to the real-time, hourly, multi-sensor National Precipitation Analysis (NPA) developed at the National Centers for Environmental Prediction (NCEP) in cooperation with the Office of Hydrology (OH). This analysis merges two data sources that are currently being collected in real-time by OH and NCEP. Approximately 3000 automated, hourly raingage observations are available over the contiguous 48 states from ASOS and via the GOES Data Collection Platform (DCP). In addition, hourly radar estimates of precipitation ('DPA', for Digital Precipitation Arrays) are obtained as compressed digital files via the ASOS network. The DPA estimates are created by the WSR-88D Radar Product Generator on a 131 x 131 4-km grid centered over each radar site. The data analysis routines, including a bias adjustment of the radar estimates using the gage data, have been adapted by NCEP on a national 4-km grid from algorithms developed by OH ("Stage II") and executed regionally at NWS River Forecast Centers (RFC).

Return to the Q&A index


How are the radar-only fields derived?

The radar-only product (ST2rd.* and ST2un.*) consists of approximately 140 WSR-88D radars which report to NCEP in real-time via ASOS. Each individual radar estimate is merged together on the national Hydrologic Rainfall Analysis Project (HRAP) grid and bins which contain more than one radar estimate are averaged together using a simple inverse-distance weighted average. Currently, there is no quality control of the radar estimates, such as removal of anomalous propagation.

There are current 2 types of radar-only analyses, one based on the raw radar (DPA) data (ST2rd.*) and the other is gage-adjusted (ST2un.*). The radar bias adjustment algorithm follows Smith and Krajewski (1991). These gage-adjusted radar estimates are available after an ~6h delay, to allow gage data necessary to estimate the biases to arrive.

Return to the Q&A index


How are the gage-only and multi-sensor fields derived?

In contrast to the simple radar-only mosaic technique, the analysis schemes used in the gage-only and multi-sensor analyses utilize optimal estimation theory. These were developed by Seo (1998). The schemes are fundamentally similar, and optimally estimate rainfall fields using raingage and radar data under partial data coverage conditions. This is preferred over previous statistically-based techniques because it takes into account the variability due to fractional coverage of rainfall, as well as within-storm variability. By objectively taking the spatial coverage into account, more accurate estimates of the rain versus no-rain area are obtained. Accurate delineation of this area is as important as accurate estimation of rainfall within the rain area. One of the underlying assumptions in the radar-gage analysis scheme is that the radar estimates are unbiased. Currently, radar biases are adjusted prior to the multi-sensor analysis by the technique developed by Smith and Krajewski (1991). This method attempts to remove the mean bias but does not attempt to remove range-dependent biases.

The 24h "rfc" analysis uses the same gage-only algorithm as the hourly analyses, except with the RFC 24h gage database. There are many more stations (7-8,000) than the hourly data.

Return to the Q&A index


How is the grib bitmap used to denote missing regions?

Each type of analysis uses the GRIB bitmap feature to denote the area of the domain that has enough data to provide an analyzed value. In other words, the bitmap tells you if you are inside the data domain or not. In the W3FI63 unpacker, a logical array is returned that is .TRUE. where the bitmap is turned on, and .FALSE. where the bitmap is turned off. The gribplot.f code takes advantage of this feature to denote the extent of the data domain. The gage-only analysis is considered to be inside the data domain within approximately 50km of the nearest gage report. The radar-only analysis is considered inside of the data domain within approximately 200km of each successfully decoded radar report. The multi-sensor analysis will use the gage-only value if no radar data are available, and the radar-only value if no gage data are available.

Return to the Q&A index


What kind of quality control steps are taken?

At the present time, the NCEP Stage II does not have manual quality control steps that are a hallmark of the RFC MPE analyses. However, some initial quality control steps have been implemented into the current Stage II system. There is a gross error check on the gage data so that no gage report of more than 1"/hr gets into the analysis. Also, a list of consistently bad raingages is in use, so that observations will be ignored from these reporting stations in the analyses. This list is subjectively determined by examination of numerous cases where the gages reported heavy rainfall for several hours while nearby gage and radar reports contained zero rainfall.

NCEP is working with FSL and OH to develop an automated quality control of the hourly gauges. An overview can be found here.

Return to the Q&A index


Why the new grid?

The official NWS HRAP (Hydrologic Rainfall Analysis Project) was created in the 1970's. Information dissemination was not easy back then (no internet), the exact specifications of the grid was not widely known. In 1996, when the National Stage II Analysis was first developed at NCEP (in cooperation with Office of Hydrology), Mike Baldwin created a grid (the HRAP grid that had been used in the NCEP Stage II analysis during 1996-2001) based on what scant information that he could find about the HRAP grid. The exact HRAP grid specification became known to us in 2000, and we decided to switch.

Return to the Q&A index


Why the new file names?

The first reason: there were two different sets of old file names. Take the old 4km multi-sensor files. In the 'current and one-week archive' files kept at the time, they were called "USAmlmmddyyhh.Grb.Z" (this odd date stamp format was hardwired in the old software), and in the long term archive, they were called "mul4.yyyymmdd.hh..Z" (we used more sensible date stamps for the long term archive). Users found both the mmddyyhh date stamp and the different names in the long term archive confusing.
The second reason: since we have changed the grid, we decided that it is better to have totally different file names in case users don't catch this change in time (we figured if we use the same file name, there is a chance that someone would not be aware of the grid change, and run something out for 6 months with the wrong grid information).

Return to the Q&A index


Why did the multi-sensor data look so bad during spring-early Aug 2000?

As a part of the NWS conversion from AFOS to AWIPS, During Sep 1999 - Aug 2000, the radar precipitation products from the 137 radar sites over CONUS gradually changed from the HDP (Hourly Digital Precipitation) to the DPA (Digital Precipitation Array) format. NCEP had been receiving only the HDP data until Jul 2000, and the decoding of the DPA data did not start until 11 Aug 2000 (i.e. until then the multi-sensor analysis had been depending on an ever-dwindling number of radar sites that had yet to be converted from HDP to DPA). During Sep-Dec 1999, only 8 radar sites underwent the conversion. By Feb 2000, the conversion was in earnest, so during the next month, the reduced radar coverage became quite noticeable in the multi-sensor product, and by that summer, the radar coverage was reduced to practically nothing, until we switched to decoding the DPA on 11 Aug 2000.

We would recommend to the user who is planning on choosing a time period for an experiment that involves the NCEP Stage II analysis to avoid the period between 1 Mar - 10 Aug 2000 (February could be problematic too, the user would need to look at the snapshots of the data and decide on his own).

If a user must use data from this period, he might consider using the gauge-only analyses instead of the multi-sensor. The gauge-only analyses were, of course, not affected by the decrease in radar coverage. One of the downside of the gauge-only data (besides not being multi-sensor) is that the analyses are usually 'holey' (in contrast, during normal conditions, the radar analysis over CONUS have relatively few non-covered areas).

Date of conversion for each WSR-88D site
(courtesy of Fred Branski)

Bar Chart for the Monthly Conversions

Snapshots of the Hourly Analyses
yyyymmddhh Multi-sensor Gauge-only
1999091112 X X
1999101112 X X
1999111112 X X
1999121112 X X
2000011112 X X
2000021112 X X
2000031112 X X
2000041112 X X
2000051112 X X
2000061112 X X
2000071112 X X
2000081112 X X

Return to the Q&A index


Are the Stage II files on the 15km grid saved in the archive? How do I map the data from the 4km grid to the 15km grid?

The Stage II files mapped to the coarser 15km grid are not part of the archive at NCAR. For the pre-2002 Stage II, it is not saved in the 6-month on-line archive either. Starting from 1 Jan 2002, the 15km Stage II files are included in the near-term archive (6 months or less, space permitting), but it will not be part of of the permanent archive at NCAR.

If only the 4km-grid data are available to you and you would like the data on the coarser, 15km grid, you will need to run a re-mapping program. This is done by simply averaging the values of the 15km grid's nearest 3x3 neighbor's on the 4km grid (only one of the 3x3 points on the 4km grid is required to be non-missing to create a 15km grid point value). The (1,1) point on the 15km grid is at the same location as the (2,2) point on the 4km grid. You can use our remapping program as a template. The 4km analysis file is read in from unit 11, and the 15km analysis written out to unit 51. Note that if you are dealing with the pre-2002, analysis on the old HRAP grid, you need to change the parameters IMR, JMR, alat1, alon1 to those of the the old HRAP grid values that can be found here.

If it is not easy for you to write out data in GRIB format, you can just read in the 4km data, then do the simple 3x3 remapping as above.

Return to the Q&A index


Why were there streaks in the ST2rd/rad4 files prior to 20 Nov 2003?

This was caused by a bug in the way precipitation estimates under individual radar umbrellas were mosaicked into a national product. For the 'radar-only' product, when a given grid point is covered by more than one radar, a weighted sum is used as the precipitation estimate at this point. The inverse of the distance from this grid point to a radar is used to weigh the contribution from this radar. When this grid point is right at the radar site (i.e. if both delta(x) and delta(y) are zero), the weight is capped at a maximum value of 1. Unfortunately, a bug in the code set the weight to 1 if either delta(x) and delta(y) were zero.

This bug affects the 'radar-only' analysis, i.e. the ST2rd files during 1 Jan 2002 - 19 Nov 2003, and the rad4 files prior to 1 Jan 2002.

Here is a snapshot of an ST2rd hourly accumulation, with the bug in the code and Here is a snapshot of an ST2rd hourly accumulation, with corrected code.

The fix of this problem was implemented operationally on 20 Nov 2003.

Return to the Q&A index


For Stage IV:

How timely are the Stage IV analyses?

The NCEP Stage IV program is run each hour at approximately 35 minutes past the top of the hour. At that time, we check the current day's and the previous day's RMPA files for any new RMPA that arrived in the past hour, for hourly and 6-hourly periods within the current day and the previous day. If there are data arriving from at least one RFC, we perform (or re-do) a Stage IV mosaicking for the hour/6-hour period(s) the new data cover. So if you look at an analysis for, say, 2 hours ago, you might find that only one RFC contributed to it. If you keep checking, once an hour, you might find that hour/6-hour's analysis gradually gets 'filled in' by contributions from other RFCs.

For the hourly analysis, an RFC typically performs the MPE program twice. The first is run in automated mode, done shortly after the top of the hour with no manual quality control. That hour's analysis is made again in manual mode, where a human analyst performs the manual QC.

Due to resource limitations, an RFC sends NCEP around 4-6 'bundles' of analyses each day. A 'bundle' might contain several manually QC'd analyses for earlier hours and several automated (no QC) analyses for more recent hours.

The hourly MPE receipt time ranges (roughly) from within an hour to 12 hours after.

The four 6-hourly analyses covering the previous 12Z-12Z are generally received by 15Z (for the automated runs) and 21Z (the manually QC'd runs).

The 24-hour analysis (12Z-12Z) is summed from the four 6-hourly analysis. This is done initially at 15:35Z, then redone twice at 21:35Z and 23:35Z.

A One-day snapshot of data flow from MBRFC

How far back do we go to re-do the mosaicking: the regional multi-sensor precipitation analysis files (RMPAs) received from the RFCs are kept in one file per RFC per day on NCEP machines (new arrivals are concatenated to the appropriate file). Each file for day contain the hourly accumulations ending at 00, 01, ..., 23Z, and the 6-hourly accumulations covering 12Z day to 12Z day+1 (for the particularities in the 6-hourly analysis's time stamp convention, see this link). We currently go back one day, to check for new arrivals from the RFCs and re-do the analyses when we see new arrivals for an hourly or a 6-hourly period. Which means that, the last time hourly and analysis ending at 00Z, 01Z, ..., 23Z for day, and 6-hourly analysis falling between 12Z day and 12Z day+1, might be re-mosaicked is at the 23:35Z run on day+1. In the future we might go back more than one day to check for new updates from the RFCs. In the future we might go back 7-10 days, though it might not be possible to go back that far in every hour's Stage IV run, it might take too long for each hour's Stage IV run to finish in time. We are considering doing the 7- or 10-day check once a day.

We maintain two daily logs that keep track of the timing of the incoming MPE data:

Logs that shows incoming MPE data stream throughout the day (time stamp is UTC).

Logs that summarizes arrival times for each hourly/6-hourly analysis from each RFC (time stamp is UTC after 28 Mar 2002, ET - EST/EDT depending on the season - before 25 Mar 2002. Between 25-28 Mar there could be a mixture of both):

Return to the Q&A index


Why were the 1-hour analyses ending at 00Z so often missing along the SE coast between mid-May to mid-Sept 2002?

Short answer: a script glitch prevented many of the SERFC input for that hour to be used in the national mosaic.

Long answer: on 8 May 2002, we started running the Stage IV processing at 20 minutes after each hour (it was run less frequently before that). SERFC sends out a batch of data, including the 1-hour accumulation ending at 00Z, at 0520Z (analysis for this hour is sent only once by SERFC - see the daily log page for the time line). The Stage IV job script checked the current day and the previous day's incoming data for any new analyses from the RFCs. Any new analyses from the RFCs were split out and copied into subdirectories $day/$rfcid (which was created during the processing for the current day ($day), when new data from an RFC ($rfcid) were detected in the data 'tank'), the current day was processed first and then the previous day was processed. What I forgot was, in the incoming data 'tank', the 1-hour accumulation ending at 00Z was placed in 'previous day' (they sort data by the starting time of the accumulation period, while I do so by the ending time). When I ran the processing at 0520Z, around the same time when SERFC transmitted their batch of data, when I ran the 'current day', the SERFC batch hadn't come in yet, so no $day/$rfcid for SERFC was made. When I proceeded to run the 'previous day', the SERFC batch had arrived, my job detected the new 00Z SERFC analysis (hence you see it in the daily log files), extracted it and tried to place it in subdirectory $day/$rfcid - but $day/$rfcid didn't exist yet.

At the time of this writing (19 Sept 2002), the Stage IV processing was about to becoming operational. The soon-to-be operational scripts do not use the aforementioned subdirectory set up and are not affected by this glitch.

This problem affected the 00Z Stage IV hourly analysis produced between mid-May to 15 Sept 2002. Not all 00Z analyses during this time period were affected though - if the timing of the 0520Z SERFC transmission or my 0520Z Stage IV run was a little off, then the window for the glitch was closed and there'd be input from SERFC for the 00Z Stage IV analysis. A bug fix was put in on 16 Sept. The 00Z 16 Sept analysis, made before the bug fix, did contain input from SERFC - apparently the batch of data from SERFC arrived seconds later (or my job was run seconds earlier) than usual, and they were all processed an hour later, at 0620Z, so the problem didn't show up.

Return to the Q&A index


How is the mosaicking done?

The mosaicking for each hour/6-hour period is done via the following steps:

  1. Gather all regional analyses available for that accumulation period; map the regional grid values to the national HRAP grid
  2. If a point on the national HRAP grid falls into one of the RFC's domain, then the value from that RFC's analysis is used for this point
  3. If a point on the national HRAP grid is not in any of the RFC's domain, but is covered by one or more RFC's analysis grid(s) (an RFC's analysis grid is a regional polar stereographic HRAP grid that encompasses the RFC's domain), then the average value of all regional analyses covering this point is used as the mosaic value.
  4. If a point on the national HRAP grid falls into one of the RFC's domain, but analysis for that RFC is not available, then the average value of all neighboring RFCs' analyses for this point, if available, is used as mosaic value.
We are re-assessing the No. 3 and 4 above, since we have learned that RFCs generally do not pay as much attention to areas outside of its domain proper (though within their analysis grid) and it might be better to leave an RFC's domain blank if data have not yet come in from that RFC. We plan to make this change in the future.

Return to the Q&A index


How is the 24h Stage IV summed?

The 24h (12Z-12Z) Stage IV is summed from the four 6-hourly Stage IV analyses covering the 24h period.

Return to the Q&A index


Why do I sometimes see discrepancies between hourly and 6-hourly analyses?

The Stage IV 6-hourlies are not simple summations of the hourlies. The RFCs send us both hourly and 6-hourly analyses for their local domains, and typically 6-hourlies have better QC. Some of the hourlies we receive are only the automated runs (no QC), while for the 6-hourlies we almost always receive the QC'd version. NWRFC does not produce an hourly analysis, only 6-hourlies. In addition, sometimes we do not receive an hourly analysis for an RFC (in that case the neighboring RFCs' analysis contribute to the Stage IV values in this RFC's domain - see How is the mosaicking done?), which is another contributing factor for the discrepancy.

Return to the Q&A index


What happened to the Stage IV hourly files with parameter number of 237 instead of 61, during 20080314-20080326?

On 14 Mar 2008, WGRFC began using the AWIPS OB8.2 version of QPE, which sends out the non-QC'd hourly analysis (the early, automated runs) as parameter 237 on GRIB1 Table 128. The later, QC'd analysis is still sent out using the standard parameter 61 on GRIB1 Table 2 (which is "precipitation accumulation"). In the NCEP Stage IV mosaicking process, for each hour, the program reads in the available data from the RFCs sequentially, based on the RFC's ID number (150, 152, 153, ..., 161, 162; 151 [Alaska] is not used), and up until 27 Mar, PDS for the Stage IV analysis was based on the PDS from the last RFC GRIB file read in. WGRFC's ID number is 162, so for any hour, if WGRFC's non-QC'd analysis was the latest available, then the Stage IV analysis would have a PDS that says it's parameter is 237 on GRIB Table 128.

During 14-26 Mar 2008, if you were getting the Stage IV data in "real time", say a few hours later, then the data you were getting were likely of "parm 237". If you got the data a day or two later, after the QC'd analysis from WGRFC had come in for that hour, then most likely you would get the conventional "parm 61". We go back one day to re-do the hourly analysis, so if QC'd analysis from WGRFC had not come in by 23:30 UTC the next day, then the Stage IV analysis for that hour would have parm=237. GEMPAK (our plotting program) does not recognize parm=237 as precipitation and will only produce a plot if parm=61.

LMRFC and MBRFC started using the AWIPS OB8.2 version of QPE before WGRFC did, but because their ID numbers are 154 and 156, respectively, their 'parm 237' PDS did not affect the resulting Stage IV analysis.

If you have been reading the Stage IV analysis in real time on 18 Mar, you might have noticed that most of the hourlies were missing during the earlier hours. That was a problem separate from the 'parm=237': there was apparently a data outage between 14 UTC 17 Mar - 18 UTC 18 Mar, we were not receiving QPE from any RFCs.

On 16:33 UTC 27 Mar a solution to get around this problem was implemented. We now have the Stage IV analysis PDS hardwired as "parm 61/table 2".

If you are using Stage IV files between 14-26 Mar 2008 and you find that some of them have parm=237: the field values are still good. You just need to read the fields in without specifying that the parm needs to be '61'.

Return to the Q&A index


Ying[dot]Lin[at]noaa[dot]gov

Back to NCEP/EMC Hourly Precipitation Analysis Page

NCEP/EMC Mesoscale Modeling Branch