For both Stage II/Stage IV:
For Stage II:
For Stage IV:
What do the names (Stage I/II/III/IV) mean?
The nomenclature is somewhat confusing. Here is an attempt to clarify:
Stage 1/Stage I: Digital Precipitation Arrays (DPAs) derived from radar reflectivities, on 4km 131x131 polarstereographic grids centered on individual radar sites.
Stage 2/Stage II: merged radar+gauges product on the local 131x131 grid (done at each of the 12 CONUS RFCs for Stage 3 in the past, and at NCEP for the 'NCEP Stage 2'). Also used to refer to the NCEP product after we mosaic the 'local (131x131) Stage 2' into a national product.
Stage 3/Stage III/QPE: In the past, "Stage III" refers to 1-hour and 6-hour analyses from the RFCs that were mosaicked (non-NCEP) Stage 2 over each RFC domain, with manual QC. Currently the RFCs have transitioned to other algorithms - MPE (multisensor precipitation estimator) for most RFCs, Mountain Mapper for the Western RFCs and P1 for ABRFC. We shall hereby refer to the regional 1-hour/6-hour analyses we receive from the RFCs as the QPEs (for Quantitative Precipitation Estimates).
Stage 4/Stage IV: RFC QPEs mosaicked/quilted into a national product at NCEP
In the past (prior to 2000 or so) we sometimes used the names Stage II/Stage IV interchangably to refer to the NCEP hourly multi-sensor precipitation analysis. Since the RFCs started routinely producing/transmitting the manually QC'd QPEs over their respective domains, and NCEP has started to mosaick/quilt the MPE/Stage IIIs into a national 'Stage IV' product, we are now calling the non-QC'd multi-sensor product generated at NCEP (directly from the radar and gauge data, as opposed to the mosaic of the QPEs) the 'NCEP Stage II'. For more information regarding the MPE algorithm/products at the RFCs, please see a presentation by Rich Fulton of NWS/OHD: http://www.nws.noaa.gov/oh/hrl/papers/wsr88d/tac_nov02.pdf. Additional information about the MPE and WSR-88D rainfall algorithm can be found in http://www.nws.noaa.gov/oh/hrl/papers/papers.htm#wsr88d.
What is the grid specification?
For the 4km grid (applicable to Stage IV1 and Stage II archived since 1 Jan 20022), the NWP HRAP (Hydrologic Rainfall Analysis Project) grid:
All the grid lat/lon information is considered valid at the center of the grid box. If you compare the above specifications for the 4km grid against that given for the Grid 240 (the HRAP grid) in the NCEP GRIB manual, you might have noticed that the lat/lon coordinates for the corner points and the pole point seem to be shifted by half of a grid space. This is because the hydrologists who first created the HRAP grid (the national HRAP grid defined as above as well as the local HRAP grids the RFCs use for their Stage III/MPE data) defined a grid's coordinate using the 'lower-left' corner of the grid box, while meteorologists usually define a grid box's coordinate using the center of the box. In early 2000, when Brett MacDonald (then at NCEP/HPC) first plotted the regional analyses from the RFCs using GEMPAK (which considers a grid box's coordinate as the center of the box), he noticed that all the precipitation plots seemed to be 1/2-grid off. So he and I decided that, when mosaicking the local MPE products into the national Stage IV analysis, we should receive the MPE data as they are, mosaic them into Stage IV on the national HRAP grid, but provide our own grid definition for the HRAP for the atmospheric people (i.e. it is the same physical grid, but we define the grid coordinates using the center of the grid box, rather than the lower-left corner).
1The grid for Stage IV data from 1 Jan 2002 to 10 May 2004 was on a slightly shifted grid - the (1,1) point was mistakenly set to (23.117N,119.017W). Please see this page for further details.
2For the old 4km grid (applicable to the archived Stage II analysis prior to 1 Jan 2002), please see the old vs. new Stage II analyses comparison page.
For a more detailed discussion of the HRAP grid, please see the NWS/OHD documentation at http://www.nws.noaa.gov/oh/hrl/nwsrfs/users_manual/part2/_pdf/21hrapgrid.pdf.
What is the unit of the precipitation accumulation?
The unit is kg/m**2, which is equivalent to depth in mm.
How to find the lat/lon location of a grid point?
Two subroutines, w3fb06.f and w3fb07.f, can be used to find the nearest grid-coordinate for a given lat/lon location, and to find the lat/lon of a given grid point, respectively. Here are two examples of how to use the subroutines (please read the DISCLAIMER):
To find the grid coordinate of a location given by (rlat,rlon): call w3fb06(rlat,rlon,alat1,alon1,dx,alonv,xi,xj) where dx=4762.5 alonv=255. For the new (official NWP) HRAP grid (applicable to Stage IV analysis since 11 May 2004, and Stage II analysis after 1 Jan 2002): alat1=23.117 alon1=240.977 (=360.-119.023) For Stage IV analysis between 1 Jan 2002 - 10 May 2004, where the grid was slightly shifted: alat1=23.117 alon1=240.983 (=360.-119.017) For the old HRAP grid (Stage II analysis archive, 1 May 1996 - 31 Dec 2001): alat1=22.7736 alon1=239.624 (=360.-120.376) The return argument, (xi,xj), gives the location of point (rlat,rlon) on the 4km HRAP grid. (nint(xi),nint(xj)) is the grid point closest to (rlat,rlon). To find the lat/lon location of a grid point (i,j): xi=float(i) xj=float(j) call w3fb07(xi,xj,alat1,alon1,dx,alonv,alat,alon) (alat,alon) is the lat/lon of the grid point (i,j)For more information, please see the documentation block of these two subroutines.
Is there a land/sea mask for the HRAP grid?
Beth Ebert of the Bureau of Meteorology Research Centre kindly provided two land-sea masks on the HRAP grid created using IDL. The hi-res version has more spatial detail and the lo-res version has less. These are simple text files with 0's for water and 1's for land. The grid dimensions are 1121x881 and the grid origin is in the southwest.
How do I decode these files?
The format of the files is GRIB1. The files are compressed using the UNIX "compress" command and "uncompress" must be used before decoding (this refers to the Stage II/IV files. If you are referred to this section from some other EMC pages, then the file your are trying to decode might not have been compressed).
We regret that due to severe resource limitations, EMC cannot provide individual support on decoding these GRIB files or on GRIB decoding software. Please consult technical support department in your organization about GRIB/GRIB decoding. If you do not have access to other, non-NCEP decoders, you might want to check out the list below. Contact information for these software are listed in their web sites, but it seems that these are all more or less DIY programs, with the expectation that the user will be able to figure out answers to any questions through inspection of the documentation and the code.
1) wgrib, an excellent GRIB decoding, inventory, and manipulation package that is easy to compile on nearly any platform
2) NCEP GRIB Documentation and software (for GRIB enthusiasts who require no technical support)
How do I plot these grids?
There are numerous grid plotting packages, many of which are "GRIB-friendly". The following are suggestions on how to plot with GEMPAK, NCARGRAPHICS, and GRADS. Caveat: the suggestions are provided only for information purposes. A user will have to make any necessary changes to the code, scripts etc. and resolve any questions on his own, without assistance from NCEP.
You will need to run nagrib, version 5.3 or higher. The 4km grid is over 1,000,000 points, so in order to use nagrib on these grids, a recompile is necessary to expand the size of the maximum allowable grid. The 15km grid, available for Stage II analyses for the past 24h or so, is about 110,000 points, and should be decodable with nagrib.
You will need to subset the 15km grids so that other GEMPAK programs (GDPLOT, etc.) will be able to handle them. In nagrib:
Mike Baldwin (the original developer of the NCEP Stage II) had made up a sample decoding and plotting code using NCARGRAPHICS and W3LIB GRIB decoding routines. It is available for downloading here. This code requires the W3LIB GRIB decoding routines described earlier. This code will plot the 4km and 15km grids and produces a map similar to those found on the Stage II/IV web pages.
Refer to these instructions, provided by Jamie Kousky of OM. Note that in this set of instructions the grid specs in the *.ctl file is for the old HRAP grid, and for the data on the new HRAP grid you'll want to modify the array dimension, pole point etc. for the *.ctl.
Problem with grib2ctl (credit to Tristan Hall of FSU, 9 Jan 2015): when creating GrADS-readable ctl and idx files for Stage IV using grib2ctl and gribmap, the data files are blank for, say, 2005080100.01h and 2005080100.06h. This is likely due to the file names reflecting the ending hour of the accumulation period, instead of the beginning hour. Solution: grib2ctl has an option that uses the verification time (-verf). This fixes the problem.
Is there an archive? Where should I go to get the data?
The Past 14 days' data at http://nomads.ncep.noaa.gov/pub/data/nccf/com/pcpanl/prod/ pcpanl.yyyymmdd are placed on line by NCEP Central Operations as soon as the Stage II/Stage IV jobs are done. This is the quickest, most reliable place to get the current data.
The daily tar files are provided on a space-available bases. Caveat: if we find we are running low on space, these files might be deleted with no advance notice. Location and format of the tar files might be changed with no advance notice. Tar files might be missing or empty due to machine or connection problems, and missing tar files will not be available from NCEP. The daily tar file archive was first set up years ago when the NCAR Archive was still new and not as easy to use as today.
The NCAR CODIAC is the best place to get archived data. Here are the links to the data:
Images at NCAR CODIAC:
NCEP users with access to the NCEP WCOSS and HPSS:
Past 14 days' data at WCOSS:/com2/pcpanl/prod/pcpanl.yyyymmdd
HPSS tape archive:
Please note that NCEP has very little resource to provide user support for the Stage II/IV product. If a product is not available on line, users will have to rely on the NCAR CODIAC archive for the data. Please send technical questions/issues with the CODIAC archive at email@example.com. If you wish to obtain many months of the data all at once and have difficulties with the automated process through the above links, please contact firstname.lastname@example.org for help.
What's the difference among nomads.[ftpprd].[ftp].ncep.noaa.gov sites? How do I keep up with changes in data path?
NOMADS is the actual site for the past 14 days' of data placed there by the NCEP production job. 'ftpprd.ncep' and 'ftp.ncep' both point to the NOMADS site. In 2015 there was an effort to discontinue the ftp/ftpprd.ncep.noaa.gov addresses and just have nomads.ncep.noaa.gov - there was a TIN out for it, but it was not executed, presumably because turning the old addresses off would have affected too many people/jobs.
On 2 May 2017 the precipitation analyses data path changed from http://nomads.ncep.noaa.gov/pub/data/nccf/com/hourly/prod/ nam_pcpn_anal.yyyymmdd to http://nomads.ncep.noaa.gov/pub/data/nccf/com/pcpanl/prod/ pcpanl.yyyymmdd/. This was to meet some central operation standards. When asked in early March how users could be notified of such changes, we were told that anyone can receive notices by either
A request to set up a ListServe email list for the Stage II/IV users has been filed. If it (ever) gets approved, users can subscribe to the list to be notified of upcoming changes.
We hope there would not be another change of data paths in the future.
How is the observation time reflected in the file name?
The yyyymmddhh in the file name is the ending hour of the accumulation period. If yyyymmddhh is followed by xxh, then you are looking at xxh accumulation ending at yyyymmddhh.
For the Stage II hourly analysis files, only the multi-hour (6h, 24h) analyses contain the indicator xxh.
The time stamp in the grib headings of the 6-hourly Stage IV analysis follows that used in the 6-hourly regional analyses produced by the River Forecast Centers (RFCs). The 6-hourly Stage IV analysis is the mosaic of 6-hourly MPE analyses from the RFCs (the RFCs perform 6-hourly analyses separately from their hourly analyses, consequently the 6-hourly Stage IV is not simply the sum of 6 consecutive hourly analyses). The 24h-hour Stage IV totals are the sum of four 6-hourly Stage IV analyses. For their 6-hourly analyses, the RFCs use the previous 12Z as the "reference" time in the GRIB heading (grib PDS octets 13-17), and the accumulation beginning/ending times are given in octets 19-20. When we mosaic the 6-hourly MPEs into the 6-hourly Stage IV, we follow this convention, (and when we add up the 6-hourlies into 24h totals, the time stamp is similar to that in the 6-hourlies). Because of this, the time stamps in the 6-hourly/24h-hourly GRIB headers look like that of forecasts rather than that of analyses, though we do give the resulting Stage IV files names more consistent with its analysis nature, with the accumulating time period and ending hour of accumulation. Here are how things go:
Stage IV Actual accum Oct 13-16 Oct 19 Oct 20 file name ending time ref yymmddhh P1 P2 ST4.2002032618.06h 2002032618 02032612 0 6 ST4.2002032700.06h 2002032700 02032612 6 12 ST4.2002032706.06h 2002032706 02032612 12 18 ST4.2002032712.06h 2002032712 02032612 18 24 ST4.2002032712.24h 2002032712 02032612 00 24Note: analyses ending at 00Z (ST4.2002032700.01h, ST4.2002032700.06h etc.) are placed in that day's (i.e. 20020327) directory/tar file, even though their accumulation periods fell into the previous day (20020326).
All time in the Stage II/IV is UTC.
What causes the occassional "reflectivity rings" in the analyses?
Tim O'Bannon of the WSR-88D Radar Operations Center explained that this is an automatic gain control (AGC) problem that occasionally show up at some radar sites, and the crew at the WSR-88D Operations Center works with the sites to fix the problems as they arise. An example of this is in the 1-hour NCEP Stage II and Stage IV analysis ending at 2002032106, which is caused by the 200203210600 digital precipitation array (DPA) from TLH (Tallahassee, FL). We see this problem more often with the NCEP Stage II's than the Stage IV's, because at the RFCs where the regional MPE analyses are generated, an analyst is likely to notice the "ring" in the earlier, automated version of the analysis and re-run the later, QC'd version of the same analysis without the problem radar. 2002032106 is a rare example of the "reflectivity ring" showing up in the Stage IV hourly analysis. For this hour, we received only the automated version of the MPE analysis from SERFC. We did not receive the later, manually QC'd version from SERFC for this hour, but the problem was evidently noticed and corrected by SERFC during the QC, since the ring did not show up in the subsequent 6-hourly MPE analysis we received from them. We see the "reflectivity rings" more often in the NCEP Stage II since it does not have manual QC.
Which should I use, Stage II or Stage IV?
Since Stage IV benefits from manual QC at the local RFCs, it is generally better than Stage II, which has no manual QC. There are a number of other issues for your consideration:
1. Coverage period: Stage II began to be archived at NCAR on 1 May 1996. Stage IV did not began to be archived until 1 Jan 2002.
2. Do you need hourly or 6-hourly analysis? For 6-hourlies, Stage IV is the one to use (if you don't need the pre-2002 data). Stage IV hourlies have some coverage problem since we do not receive the local analyses from all the RFCs for all hours.
3. How soon do you need the analysis to be available? Stage II hourly analysis is made at 35 minutes after the top of the hour, then re-made twice at 6 hours and 18 hours later. Check out timeliness for Stage IV here.
4. If I choose Stage II, which type should I use? We recommend the multi-sensor analysis with the exception of early spring - early Aug 2000, when the radar coverage was bad. You might want to use the gauge-only analysis for that time period.
How were the data files compressed?
Data files up until 24 Jul 2013 have a suffix of ".Z" ('compress/uncompress'). Starting 25 Jul 2013, we transitioned to a new computer system and 'compress/uncompress' are no longer available, and the data files are gzip'd, with a suffix of ".gz". Both *.Z and *.gz files can be uncompressed using 'gunzip'.
Why is this file not called FAQ?
I do not know if any of these questions would be asked frequently. I thought about calling this 'F/IFAQ', but settled on Q&A.
Why are there missing/incomplete/unreadable hours? Can they be re-made?
Missing hours and hours with incomplete data coverage are usually caused by outage of input data streams (radar/gauge data for Stage II, regional analysis for Stage IV). Files that cannot be uncompressed or read might be the result of data transmission error somewhere along the way from production machine to workstation, then to NCAR CODIAC. Due to resource limitations (see the next item in the Q&A) we cannot go back and re-process a problem hour far in the past. In real time, when we encounter a problem, we try to resolve it quickly (by notifying agencies supplying the data, or by making a quick fix/reroute on our end), then try to salvage as much "recent" data as we can; after that we have to move on.
I have started lists of problem hours for Stage II and for Stage IV. If you encounter a problem hour that is not listed, please contact me at Ying[dot]Lin[at]noaa[dot]gov.
Why can't I get more user support?
Lack of resources - the development/maintenance/user support for the entire Stage II/Stage IV operation is supposed to be done by a fraction (about 25%) of one person's time. We became a de facto data provider accidentally - the Stage II/IV analyses were originally produced for our own data assimilation use in our model, so it was a side project for one person who was also doing a number of other things. Over time the Stage II/IV are used by many more people, but there has been no additional time allocated to their support. Given this time constraint, the only way to sustain the product and to best serve NCEP and outside users is to have most user questions answered by on-line documentation such as this Q&A file and to severely limit hands-on individual user support. This means, for example, that users are asked to obtain the data and decode the files by themselves, or obtain assistance from co-workers or from technical support department in their own organizations. We apologize for this inconvience, but we really do not have an alternative to keep this going.
What is the NCEP Stage II?
NCEP 'Stage II' refers to the real-time, hourly, multi-sensor National Precipitation Analysis (NPA) developed at the National Centers for Environmental Prediction (NCEP) in cooperation with the Office of Hydrology (OH). This analysis merges two data sources that are currently being collected in real-time by OH and NCEP. Approximately 3000 automated, hourly raingage observations are available over the contiguous 48 states from ASOS and via the GOES Data Collection Platform (DCP). In addition, hourly radar estimates of precipitation ('DPA', for Digital Precipitation Arrays) are obtained as compressed digital files via the ASOS network. The DPA estimates are created by the WSR-88D Radar Product Generator on a 131 x 131 4-km grid centered over each radar site. The data analysis routines, including a bias adjustment of the radar estimates using the gage data, have been adapted by NCEP on a national 4-km grid from algorithms developed by OH ("Stage II") and executed regionally at NWS River Forecast Centers (RFC).
How are the radar-only fields derived?
The radar-only product (ST2rd.* and ST2un.*) consists of approximately 140 WSR-88D radars which report to NCEP in real-time via ASOS. Each individual radar estimate is merged together on the national Hydrologic Rainfall Analysis Project (HRAP) grid and bins which contain more than one radar estimate are averaged together using a simple inverse-distance weighted average. Currently, there is no quality control of the radar estimates, such as removal of anomalous propagation.
There are current 2 types of radar-only analyses, one based on the raw radar (DPA) data (ST2rd.*) and the other is gage-adjusted (ST2un.*). The radar bias adjustment algorithm follows Smith and Krajewski (1991). These gage-adjusted radar estimates are available after an ~6h delay, to allow gage data necessary to estimate the biases to arrive.
How are the gage-only and multi-sensor fields derived?
In contrast to the simple radar-only mosaic technique, the analysis schemes used in the gage-only and multi-sensor analyses utilize optimal estimation theory. These were developed by Seo (1998). The schemes are fundamentally similar, and optimally estimate rainfall fields using raingage and radar data under partial data coverage conditions. This is preferred over previous statistically-based techniques because it takes into account the variability due to fractional coverage of rainfall, as well as within-storm variability. By objectively taking the spatial coverage into account, more accurate estimates of the rain versus no-rain area are obtained. Accurate delineation of this area is as important as accurate estimation of rainfall within the rain area. One of the underlying assumptions in the radar-gage analysis scheme is that the radar estimates are unbiased. Currently, radar biases are adjusted prior to the multi-sensor analysis by the technique developed by Smith and Krajewski (1991). This method attempts to remove the mean bias but does not attempt to remove range-dependent biases.
The 24h "rfc" analysis uses the same gage-only algorithm as the hourly analyses, except with the RFC 24h gage database. There are many more stations (7-8,000) than the hourly data.
How is the grib bitmap used to denote missing regions?
Each type of analysis uses the GRIB bitmap feature to denote the area of the domain that has enough data to provide an analyzed value. In other words, the bitmap tells you if you are inside the data domain or not. In the W3FI63 unpacker, a logical array is returned that is .TRUE. where the bitmap is turned on, and .FALSE. where the bitmap is turned off. The gribplot.f code takes advantage of this feature to denote the extent of the data domain. The gage-only analysis is considered to be inside the data domain within approximately 50km of the nearest gage report. The radar-only analysis is considered inside of the data domain within approximately 200km of each successfully decoded radar report. The multi-sensor analysis will use the gage-only value if no radar data are available, and the radar-only value if no gage data are available.
What kind of quality control steps are taken?
At the present time, the NCEP Stage II does not have manual quality control steps that are a hallmark of the RFC MPE analyses. However, some initial quality control steps have been implemented into the current Stage II system. There is a gross error check on the gage data so that no gage report of more than 1"/hr gets into the analysis. Also, a list of consistently bad raingages is in use, so that observations will be ignored from these reporting stations in the analyses. This list is subjectively determined by examination of numerous cases where the gages reported heavy rainfall for several hours while nearby gage and radar reports contained zero rainfall.
NCEP is working with FSL and OH to develop an automated quality control of the hourly gauges. An overview can be found here.
Why the new grid?
The official NWS HRAP (Hydrologic Rainfall Analysis Project) was created in the 1970's. Information dissemination was not easy back then (no internet), the exact specifications of the grid was not widely known. In 1996, when the National Stage II Analysis was first developed at NCEP (in cooperation with Office of Hydrology), Mike Baldwin created a grid (the HRAP grid that had been used in the NCEP Stage II analysis during 1996-2001) based on what scant information that he could find about the HRAP grid. The exact HRAP grid specification became known to us in 2000, and we decided to switch.
Why the new file names?
The first reason: there were two different sets of old file names.
Take the old 4km multi-sensor files. In the 'current and one-week archive'
files kept at the time, they were called "USAmlmmddyyhh.Grb.Z"
(this odd date stamp
format was hardwired in the old software), and in the long term archive,
they were called "mul4.yyyymmdd.hh..Z" (we used more sensible date
stamps for the long term archive). Users found both the mmddyyhh
date stamp and the different names in the long term archive confusing.
The second reason: since we have changed the grid, we decided that it is better to have totally different file names in case users don't catch this change in time (we figured if we use the same file name, there is a chance that someone would not be aware of the grid change, and run something out for 6 months with the wrong grid information).
Why did the multi-sensor data look so bad during spring-early Aug 2000?
As a part of the NWS conversion from AFOS to AWIPS, During Sep 1999 - Aug 2000, the radar precipitation products from the 137 radar sites over CONUS gradually changed from the HDP (Hourly Digital Precipitation) to the DPA (Digital Precipitation Array) format. NCEP had been receiving only the HDP data until Jul 2000, and the decoding of the DPA data did not start until 11 Aug 2000 (i.e. until then the multi-sensor analysis had been depending on an ever-dwindling number of radar sites that had yet to be converted from HDP to DPA). During Sep-Dec 1999, only 8 radar sites underwent the conversion. By Feb 2000, the conversion was in earnest, so during the next month, the reduced radar coverage became quite noticeable in the multi-sensor product, and by that summer, the radar coverage was reduced to practically nothing, until we switched to decoding the DPA on 11 Aug 2000.
We would recommend to the user who is planning on choosing a time period for an experiment that involves the NCEP Stage II analysis to avoid the period between 1 Mar - 10 Aug 2000 (February could be problematic too, the user would need to look at the snapshots of the data and decide on his own).
If a user must use data from this period, he might consider using the gauge-only analyses instead of the multi-sensor. The gauge-only analyses were, of course, not affected by the decrease in radar coverage. One of the downside of the gauge-only data (besides not being multi-sensor) is that the analyses are usually 'holey' (in contrast, during normal conditions, the radar analysis over CONUS have relatively few non-covered areas).
Bar Chart for the Monthly Conversions
|Snapshots of the Hourly Analyses|
Are the Stage II files on the 15km grid saved in the archive? How do I map the data from the 4km grid to the 15km grid?
The Stage II files mapped to the coarser 15km grid are not part of the archive at NCAR. For the pre-2002 Stage II, it is not saved in the 6-month on-line archive either. Starting from 1 Jan 2002, the 15km Stage II files are included in the near-term archive (6 months or less, space permitting), but it will not be part of of the permanent archive at NCAR.
If only the 4km-grid data are available to you and you would like the data on the coarser, 15km grid, you will need to run a re-mapping program. This is done by simply averaging the values of the 15km grid's nearest 3x3 neighbor's on the 4km grid (only one of the 3x3 points on the 4km grid is required to be non-missing to create a 15km grid point value). The (1,1) point on the 15km grid is at the same location as the (2,2) point on the 4km grid. You can use our remapping program as a template. The 4km analysis file is read in from unit 11, and the 15km analysis written out to unit 51. Note that if you are dealing with the pre-2002, analysis on the old HRAP grid, you need to change the parameters IMR, JMR, alat1, alon1 to those of the the old HRAP grid values that can be found here.
If it is not easy for you to write out data in GRIB format, you can just read in the 4km data, then do the simple 3x3 remapping as above.
Why were there streaks in the ST2rd/rad4 files prior to 20 Nov 2003?
This was caused by a bug in the way precipitation estimates under individual radar umbrellas were mosaicked into a national product. For the 'radar-only' product, when a given grid point is covered by more than one radar, a weighted sum is used as the precipitation estimate at this point. The inverse of the distance from this grid point to a radar is used to weigh the contribution from this radar. When this grid point is right at the radar site (i.e. if both delta(x) and delta(y) are zero), the weight is capped at a maximum value of 1. Unfortunately, a bug in the code set the weight to 1 if either delta(x) and delta(y) were zero.
This bug affects the 'radar-only' analysis, i.e. the ST2rd files during 1 Jan 2002 - 19 Nov 2003, and the rad4 files prior to 1 Jan 2002.
Here is a snapshot of an ST2rd hourly accumulation, with the bug in the code and Here is a snapshot of an ST2rd hourly accumulation, with corrected code.
The fix of this problem was implemented operationally on 20 Nov 2003.
For Stage IV:
How timely are the Stage IV analyses?
The NCEP Stage IV program is run each hour at approximately 35 minutes past the top of the hour. At that time, we check for any regional analysis that had arrived in the past hour and has a valid time that falls in the most recent 23 hours/47 hours for the 1h/6h mosaics; if there is input from at least one RFC, we do (or re-do) the analysis for that hour/6-hour. If you look at an analysis for, say, 2 hours ago, you might find that only a few RFCs have contributed to it. If you keep checking, once an hour, you might find that hour/6-hour's analysis gradually gets 'filled in' by contributions from other RFCs.
The hourly MPE receipt time ranges (roughly) from within 20 minutes to hour to 12 hours after.
The four 6-hourly analyses covering the previous 12Z-12Z are generally received by 15Z (for the automated runs) and 21Z (the manually QC'd runs). Some RFCs produce their 12-18Z, 18-00Z, 00-06Z analysis sooner.
How far back do we go to re-do the mosaicking: from 14 Apr 2015 on, we go back 7 days. Hourly Stage IV is re-made hourly (if there is new input) for the first 23 hours after valid time, then again at 1/3/5/7 days after valid time. The 6-hourly Stage IV is re-made hourly (if there is new input) for 47 hours after valid time. The four 6-hourlies covering a 12Z-12Z 24h period are also re-made on a fixed schedule at 1/3/5/7 days after the ending 12Z (i.e. at the 12:33Z run); at which time the corresponding 24h totals are also re-made. The current day's 24h totals are made/remade hourly at 12:33/13:33 ... 23:33Z run cycles.
Prior to 14 Apr 2015, hourly analysis valid in each day is re-made hourly (when there was new input) until 23:33Z the next day; 6-hourly analysis covering a 12Z-12Z 24h period is re-made hourly (when there was new input) until 23:33Z of the same day of the ending 12Z.
QPEs from Alaska (6h/24h) and Puerto Rico (1h/6h/24h) began to be included in the Stage IV on 2 May 2017. These are made/remade when QPEs are received (at the next xxh:33m run), and a final re-run made 7 days later.
Daily logs summarize arrival times for each hourly/6-hourly analysis from each RFC (ancient history: time stamp is UTC after 28 Mar 2002, ET - EST/EDT depending on the season - before 25 Mar 2002. Between 25-28 Mar there could be a mixture of both).
Why were the 1-hour analyses ending at 00Z so often missing along the SE coast between mid-May to mid-Sept 2002?
Short answer: a script glitch prevented many of the SERFC input for that hour to be used in the national mosaic.
Long answer: on 8 May 2002, we started running the Stage IV processing at 20 minutes after each hour (it was run less frequently before that). SERFC sends out a batch of data, including the 1-hour accumulation ending at 00Z, at 0520Z (analysis for this hour is sent only once by SERFC - see the daily log page for the time line). The Stage IV job script checked the current day and the previous day's incoming data for any new analyses from the RFCs. Any new analyses from the RFCs were split out and copied into subdirectories $day/$rfcid (which was created during the processing for the current day ($day), when new data from an RFC ($rfcid) were detected in the data 'tank'), the current day was processed first and then the previous day was processed. What I forgot was, in the incoming data 'tank', the 1-hour accumulation ending at 00Z was placed in 'previous day' (they sort data by the starting time of the accumulation period, while I do so by the ending time). When I ran the processing at 0520Z, around the same time when SERFC transmitted their batch of data, when I ran the 'current day', the SERFC batch hadn't come in yet, so no $day/$rfcid for SERFC was made. When I proceeded to run the 'previous day', the SERFC batch had arrived, my job detected the new 00Z SERFC analysis (hence you see it in the daily log files), extracted it and tried to place it in subdirectory $day/$rfcid - but $day/$rfcid didn't exist yet.
At the time of this writing (19 Sept 2002), the Stage IV processing was about to becoming operational. The soon-to-be operational scripts do not use the aforementioned subdirectory set up and are not affected by this glitch.
This problem affected the 00Z Stage IV hourly analysis produced between mid-May to 15 Sept 2002. Not all 00Z analyses during this time period were affected though - if the timing of the 0520Z SERFC transmission or my 0520Z Stage IV run was a little off, then the window for the glitch was closed and there'd be input from SERFC for the 00Z Stage IV analysis. A bug fix was put in on 16 Sept. The 00Z 16 Sept analysis, made before the bug fix, did contain input from SERFC - apparently the batch of data from SERFC arrived seconds later (or my job was run seconds earlier) than usual, and they were all processed an hour later, at 0620Z, so the problem didn't show up.
How is the mosaicking done?
The mosaicking for each hour/6-hour period is done via the following steps:
What happened to the Stage IV hourly files with parameter number of 237 instead of 61, during 20080314-20080326?
On 14 Mar 2008, WGRFC began using the AWIPS OB8.2 version of QPE, which sends out the non-QC'd hourly analysis (the early, automated runs) as parameter 237 on GRIB1 Table 128. The later, QC'd analysis is still sent out using the standard parameter 61 on GRIB1 Table 2 (which is "precipitation accumulation"). In the NCEP Stage IV mosaicking process, for each hour, the program reads in the available data from the RFCs sequentially, based on the RFC's ID number (150, 152, 153, ..., 161, 162; 105 [Puerto Rico] and 151 [Alaska] are processed separately from ConUS), and up until 27 Mar, PDS for the Stage IV analysis was based on the PDS from the last RFC GRIB file read in. WGRFC's ID number is 162, so for any hour, if WGRFC's non-QC'd analysis was the latest available, then the Stage IV analysis would have a PDS that says it's parameter is 237 on GRIB Table 128.
During 14-26 Mar 2008, if you were getting the Stage IV data in "real time", say a few hours later, then the data you were getting were likely of "parm 237". If you got the data a day or two later, after the QC'd analysis from WGRFC had come in for that hour, then most likely you would get the conventional "parm 61". We go back one day to re-do the hourly analysis, so if QC'd analysis from WGRFC had not come in by 23:30 UTC the next day, then the Stage IV analysis for that hour would have parm=237. GEMPAK (our plotting program) does not recognize parm=237 as precipitation and will only produce a plot if parm=61.
LMRFC and MBRFC started using the AWIPS OB8.2 version of QPE before WGRFC did, but because their ID numbers are 154 and 156, respectively, their 'parm 237' PDS did not affect the resulting Stage IV analysis.
If you have been reading the Stage IV analysis in real time on 18 Mar, you might have noticed that most of the hourlies were missing during the earlier hours. That was a problem separate from the 'parm=237': there was apparently a data outage between 14 UTC 17 Mar - 18 UTC 18 Mar, we were not receiving QPE from any RFCs.
On 16:33 UTC 27 Mar a solution to get around this problem was implemented. We now have the Stage IV analysis PDS hardwired as "parm 61/table 2".
If you are using Stage IV files between 14-26 Mar 2008 and you find that some of them have parm=237: the field values are still good. You just need to read the fields in without specifying that the parm needs to be '61'.
What are the meanings of the different colors in the Stage IV coverage map? How much can we trust the coverage in different areas?
The RFCs don't pay much attention to areas outside of their domains proper, so for quantitative purposes you should use only grid points within the RFC domains: the non-white, non-beige areas in the ConUS RFC coverage map. In the map, the white and beige areas are both outside of RFCs' domains. In the Stage IV mosaic since Jan 2016, the white areas are set to be "missing", The beige areas in the Gulf Coast and off the Atlantic Coast are included for visual interest. The central/eastern RFCs' QPEs have some incidental offshore radar coverage and it's a shame to throw it away, since when there is a coastal or tropical storm (e.g. Hurrican Matthew in Oct 2016) it can be quite interesting, so we decided to retain these off shore coverage (denoted by the beige areas in the Gulf of Mexico and off the Atlantic Coast). Note that since RFCs do not use a data/no data mask, areas outside of radar range in the beige area will always simply show "zero precip". You should place MUCH less trust to the coverage in the beige areas.
NWRFC and CNRFC's QPEs are gauge-only, so their coverage stops at the West Coast.
In Alaska, due to the lack of reliable observations, especially in winter, the analysis should be considered "qualitative" and should not be used for quantitative verification scores comparisons or for the tuning of models. We run OCoNUS model QPF verification using the AK QPE (e.g. here), but it's just for the visual comparisons, we do not use the statistics generated in the verification. For one thing, there is really no way to tell - especially in the more remote areas and/or in winter - whether an area showing zero precip simply did not have observations. This is why the Alaska Stage IV does not include a data mask - we initially created a land-only mask for the Alaska QPE, but later decided not to use it, because using the mask might imply more confidence in some land areas in AK than can be warranted.
Back to NCEP/EMC Hourly Precipitation Analysis Page
NCEP/EMC Mesoscale Modeling Branch