We typically receive the radar 'digital precipitation arrays' (DPAs) from each radar site every 6-10 minutes. For each radar site, we use the DPA that is closest to the top of the hour and within a +/- 10 minutes window for that hour's analysis. DPAs outside of the +/- 10 minutes window (e.g. 1750-1810Z, for the 18Z analysis) are not used. Occasionally there is a data transmission problem that leads to a gap in the radar data. If there are no DPA reports covering the +/- 10 minutes window at the top of an hour, then radar-related analyses (i.e. all except the 'gauge-only' analysis) are not made for that hour (note: since 20*, when there are no DPA files, the Stage II job terminates without making the gauge-only analysis) Data transmission outages tend to happen when the weather is quite active over ConUS, because that's when the transmission lines tend to have traffic that exceeds their capacity. When there is a missing hourly analysis, the 6h/24h analyses that contains that hour are also considered missing. Hour ending Problem Cause (UTC) ------------------------------------------------------------- 20181123: NCF/NWSTG outage affected 13-18Z DPA. No DPA for 13:51-15:16. DPAs late through dpa_18. Also no incoming RFC QPEs 13:37-18:28Z. No ST2 radar-related products for the hour valid 15Z and the affected 6h/24h. 14Z only has coverage at OKC and JFK (two DPAs valid at 13:50Z, from OKC and JFC radars, made it through before outage; these were used for the 14Z ST2). PCPRTMA normally runs in near real-time; NCO re-ran 15-17Z pcprtma at 21:18/21:23/21:25Z. 14Z and 18Z pcprtma ran in real time, so no 14Z pcprtma (the two DPAs at OKC and JFK valid at 13:50Z didn't arrive in time), and 18Z pcprtma has incomplete coverage (DPA just starting to coming back at the time of 18Z pcprtma run), 20170213: Network outage 1808Z to 2044Z. No DPA for hour ending 19Z, very few for hour ending 20Z. This affects hourly Stage II ending at 19Z and 20Z, 6h ending at 00Z 14 Feb and 24h ending at 12Z 14 Feb. 20140720: TOC outage around 03Z. Most DPAs missing. This affects hourly Stage II ending at 03Z, 6-hourly ending at 06Z, and 24h ending at 12Z. 20130822-0823: Problem at TOC. Many DPAs missing during 22Z 22 - 02Z 23 Aug. Sparse coverage in Stage II during 23Z 22 - 02Z 23 Aug. 2013061607: Missing DPAS from Riverton, WY and all radars from MT during 06:50-07:10 UTC. 2012091211: Many western DPAs were missing during 10:50-11:10 UTC. 20110817/15Z - 0818/14Z Most DPAs were missing. 20110725 18-19Z: No DPAs. 20110719 16-18Z: Most DPAs were missing. 20110601-02: Many missing DPAs from 15Z 1st to 14Z 2nd of June. Sparse coverage in Stage II analysis during these hours. 20100923 Many missing DPAs during 03-06Z. Sparse radar coverage seen in hourly Stage II analysis ending 04/05/06Z. 2010092116-092215: Very few DPAs received during this time. Likely problem at the Gateway (TOC). 20100824: No Stage II analysis for hours ending at 17-23Z, scant radar coverage for hours ending 15-16Z. Reason: AWIPS header for this data type was changed today, there was some confusion as to the starting time of the change, so the number of DPAs received at NCEP started to dwindle beginning at 14Z, and the data stopped completely during 17-21Z. In the past (the last time we had a DPA outage appears to be 20080116), when there were no DPA files available, the part of the program that reads the DPA files exited with a return code of 99, and the job proceeded to the next step in the Stage II program, so the gauge-only analysis was still available when there was a DPA outage. At this time, however, the production script killed the entire job when there is a non-zero return code from any part of the program, so when there was no DPA input, there was no gauge-only analysis either. Also, each hour we run the Stage II for this hour (T0), and also the T0-6h, T0-18h (the "early/mid/late" runs), so that after 21Z, the mid/late runs began to fail as they rolled by the hours with missing DPAs, causing the entire Stage II job to fail. Fixes were put in to deal with this problem on the morning of 25 Aug, but the Stage II files for the hours ending 22-23Z 24 Aug were also lost. 2008011601: No DPA received between 00:22-01:35Z. 20071119: Dataflow from TOC disrupted (per SDM log). Gauge and radar data not available for 11-12Z. No analysis. 2007101505: No DPA received between 04:30-05:23Z. 2007080422: No DPA received between 21:47-22:40Z. 2006072513: No DPA received between 12:31-13:37Z. 20060502: No DPA report during 08:44-10:10Z. No METAR gauge data for 09-14Z. HADS gauge data appear to be missing/short 08-16Z. 2006021516: Only 1 DPA report (from KGSP in South Carolina) was received between 15:15-16:15Z. 2005101319: Very scant coverage due to lack of DPA. Likely related to general outage of IBM White 1400L to 1820L. 2004121317: There are no radar-related analysis for 17Z and 18Z. No DPA received from 16:41-17:45Z. 20040920-21: There are no Stage II analysis for 10-11Z and 17-23Z 20 Sept, and 00Z 21 Sept. No radar data received between 09:32-11:25Z, No METAR gauge data for 11-12Z, and 16-23Z. In the process of coping with the wide-spread data outage, the the hourly precipitation analysis script was modified to eliminate the 'mid' run, which unfortunately led to a mismatch between the number of POE jobs to be run and the number of total tasks in the SMS script, causing the Stage II jobs to fail. This problem was fixed on 20Z 21 Sept. Re-run went back to 01Z 21 Sept. 2004061002: No radar data received between 01:45-02:02Z. 20040509: No radar data received between 05:50-07:13Z. No gauge data 05-08Z. So no gauge-only analysis for 05-08Z. No radar-related analysis for 06-07Z. 2004043018/2004050112: 6-hourly gauge-only accumulation ending 2004043018 and 24-hour gauge-only accumulation ending 2004050112 were missing. Hourly analyses during 2004043013-2004043018 were too sparse, causing the 6-hour totals ending 2004043018 to be all missing. 2003122614: No radar data received between 13:21-14:19Z. 2003091020: Received only 3 radar reports between 19:50-20:10Z. 2003082704-2003082808: Sparse hourly coverage 04-07Z; no data 08Z. Reason: power outage during the evening of Tue 26 Aug. Even after the computers recovered, the outage-related backlogs caused disruption in incoming dataflow. 2003072201 No radar data received during 0047-0120Z 2003071205-2003071210, 2003071223-2003071322: Major network disruption of over 13 hours during the first half (UTC, approx.) of Saturday 12 Jul. No radar data during 04:08Z-10:27Z. In the process of rescuing as many of the operational jobs as possible, the hourly precipitation analysis script was modified on 23Z 12 Jul, which unfortunately led to a mismatch between the number of POE jobs to be run and the number of total tasks in the SMS script, causing the Stage II jobs to fail. This problem was fixed on 17Z 14 Jul. Re-run went back to 23Z 13 Jul. 2003061400 No radar data received during 2335-0015Z. 2003010117 Only one radar report received during 1650-1710Z 2003010114 No radar data received during 1322-1410Z 20021213 No radar data received during 0542-0616Z, 1343-1428Z, 2245-2315Z, hence no radar-related analyses for 06Z, 14Z and 23Z. 16Z radar-related analyses are sparse due to many unavailable radar reports during 1550-1610Z. 20020505 daily tar file corrupted, some compressed files cannot be decompressed. 2002032719 No analysis made OSO data outage. No radar data received during 1840-1926Z (1850-1910 data needed to make 19Z analyses) 2002010103 Spotty radar coverage Majority of radar reports were not received at NCEP for that hour 2001083118 No analysis made OSO data outage. No radar data received during 1735-1843Z (1750-1810 data needed to make 18Z analyses)