(bufrtab.003, bufrtab.005, bufrtab.008, bufrtab.012 and bufrtab.021)

Dennis Keyser - NOAA/NWS/NCEP/EMC
(Last Revised 11/27/2017)

Please take a moment to read the Disclaimer for this non-operational web page.
  
NOTE: Links open in a new window
IMPORTANT: All BUFR tables associated with the subtypes generated by the satellite
ingest are now separate and distinct from those associated with the NCO decoder
processing. Thus any changes to the satellite ingest BUFR tables have no affect on
NCO decoder processing. See
http://www2.nco.ncep.noaa.gov/sib/decoders/Administrator/index.shtml for instructions
on implementing BUFR table changes to BUFR tables associated with NCO decoder processing.
(Note to NCO/SIB: Please remove all references to satellite ingest in this web page.)


PLEASE NOTE: When changing BUFR tables for satellite ingest data types,, complete the
following procedure in its entirety on each WCOSS machine in which the satellite ingest
is running and filling BUFR tanks in /dcom.


BUFR table changes add an extra degree of complexity to an implementation, because the
BUFR tables are used as a definitive layout of the contents of a particular BUFR type/
subtype. Specifically, the same BUFR table that is used by a WCOSS satellite ingest
job to encode a particular type of observation into BUFR is also used by the subsequent
tranjb process to append those same observations to the BUFR files for that type within
the BUFR database /dcom/us007003. So whenever the BUFR table is changed for a particular
type/subtype, a copy of the new table needs to be written into all of the satellite
ingest BUFR files for that type/subtype within the database, at the point directly
before which observations encoded according to the new table will begin to be appended
to each file. Applications which read these satellite ingest database files will then
respond to the new table and adjust accordingly to the new layout definition as they
continue reading through the files.

Note that, while one might assume that it would only be technically necessary to write a
copy of the new table to files of the actual type/subtype(s) which are changing, one
cannot guarantee 100% success in subsequent tranjb processing unless a copy of the new
table is appended to all satellite ingest files in the database which use the same
table. In other words, if, e.g., a change is being made to type/subtype b021/xx201
within bufrtab.021, one should update all satellite ingest files of type b021 within the
database, not just the xx201 files. The reason for this is two-fold. First, it is
possible that a change to the BUFR table for a particular BUFR type could cause tranjb
to fail when processing compressed BUFR messages for this BUFR type, even for subtypes
other than those modified in the table (if the new table is not copied into these
files). Second, even if tranjb processing is successful in this case, it is possible
that downstream dumpjb processing could fail, again even for subtypes other than those
modified in the table (if the new table is not copied into these files).

Here is the procedure to follow for updating satellite ingest files in the database when
a BUFR table change is being implemented:

Suspend, via ECflow, the satellite ingest job(s) that encode data into the tankfiles
in the database for the BUFR table being implemented.
For changes to bufrtab.003, current jobs to suspend are: JIGOES_RADSND and 
JITOV.
For changes to bufrtab.005, current job to suspend is: JISWD.
For changes to bufrtab.008, current jobs to suspend are: JIOZONE_ORBIT,
JIAEROSOL and JINPP. 
For changes to bufrtab.012, current jobs to suspend are: JIGOES_RADSND,
JIGOES_SST, JIPOES_SST, JIQST and JISMI.
For changes to bufrtab.021, current jobs to suspend are: JIAIRS, JIAVHRR,
JIGOES_RADSND, JINPP, JINPP_ATMS, JINPP_CRIS_2211, JINPP_CRIS_431, JISMI and
JITOV.

(no longer applicable - skip to item 3 below)

At this point, it is now safe to cd to the BUFR table directory
/nwprod/obsproc_satingest.vX.Y.Z/fix (where X.Y.Z is the latest version of 
obsproc_satingest) and actually copy the new version of the BUFR table from the
user work area into this directory. This is also the time to implement any other
changes to the underlying satellite ingest processing (i.e. new executables, new
scripts, etc.) by copying them into their proper respective locations within
/nwprod.

(no longer applicable - skip to item 6 below)

(no longer applicable - skip to item 6 below)

We are now ready to update the satellite ingest database with the new table. We
need to update all of the satellite ingest-generated tanks in the database
corresponding to that BUFR table.
For changes to bufrtab.003, use: 
$ export APXDX_TYPE=satingest
$ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 \
  /nwprod/obsproc_satingest.vX.Y.Z/fix/bufrtab.003
(where X.Y.Z is latest version of obsproc_satingest containing updated bufrtab.003)
For changes to bufrtab.005, use:
$ export APXDX_TYPE=satingest
$ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 \
  /nwprod/obsproc_satingest.vX.Y.Z/fix/bufrtab.005
(where X.Y.Z is latest version of obsproc_satingest containing updated bufrtab.005)
For changes to bufrtab.008, use:
$ export APXDX_TYPE=satingest
$ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 \
  /nwprod/obsproc_satingest.vX.Y.Z/fix/bufrtab.008
(where X.Y.Z is latest version of obsproc_satingest containing updated bufrtab.008)
For changes to bufrtab.012, use:
$ export APXDX_TYPE=satingest
$ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 \
  /nwprod/obsproc_satingest.vX.Y.Z/fix/bufrtab.012
(where X.Y.Z is latest version of obsproc_satingest containing updated bufrtab.012)
For changes to bufrtab.021, use:
$ export APXDX_TYPE=satingest
$ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 \
  /nwprod/obsproc_satingest.vX.Y.Z/fix/bufrtab.021
(where X.Y.Z is latest version of obsproc_satingest containing updated bufrtab.021)
In each case, the last command may take several minutes to complete depending on
the number of tankfiles being updated within the database as well as their size
(some of the b021 files are very large), so be patient!

The implementation is now complete and we are ready to turn back on, via ECflow,
the satellite ingest job(s) which had been suspended.

Don't forget to notify the SDM, operators, etc. that the implementation has been
completed on this machine (i.e. tide or gyre), and don't forget to repeat the
above procedure on the other machine as well!