Printer image icon. Send To Printer
 




How to Implement BUFR Table Changes for Satellite Ingest Types

(bufrtab.003, bufrtab.005, bufrtab.008, bufrtab.012 and bufrtab.021)

Dennis Keyser - NOAA/NWS/NCEP/EMC
(Last Revised 03/13/2012)


Please take a moment to read the Disclaimer for this non-operational web page.
 

NOTE: Links open in a new window




PLEASE NOTE: When changing BUFR tables, complete the following procedure in its
entirety on each machine (i.e., cirrus and stratus).



BUFR table changes add an extra degree of complexity to an implementation,
because the BUFR tables are used as a definitive layout of the contents of a
particular BUFR type/subtype. Specifically, the same BUFR table that is used by a

cirrus/stratus satellite ingest job to encode a particular type of observation into
BUFR is also used by the subsequent tranjb process to append those same observations
to the BUFR files for that type within the BUFR database /dcom/us007003. So whenever

the BUFR table is changed for a particular type/subtype, a copy of the new table needs
to be written into all of the satellite ingest BUFR files for that type/subtype within
the database, at the point directly before which observations encoded according to the

new table will begin to be appended to each file. Applications which read these
satellite ingest database files will then respond to the new table and adjust
accordingly to the new layout definition as they continue reading through the files.


Note that, while one might assume that it would only be technically necessary to
write a copy of the new table to files of the actual type/subtype(s) which are
changing, one cannot guarantee 100% success in subsequent tranjb processing unless a
copy of the new table is appended to all satellite ingest files in the database which
use the same table.  In other words, if, e.g., a change is being made to type/subtype
b021/xx201 within bufrtab.021, one should update all satellite ingest files of type
b021 within the database, not just the xx201 files. The reason for this is two-fold.
First, it is possible that a change to the BUFR table for a particular BUFR type could
cause tranjb to fail when processing compressed BUFR messages for this BUFR type, even
for subtypes other than those modified in the table (if the new table is not copied
into these files).  Second, even if tranjb processing is successful in this case, it
is possible that downstream dumpjb processing could fail, again even for subtypes
other than those modified in the table (if the new table is not copied into these
files).


Here is the procedure to follow for updating satellite ingest files in the database
when a BUFR table change is being implemented:

  1. Suspend, via SMS, the satellite ingest job(s) that encode data into the tankfiles
    in the database for the BUFR table being implemented.
    • For changes to bufrtab.003, current jobs to suspend are:
      JIGOES_RADSND.sms.prod and JITOV.sms.prod.
    • For changes to bufrtab.005, current job to suspend is: JISWD.sms.prod.
    • For changes to bufrtab.008, current jobs to suspend are:
      JIOZONE_ORBIT.sms.prod and JIAEROSOL.sms.prod.
    • For changes to bufrtab.012, current jobs to suspend are: 
      JIAMSRE_SWATH.sms.prod, JIGOES_RADSND.sms.prod, JIGOES_SST.sms.prod,
      JIPOES_SST.sms.prod, JIQST.sms.prod, JISMI.sms.prod, JITMI.sms.prod and
      JIWINDSAT.sms.prod.
    • For changes to bufrtab.021, current jobs to suspend are:
      JIAIRS.sms.prod, JIAVHRR.sms.prod, JIGOES_RADSND.sms.prod, JINPP.sms.prod,
      JISMI.sms.prod and JITOV.sms.prod.


  2. (Optional) This document is intended to deal with the updating of only those
    database tankfiles generated by the satellite ingest processing. Since at this
    time every BUFR table associated with the subtypes generated by the satellite
    ingest also includes one or more subtypes generated by the NCO decoder
    processing, it may be desirable to also update the database tankfiles
    associated with those subtypes at the same time.  Please consult with NCO/SIB to
    determine if this should be done in this particular instance.  If the decoder
    subtypes in the database are to also be updated, please perform items 1 through 3
    in NCO's procedure for implementing BUFR table changes for decoder types at 
    http://www2.nco.ncep.noaa.gov/sib/decoders/Administrator/index.shtml
    .

  3. At this point, it is now safe to cd to the BUFR table directory /nwprod/fix
    and actually copy the new version of the BUFR table from the user work area
    into this directory. This is also the time to implement any other changes to
    the underlying satellite ingest processing (i.e. new executables, new scripts,
    etc.) by copying them into their proper respective locations within /nwprod.

  4. (Optional) If the decoder subtypes in the database are also being updated, we
    are now ready to update these with the new table. When a BUFR table is being
    implemented, we need to update all of the decoder-generated tanks in the
    database corresponding to that BUFR table..  The satellite ingest-generated

    tanks are to be excluded at this point.
    • For changes to bufrtab.003, use:
      $ export APXDX_EXCLUDE=xx002,xx003,xx104
      $ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 /nwprod/fix/bufrtab.003
    • For changes to bufrtab.005, use:
      $ export APXDX_EXCLUDE=xx010,xx011,xx012,xx014,xx019,xx070,xx071,xx080
      $ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 /nwprod/fix/bufrtab.005
    •  For changes to bufrtab.008, use:
      $ export APXDX_EXCLUDE=xx011,xx012,xx013,xx014,xx015,xx041,xx042
      $ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 /nwprod/fix/bufrtab.008
    • For changes to bufrtab.012, use:
      $ export APXDX_EXCLUDE=xx001,xx002,xx004,xx011,xx012,xx013,xx017,xx018,\
      $ xx022,xx023,xx031,xx034,xx103,xx122,xx123,xx138,xx139,xx150
      $ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 /nwprod/fix/bufrtab.012
    • For changes to bufrtab.021, use:
      $ export APXDX_EXCLUDE=xx023,xx024,xx025,xx027,xx028,xx041,xx051,xx052,\
      $ xx053,xx054,xx123,xx201,xx202,xx203,xx204,xx240,xx241,xx249,xx254
      $ /nwprod/util/ush/run_apxdx.sh /dcom/us007003 /nwprod/fix/bufrtab.021
    In each case, the last command may take a minute or so to complete depending on
    the number of tankfiles being updated within the database, so be patient!!


  5. (Optional) If the decoder subtypes in the database are also being updated, please
    perform item 6 in NCO's procedure for implementing BUFR table changes for decoder
    types at http://www2.nco.ncep.noaa.gov/sib/decoders/Administrator/index.shtml.
    This will turn back on the dataflow to the NCO decoders which had been suspended.
    NOTE: THE SATELLITE INGEST PROCESSING IS STILL SUSPENDED AT THIS POINT!   Since
    the NCO decoders are very time sensitive, they are turned back on at this point,
    prior to the update of the database tankfiles generated by the satellite ingest
    processing. The implementation is now complete and we are ready to turn the data
    flow back to all of the satellite ingest jobs and NCO decoders which were
    previously suspended. In the case of NCO decoder processing, please refer to item
    6 in the procedure for implementing BUFR table changes in the NCO decoder
    processing at http://www2.nco.ncep.noaa.gov/sib/decoders/Administrator/index.shtml.

  6. We are now ready to update the satellite ingest database with the new table. We
    need to update all of the satellite ingest-generated tanks in the database
    corresponding to that BUFR table. Decoder-generated tanks in the database
    corresponding to the new BUFR table are NOT updated here.  These were either
    already updated (see 5 above) or are not being updated in this particular instance.
    • For changes to bufrtab.003, use: 
      $ export APXDX_TYPE=satingest
      $ /meso/save/wx22dk/dcom.jifs/run_apxdx.sh /dcom/us007003/nwprod/fix/bufrtab.003
    • For changes to bufrtab.005, use:
      $ export APXDX_TYPE=satingest
      $ /meso/save/wx22dk/dcom.jifs/run_apxdx.sh /dcom/us007003/nwprod/fix/bufrtab.005
    • For changes to bufrtab.008, use:
      $ export APXDX_TYPE=satingest
      $ /meso/save/wx22dk/dcom.jifs/run_apxdx.sh /dcom/us007003/nwprod/fix/bufrtab.008
    • For changes to bufrtab.012, use:
      $ export APXDX_TYPE=satingest
      $ /meso/save/wx22dk/dcom.jifs/run_apxdx.sh /dcom/us007003/nwprod/fix/bufrtab.012
    • For changes to bufrtab.021, use:
      $ export APXDX_TYPE=satingest
      $ /meso/save/wx22dk/dcom.jifs/run_apxdx.sh /dcom/us007003/nwprod/fix/bufrtab.021
    In each case, the last command may take several minutes to complete depending on
    the number of tankfiles being updated within the database as well as their size
    (some of the b021 files are very large), so be patient!


  7. The implementation is now complete and we are ready to turn back on, via sms,  the
    satellite ingest jobs which had been suspended.


  8. Don't forget to notify the SDM, operators, etc. that the implementation has been
    completed on this machine (i.e. cirrus or stratus), and don't forget to repeat the
    above procedure on the other machine as well!