CHANGE LOG FOR UNIFIED 3DVAR CODE ------------------------------------------- Author(s) : John Derber, Kozo Okamoto, Dave Parrish, Ricardo Todling, Russ Treadon, Wan-Shu Wu, Banghua Yan Reviewer(s) : Russ Treadon Date : 4 August 2004 GMAO CVS tag: NCEP CVS tag: ncep-gsi-2004_07 REASON FOR CHANGES ------------------ Changes are made to the gsi code for the following reasons a) Reduce memory requirements for thinning satellite data b) Allow inclusion of conventional SST observations (not tested) c) Addition of interface to allow connection with the WRF MASS CORE model. This change pertains only to the regional system d) Generalize transform from earth lat-lon to model grid coordinate. This is necessary, because in regional mode, the WRF mass core uses a lambert conformal grid. The new code figures out how to transform automatically from supplied earth lats and lons of the model grid (these fields are currently available in the WRF restart files for both NMM and MASS CORE). This change results in medium changes for all regional runs. This change only pertains to the regional system. e) There was a problem with noisy vorticity increments in the top layers when running in global mode. The following change was made to the code to address this problem: Redefine lookup table for coefficients of recursive filters. The table is defined for just the scales needed locally in each PE and the increment of the table is in unit of scale instead of 1/scale. To be safe, the size of the table is also increased (less than 1% increase in total memory) f) Negative horizontal scales were found in the polar patches (global mode) due to sharp gradients near the coast line. The following quick, temporary fix is made to address this problem: Remove sea-land contrast on horizontal scales for surface temperature (sst,ice,land). g) Convert some doc blocks to protex-compliant form (only done for a few routines, still needs to be done for majority of routines) h) Improve emissivity models over snow/ice for AMSU-A/B i) Subroutine tbalance.f90 was using qsatg, not qsatg_fix, to normalize q. Array qsatg saturation specific humidity) is updated on each outer iteration. Array qsatg_fix, as it's name implies, is held constant throughout the analysis. We should use the fixed saturation specific humidity when normalizing q in the minimization. j) The logic around the call to set_ozone_var in glbsoi.f90 was incorrect. Routine set_ozone_var, found in berror.f90, scales the ozone background error, held in array dssv, based on a global mean ozone profile (computed from the guess). The background error array dssv is defined and applied on subdomains. However, the glbsoi.f90 logic around the call to set_ozone_var only rescaled the ozone background error on those tasks which horizontally smooth ozone. For the remaining tasks, the ozone background error remained that which was set in prewgt.f90 Thus, we had a mixture of unscaled and scaled versions of the ozone background error being applied in hoper and htoper. The background error needs to be rescaled on all tasks. This bug explains why we obtain different results when running the same case using different numbers of mpi tasks. k) Bugs were found/corrected in the ozone diagnostics (setupoz.f90). Routine stpcalc.f90 is also updated, though, the bug in array indices was on a write statement that is commented out in the code. l) Remove horizontal (1-2-1) smoothing of tropopause pressure. The smoothing was done on subdomains and, thus, different smoothed values were obtained along the edges of the adjacent subdomains. This created a mpi task number dependency on the results. By removing the horizontal smoothing this dependency is removed. The tropopause pressure is used in setuprad. It was decided that adding the communication needed to smooth the field globally (ie, independent of task id) was not worth the extra wall clock time. m) The indexing of arrays in intoz and stpoz was altered from 2-d to 1-d, thereby, making these routines consistent with other int* and stp* routines. Variable names in intpcp, intrad, stppcp, and stprad were changed to be consistent with other int* and stp* routines. n) Logic was added to deter_subdomain, deternpe, sub2grid, and grid2sub to handle the case of periodic (east-west periodic) subdomains. Given the current algorithm in deter_subdomain this only occurs when running the gsi with a single mpi task. In this case the previous code did not correctly handle observations that fell between the last (western-most) and first (eastern-most) analysis grid points. deter_subdomain and deternpe were altered to detect this periodicity. sub2grid and grid2sub were altered to correctly manipulate the grids in this situation. SPECIAL NOTES ------------- a) If you are NOT thinning satellite data AND there are more than 2.e6 observation profiles, you must change the value of itxmax in satthin.F90 b) As noted above, the reading/prep/analysis of SST observations has not yet been tested since the data is not available. This capability should be tested in the near future. EXPECTED DIFFERENCES -------------------- a) The largest difference comes from the improved emissivity model over snow/ice for AMSU-A/B. The new emissivity model results in a large increase in the number of AMSU-A/B observations that pass quality control and are, thus, used in the analysis. Almost all the additional data used is found poleward of 60 degrees latitude. This additional data has its greatest impact on the polar lower troposphere. b) The qsatg_fix bug fix to tbalance.f90 only affects the moisture analysis. The overall impact is slight. It does, however, remove poor convergence issues for certain cases. c) The ozone background error bug fix to glbsoi.f90 has a minor overall impact. d) Differences in analysis results from changes of the filter coefficients and changing the scales of the SST and sfc temp. Visually, pre- and post- change results are similar. e) Regional only - change to general coordinate transformation results in differences somewhat larger than roundoff and also results in small changes - observations closer to the edge of the model domain are allowed in the analysis. The increase in observations processed by the analysis changes the minimization so differences are seen not only along the model domain edges but over the entire domain. FILES REMOVED ------------- none FILES ADDED ----------- emiss_ssmi.f90 - calculate snow/ice emissivity for SSM/I fill_mass_grid2.f90 - convert wrf mass-core c grid to a grid iceem_amsu.f90 - calculate ice emissivity for AMSU-A/B intsst.f90 - int routine for conventational sst prepsst.f90 - initial setup for conventional sst read_wrf_mass_files.F90 - same as read_wrf_nmm_files but for wrf mass-core read_wrf_mass_guess.F90 - same as read_wrf_nmm_guess but for wrf mass-core setupsst.f90 - prepare rhs for conventional sst snwem_amsu.f90 - calculate snow emissivity for AMSU-A/B sprsst.f90 - create stuff for inner iteration for conventional sst stpsst.f90 - step size routine for conventational sst unfill_mass_grid2.f90 - same as fill_nmm_grid2, but for wrf mass-core model wrwrfmassa.f90 - same as wrwrfnmma but for wrf mass-core FILES MODIFIED -------------- Atmospheric_Properties.f90 - remove CVS modification history Compare_Float_Numbers.f90 - remove CVS modification history EmisCoeff_Binary_IO.f90 - remove CVS modification history EmisCoeff_Define.f90 - remove CVS modification history Geopotential.f90 - remove CVS modification history IRSSE_Coefficients.f90 - remove CVS modification history IRSSE_Model.f90 - remove CVS modification history Level_Layer_Conversion.f90 - remove CVS modification history Profile_Utility_Parameters.f90 - remove CVS modification history SpcCoeff_Binary_IO.f90 - remove CVS modification history SpcCoeff_Define.f90 - remove CVS modification history TauCoeff_Binary_IO.f90 - remove CVS modification history TauCoeff_Define.f90 - remove CVS modification history Units_Conversion.f90 - remove CVS modification history absorber_profile.f90 - remove CVS modification history balance.f90 - remove passed variables not altered by routine; add use only and intent in/out berror.f90 - move k loop from glbsoi.f90 into set_ozone_var; add changes to allow gridded fields of grid increment dx,dy; increase recursive filter table size and changes to load the indexes from the redefined table (*medium) bkerror.f90 - add use onlh and intent in/out; add code to properly handle periodic subdomains; general code cleanup coefficient_utility.f90 - remove CVS modification history compute_pcp_grad.f90 - remove unneeded variables in call grdsphdp constants.f90 - add several new numeric constants deter_subdomain.f90 - add code to identify periodic subdomain; add use only and intent in/out deternpe.f90 - add code to correctly handle periodic subdomain; add use only and intent in/out dprodx.F90 - add use only and intent in/out emiss.f90 - incorporate snow/ice emissivity for AMSU-A/B & SSM/I (*major) error_handler.f90 - remove CVS modification history file_utility.f90 - remove CVS modification history forward_model.f90 - remove CVS modification history fundamental_constants.f90 - remove CVS modification history g2s1.F90 - add use only and intent in/out gengrid_vars.f90 - redefinition of rlons, rlats (*minor) getlalo.F90 - add use only getpln.f90 - add use only and intent in/out glbsoi.f90 - correct logic around call to set_ozone_var (*medium); add calls for wrf mass core option grdsphdp.f90 - remove noq,coef from calling list since they are available from module berror grid2sub.f90 - add use only and intent in/out; generalize to handle periodic subdomains gridmod.f90 - extensive changes to incorporate generalized transform from earth lat lon to regional coordinates when in regional mode. remove/redefine many variables related to this transform (*medium); convert doc block to protex-compliant form gsimain.F90 - modify to include conventional sst; remove some outdated namelist variables; update documentation; convert doc block to protex-compliant form gsisub.f90 - add test for wrf_mass_regional logical variable (for mass core run) guess_grids.f90 - add vertical coordinate in wrf mass core for getting 3d pressure field; eliminate dependence of regional mode on hybrid parameter; convert doc block to protex-compliant form hoper.f90 - add use only and intent in/out hopers.f90 - convert doc block to protex-compliant form htoper.f90 - add use only and intent in/out inisph.f90 - add use only and intent in/out; remove variables noq,coef since these are available from module berror initialize.f90 - remove CVS modification history intall.F90 - modify to include conventional sst intdw.f90 - add use only and intent in/out intlimq.f90 - add use only and intent in/out intoz.f90 - add use only and intent in/out; change array indices from 2d to 1d intpcp.f90 - add use only and intent in/out; rename variables intps.f90 - add use only and intent in/out intpw.f90 - add use only and intent in/out intq.f90 - add use only and intent in/out intrad.f90 - add use only and intent in/out; rename variables intref.f90 - add use only and intent in/out intrp2a.f90 - add use only and intent in/out intrp3.f90 - add use only and intent in/out intrp3oz.f90 - add use only and intent in/out intrppx.f90 - add use only and intent in/out intrw.f90 - add use only and intent in/out intspd.f90 - add use only and intent in/out intsrw.f90 - add use only and intent in/out intt.f90 - add use only and intent in/out intw.f90 - add use only and intent in/out jfunc.f90 - add use only; simplify argument list; move definition of xhat pointers into new routine -> set_pointer k_matrix_model.f90 - remove CVS modification history landem.f90 - change threshold for snow identification (*medium) locatelat_reg.f90 - simplify subroutine call list; add use only and intent in/out m1glat.f90 - add use only and intent in/out m1ipqr.f90 - add use only and intent in/out m1poly.f90 - add use only and intent in/out m1rcons.f90 - add use only and intent in/out mpimod.F90 - convert doc block to protex-compliant form obs_para.f90 - modify to include conventional sst obsmod.f90 - modify to include conventional sst; rename variables for radiance and precipitation observations pcgsoi.f90 - add use only and intent in/out; clean up subroutine calls pcpinfo.F90 - convert doc block to protex-compliant form polcarf.f90 - add use only precpd_ad.f90 - add use only and intent in/out predictors.f90 - remove CVS modification history prepdw.f90 - add use only and intent in/out prepp.f90 - add use only and intent in/out preppw.f90 - add use only and intent in/out prepq.f90 - add use only and intent in/out prepref.f90 - add use only and intent in/out preprw.f90 - add use only and intent in/out prepsrw.f90 - add use only and intent in/out prept.f90 - add use only and intent in/out prepw.f90 - add use only and intent in/out prewgt.f90 - redefine the RF-table and remove sea-land contrast on horizontal scales for surface temp: sst, ice and land (*medium) prewgt_reg.f90 - change to allow general dx,dy fields; redefine the recursive filter table and remove sea-land contrast on horizontal scales for surface temperature: sst, ice and land (*medium) psichi2uv_reg.f90 - change to allow general dx,dy fields; redefine the recursive filter table and remove sea-land contrast on horizontal scales for surface temperature: sst, ice and land (*medium) psichi2uvt_reg.f90 - change to allow general dx,dy fields (*medium) qcmod.f90 - modify to include conventional sst radiance.f90 - remove CVS modification history radinfo.f90 - convert doc block to protex-compliant form rdgesfc.f90 - add use only and intent in/out rdgesig.f90 - add use only and intent in/out rdgstat_reg.f90 - eliminate unused variables; add mass core vertical coordinate; eliminate reference to hybrid parameter; remove recalculation of sigl (*minor) rdsfull.f90 - add use only and intent in/out read_airs.f90 - make changes to eliminate obs. earlier in thinning read_avhrr_navy.f90 - make changes to eliminate obs. earlier in thinning read_bufrtovs.f90 - make changes to eliminate obs. earlier in thinning read_files.F90 - add use only and intent in/out read_goesimg.f90 - make changes to eliminate obs. earlier in thinning read_goesndr.f90 - make changes to eliminate obs. earlier in thinning read_gps_ref.f90 - add use only and intent in/out read_guess.f90 - add use only and intent in/out read_ieeetovs.f90 - make changes to eliminate obs. earlier in thinning read_lidar.f90 - add use only and intent in/out read_obs.f90 - modify to include conventional sst read_ozone.f90 - add use only and intent in/out read_pcp.f90 - add use only and intent in/out read_prepbufr.f90 - modify to include conventional sst read_radar.f90 - add use only and intent in/out read_superwinds.f90 - add use only and intent in/out read_wrf_nmm_files.F90 - add use only and intent in/out read_wrf_nmm_guess.f90 - add use only and intent in/out s2g1.F90 - add use only and intent in/out satthin.F90 - make changes to eliminate obs. earlier in thinning (*minor) sensor_planck_routines.f90 - remove CVS modification history setupdw.f90 - convert doc block to protex-compliant form setupoz.f90 - correct bugs in ozone diagnostics; convert doc block to protex-compliant form setuppcp.f90 - eliminate dependence of regional mode on hybrid parameter; convert doc block to protex-compliant form; rename variables passed to intpcp and stppcp via obsmod setupps.f90 - add use only and intent in/out setuppw.f90 - add use only and intent in/out setupq.f90 - add use only and intent in/out setuprad.f90 - correct inconsistency where txy2ll called in global mode. this was OK before, but now gives unpredictable results because tll2xy and txy2ll can only be used now in regional mode (*minor change possible, but no change noticed in test case); incorporate new snow/ice emissivity model (*major); rename variables passed to intrad and stprad via obsmod setupref.f90 - add use only and intent in/out setuprhsall.f90 - modify to include conventional sst setuprw.f90 - add use only and intent in/out setupspd.f90 - add use only and intent in/out setupsrw.f90 - add use only and intent in/out setupt.f90 - convert doc block to protex-compliant form setupw.f90 - convert doc block to protex-compliant form smoothrf.f90 - change call sub2grid and grid2sub to be consistent with changes made in these routines smoothwwrf.f90 - add only to use constants smoothzrf.f90 - add only to use constants spectral_coefficents.f90 - remove CVS modification history sproz.f90 - change indexing for intoz and stpoz from 2d to 1d sprq.f90 - reformat code spruv.f90 - convert doc block to protex-compliant form statsconv.f90 - modify to include conventional sst statsoz.f90 - add use only and intent in/out statspcp.f90 - add use only and intent in/out statsrad.f90 - add use only and intent in/out stpcalc.f90 - correct harmless bug on comment out write statement; add conventional sst stpdw.f90 - add use only and intent in/out stplimq.f90 - add use only and intent in/out stpoz.f90 - add use only and intent in/out; change array indices from 2d to 1d stppcp.f90 - add use only and intent in/out; rename variables stpps.f90 - add use only and intent in/out stppw.f90 - add use only and intent in/out stpq.f90 - add use only and intent in/out stprad.f90 - add use only and intent in/out; rename variables stpref.f90 - add use only and intent in/out stprw.f90 - add use only and intent in/out stpspd.f90 - add use only and intent in/out stpsrw.f90 - add use only and intent in/out stpt.f90 - add use only and intent in/out stpw.f90 - add use only and intent in/out stvp2uv.f90 - add use only and intent in/out sub2grid.f90 - add use only and intent in/out; generalize to handle periodic subdomains tbalance.f90 - correct bug in q normalization tintrp2a.f90 - add use only and intent in/out tintrp3.f90 - add use only and intent in/out tpause.f90 - add use only and intent in/out; remove horizontal smoothing of tropopause pressure on subdomain transmittance.f90 - remove CVS modification history transmittance_coefficients.f90 - remove CVS modification history tstvp2uv.f90 - add intent in/out type_kinds.f90 - remove CVS modification history update_ggrid.f90 - add use only and intent in/out write_all.f90 - add call to wrwrfmassa for wrf mass core regional runs; convert doc block to protex-compliant form wrsfca.F90 - add use only and intent in/out wrsiga.f90 - convert doc block to protex-compliant form wrwrfnmma.f90 - add use only and intent in/out MAKEFILE CHANGES ---------------- a) Makefile - rename bufr library export procedure (GMAO-specific change) - fix doc target - The following routines are added - emiss_ssmi.f90, fill_mass_grid2, iceem_amsu.f90, intsst.f90, prepsst.f90, read_wrf_mass_files.F90, read_wrf_mass_guess.F90, setupsst.f90, snwem_amsu.f90, sprsst.f90, stpsst.f90, unfill_mass_grid2.f90, wrwrfmassa.f90 b) Makefile.dependency - update dependencies to be consistent with Makefile and module use within routines SCRIPT CHANGES -------------- a) Both global and regional - Increase ndat to 29. data type 29 is conventional sst data. The data is not yet available, but the script is updated to process the data when it becomes available. b) Regional only - Rename rungsi_regional.sh rungsi_regional_nmm.sh - Add rungsi_regional_mass.sh (runs the regional gsi using the mass core) - Namelist variables order_a2e, order_e2a, nsoil_regional removed. - Namelist variable wrf_mass_regional added to allow use of wrf mass core model. - Namelist variables as(1), as(2), and hs changed to allow for new general map transform in regional mode. old: as(1) = .1, as(2) = .1, hs = 1. new: as(1) = .2, as(2) = .2, hs = 2. FIX FILE CHANGES ---------------- none TEST RESULTS ------------ Old code: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs.200407/ New code: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/ Machine : NCEP IBM SP (snow) GLOBAL TEST ----------- Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_global.sh Max memory old: 522428 Kbytes new: 538976 Kbytes Wall clock old: 1611.863701 seconds new: 1579.919482 seconds Output from the first iteration of the minimization old: 0: penalty,grad ,a,b= 1 0 0.350584852791143407E+06 0.807122382995729800E+07 0.218479873761016300E-02 0.000000000000000000E+00 new: 0: penalty,grad ,a,b= 1 0 0.354065267633119307E+06 0.825850122942353133E+07 0.213195093659705997E-02 0.000000000000000000E+00 Output from the final iteration of the minimization old: 0: penalty,grad ,a,b= 3 1 0.193903612918548897E+06 0.222815435696889699E+04 0.178682412289512312E-02 0.285475311901454631E+01 new: 0: penalty,grad ,a,b= 3 1 0.196369704129523452E+06 0.917160780007132416E+04 0.763019126692540336E-03 0.109842681784328544E+01 REGIONAL NMM TEST ----------------- Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_regional_nmm.sh Max memory old: 211340 Kbytes new: 222664 Kbytes Wall clock old: 431.702256 seconds new: 422.867030 seconds Output from the first iteration of the minimization old: 0: penalty,grad ,a,b= 1 0 0.609021349776586794E+05 0.108217177460558689E+07 0.514281926838884313E-02 0.000000000000000000E+00 new: 0: penalty,grad ,a,b= 1 0 0.608968963386026371E+05 0.153603012798869400E+07 0.273709247706200361E-02 0.000000000000000000E+00 Output from the final iteration of the minimization old: 0: penalty,grad ,a,b= 3 1 0.481068072604959161E+05 0.251320633109194314E+05 0.432983211053808437E-02 0.152119733929951345E+00 new: 0: penalty,grad ,a,b= 3 1 0.482418841985237668E+05 0.342297592109804464E+05 0.381681155726046936E-02 0.218324209335197472E+00 REGIONAL MASS TEST ----------------- Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_regional_mass.sh Max memory old: not available new: 243900 Kbytes Wall clock old: not available new: 361.168627 seconds Output from the first iteration of the minimization old: not available new: 0: penalty,grad ,a,b= 1 0 0.414164466875986909E+05 0.264287988596970215E+06 0.123708632254375585E-01 0.000000000000000000E+00 Output from the final iteration of the minimization old: not available new: 0: penalty,grad ,a,b= 3 1 0.349100024880398778E+05 0.226341499239726872E+05 0.175150397314721082E-01 0.315632904519947188E+00