CHANGE LOG FOR UNIFIED 3DVAR CODE ------------------------------------------- Author(s) : Lidia Cucurull, John Derber, Dave Parrish, Russ Treadon, Wan-Shu Wu Reviewer(s) : Russ Treadon Date : 8 December 2004 GMAO CVS tag: NCEP CVS tag: ncep-gsi-2004_12 REASON FOR CHANGES ------------------ Changes are made to the gsi code for the following reasons: (listing below not given in any particular order) a) Include some openMP directives allowing code to run faster on machines where this option is available. b) Include regularity in table definition for horizontal recursive filter (allows improved reproducibility when using different number of processors) c) Replace weighting of observations around boundary with logical true false flag. (Also allows removal of *_loc intermediate files in obs_para) d) Modify regional input-output to use mpi i-o for wrf binary format files. This is being done to improve efficiency of input and output (better memory usage and less time spent reading and writing large files). The test cases show no clear impact, but the impact should be substantial when running over the full operational North American domain. e) Expand variance array "dssv" to include longitude dimension. Variances are currently read in from the background error statistics file as 2d fields (vary with latitude and vertical level). With this update prewgt.f90 and prewgt_reg.f90 simply replicate the 2d variance field in the longitudinal direction. f) Declare parameters for the smallest and largest machine-representable floating point numbers (parameters huge_single, huge_r_kind, tiny_single, tiny_r_kind). Separate derived constants from regional/global-only constants. g) The following changes are made to the use of GPS local refractivity - include increments for surface pressure in control vector - include increments for virtual temperature for all levels below the observation - make calculation of pressure at observation location consistent with tangent linear and adjoint operators - implement full non-linear operator for refractivity rather than using the linear approximation. The necessity for this came out of tests noting that the non-linear effects are small but significant, especially in terms of the minimization. h) Replace IBM extensions (i.e., call mpe_*) to MPI standard with standard (i.e., call mpi_*) MPI routines. SPECIAL NOTES ------------- a) Use of OpenMP requires changes to script and makefile b) Diagnostic files no longer have multiple entries for observations on boundaries of regions. All "weight" variables in observation diagnostic files are set equal to one. At a later date the "weight" variable will be removed from the observation diagnostic files since it is no longer needed. EXPECTED DIFFERENCES -------------------- Analysis results from this version differ from those obtained using the previous version (ncep-gsi-2004_11) for the following reasons: (note that the listing below correspond letter-for-letter with the reasons for changes above) a) In putting in OpenMP directives, the order of caclulations in certain int*f90 and stp*f90 routines was changed. This leads to round-off differences in the final results b) Some small differences due to different table ganularity c) Round-off differences related to removal of fractional weights associated with sub-domains. The new code replaces these weights with a logical flag d) No differences, except for binary mass core, because there was an error in wrf_binary_interface.f90, (subroutine convert_binary_mass) in the previous version (remove line 532 of old version to correct error). line 532 is : read(in_unit,iostat=status)field3 When this error is corrected in the previous version, the results are the same. e) This change, by itself, produces no change f) This change only affects special case in which bad surface pressure or moisture observation is rejected by gross check. g) Changes in the use of GPS local refractivity only affect results when assimilating this data. This data is not available in the (current) suite of gsi test cases. Hence most users will note no difference related to these changes. h) No differences FILES REMOVED ------------- deternpe.f90 - functionality moved to disobs in obs_para.f90 gaussian.f90 - was used with gps data --> no longer needed intall.F90 - replaced by intall.f90 intall_qc.F90 - replaced by intall_qc.f90 read_files.F90 - replaced by read_files.f90 read_wrf_mass_files.F90 - replaced by read_wrf_mass_files.f90 read_wrf_nmm_files.F90 - replaced by read_wrf_nmm_files.f90 wrsfca.F90 - replaced by wrsfca.f90 FILES ADDED ----------- intall.f90 - replace mpe_iallreduce with mpi_allreduce; add arguments for call to intref intall_qc.f90 - replace mpe_iallreduce with mpi_allreduce; add brightness temperatures to nonlinear quality control; add arguments for all to intref_qc read_files.f90 - replace mpe_ibcast with mpi_bcast read_wrf_mass_files.f90 - replace mpe_ibcast with mpi_bcast read_wrf_nmm_files.f90 - replace mpe_ibcast with mpi_bcast wrsfca.f90 - replace mpe_igatherv with mpi_gatherv FILES MODIFIED -------------- berror.f90 - add longitude dimension to variance array dssv; modify to allow regular horizontal recursive error bkgcov.f90 - modify for OpenMP bkgvar.f90 - add longitude dimension to variance array dssv; modify for OpenMP constants.f90 - add parameters for tiny and huge floating point numbers; separate dervied constants from regional/global-only constants dprodx.F90 - replace mpe_iallreduce with mpi_allreduce emiss.f90 - modify for OpenMP glbsoi.f90 - add code to handle regional netcdf i/o gridmod.f90 - add netcdf logical; convert rlon_min_ll and rlat_min_ll to radians gsimain.F90 - add comments for error codes 74 and 75; initialize variables wrf_anl_filename and wrf_ges_filename; remove code to handle regional binary update intdw.f90 - remove weights add logical for boundary points intoz.f90 - remove weights add logical for boundary points intpcp.f90 - remove weights add logical for boundary points intps.f90 - remove weights add logical for boundary points intpw.f90 - remove weights add logical for boundary points intq.f90 - remove weights add logical for boundary points intrad.f90 - remove weights add logical for boundary points; alter order of operations in gradient calculation intref.f90 - remove weights add logical for boundary points; adjoint modified to be consistent with new GPS refractivity forward model intrppx.f90 - modify for openMP intrw.f90 - remove weights add logical for boundary points intspd.f90 - remove weights add logical for boundary points intsrw.f90 - remove weights add logical for boundary points intsst.f90 - remove weights add logical for boundary points intt.f90 - remove weights add logical for boundary points intw.f90 - remove weights add logical for boundary points obs_para.f90 - remove weights add logical for boundary points and rewrite to eliminate need to write *_loc files obsmod.f90 - remove weights add logical for boundary points; add new fields needed in intref and stpref pcpinfo.F90 - replace mpe_iscatterv with mpi_scatterv prepdw.f90 - remove weights add logical for boundary points prepq.f90 - remove weights add logical for boundary points prepref.f90 - remove weights add logical for boundary points; new operator to compute pressure at observation preprw.f90 - remove weights add logical for boundary points prepsrw.f90 - remove weights add logical for boundary points prepsst.f90 prept.f90 - remove weights add logical for boundary points prepw.f90 - remove weights add logical for boundary points prewgt.f90 - add longitude dimension to variance array dssv; move table definition stuff to berror prewgt_reg.f90 - add longitude dimension to variance array dssv; move table definition stuff to berror qcmod.f90 - add initialization for b_ref and pg_ref radinfo.f90 - change 110 format statement read_bufrtovs.f90 - add logic to execute newchn in appropriate cases read_gps_ref.f90 - new fields are read read_pcp.f90 - add code to read ssmi rain rates from prepbufr file read_prepbufr.f90 - add deallocate(etabl) for regional case read_wrf_mass_guess.f90 - replace subroutine read_wrf_mass_guess with two subroutines; read_wrf_mass_binary_guess, and read_wrf_mass_netcdf_guess; read_wrf_mass_netcdf_guess is identical to the old read_wrf_mass_guess, but read_wrf_mass_binary_guess uses mpi i-o to read directly from a wrf binary guess file read_wrf_nmm_guess.f90 - same as read_wrf_mass_guess.f90 but for nmm setupdw.f90 - remove weights add logical for boundary points setupoz.f90 - remove weights add logical for boundary points setuppcp.f90 - remove weights add logical for boundary points setupps.f90 - remove weights add logical for boundary points setuppw.f90 - remove weights add logical for boundary points setupq.f90 - remove weights add logical for boundary points setuprad.f90 - remove weights add logical for boundary points setupref.f90 - remove weights add logical for boundary points; non-linear operator for refractivity has been cleaned up. Arrays needed for the adjoint calculation are also computed and stored. setuprw.f90 - remove weights add logical for boundary points setupspd.f90 - remove weights add logical for boundary points setupsrw.f90 - remove weights add logical for boundary points setupsst.f90 - remove weights add logical for boundary points setupt.f90 - remove weights add logical for boundary points setupw.f90 - remove weights add logical for boundary points smoothrf.f90 - modify for OpenMP smoothzrf.f90 - add longitude dimension to variance array dssv sprdw.f90 - remove weights add logical for boundary points sproz.f90 - remove weights add logical for boundary points sprp.f90 - remove weights add logical for boundary points; add min,max bounds observation error sprpw.f90 - remove weights add logical for boundary points sprq.f90 - remove weights add logical for boundary points; add min,max bounds observation error sprref.f90 - remove weights add logical for boundary points sprrw.f90 - remove weights add logical for boundary points sprspd.f90 - remove weights add logical for boundary points sprsrw.f90 - remove weights add logical for boundary points sprsst.f90 - remove weights add logical for boundary points sprt.f90 - remove weights add logical for boundary points spruv.f90 - remove weights add logical for boundary points statsrad.f90 - change 1102 format statement stpcalc.f90 - correct comment; add arguments for call to sptref; GPS refractivity is now treated as a nonlinear term stpcalc_qc.f90 - add brightness temperatures to nonlinear quality control; add arguments for call to stpref_qc; GPS refractivity is now treated as a nonlinear term stpdw.f90 - remove weights add logical for boundary points stplimq.f90 - modify for openMP --> change order of calculations --> round off stpoz.f90 - remove weights add logical for boundary points stppcp.f90 - remove weights add logical for boundary points,add openMP stpps.f90 - remove weights add logical for boundary points stppw.f90 - remove weights add logical for boundary points stpq.f90 - remove weights add logical for boundary points stprad.f90 - remove weights add logical for boundary points,add openMP; change order of calculations in penalty calculation stpref.f90 - remove weights add logical for boundary points; modified to be consistent with new GPS refractivity forward model stprw.f90 - remove weights add logical for boundary points stpspd.f90 - remove weights add logical for boundary points,add openMP stpsrw.f90 - remove weights add logical for boundary points stpsst.f90 - remove weights add logical for boundary points stpt.f90 - remove weights add logical for boundary points stpw.f90 - remove weights add logical for boundary points wrf_binary_interface.f90 - add check on input wrf guess file, stop code if problem; replace reading of wrf binary guess and rewriting to temporary file with inventory of wrf binary file and writing of byte offset information and some 2d fields for for later use by mpi i-o to read and write fields directly from the wrf binary file; remove subroutines update_binary_mass and update_binary_nmm wrf_netcdf_interface.f90 - add return codes 74 and 75 for problems with regional i/o write_all.f90 - modify regional calls for netcdf and binary output wrsiga.f90 - replace 1000 with parameter r1000_single wrwrfmassa.f90 - rewrite for mpi-io wrwrfnmma.f90 - rewrite for mpi-io MAKEFILE CHANGES ---------------- Makefile - remove deternpe.f90, gaussian.f90 - replace intall.F90, intall_qc.F90, read_files.F90, read_wrf_mass_files.F90, read_wrf_nmm_files.F90, wrsfca.F90 with *f90 versions (replaced mpe_* calls, with standard mpi_* calls ... no need to run files through pre-processor during compile). Makefile.dependency - remove dependencies for deternpe.o, guassian.o - add gridmod.o to dependencies for obs_para.o - add jfunc.o to dependencies for wrwrfmassa.o - change *F90 dependencies for *f90 for intall.f90, intall_qc.f90, read_files.f90, read_wrf_mass_files.f90, read_wrf_nmm_files.f90, wrsfca.f90 Makefile.conf.AIX.omp Same as Makefile.conf.AIX but includes changes needed to build executable that makes use of OpenMP SCRIPT CHANGES -------------- Both global and regional scripts now copy the following fixed files from /nfsuser/g01/wx20rt/global_gsi/fix/ - global_satinfo.txt - global_satangbias.txt - global_pcpinfo.txt - prepobs_prep.bufrtable The last two fix files above (global_pcpinfo.txt and prepobs_prep.bufrtable) are identical with what is in NCEP operations (/nwprod/fix) as of 20041208. The first two file (global_satinfo.txt and global_satangbias.txt) are changed to be consistent with what will be implemented in the 1st quarter FY05 GFS-SSI implementation. FIX FILE CHANGES ---------------- Changes are made to two fix files, namely 1) global_satangbias.txt 2) global_satinfo.txt These files are changed to make them consistent with what will be implemented in the first quarter FY05 GFS-SSI implementation. Previous gsi versions copied these files from /nwprod/fix/ The key changes in global_satinfo.txt are two-fold a) observation error for numerous channels are (slightly) increased from what is in /nwprod/fix/global_satinfo.txt. This was done to given a previous change in setuprad (removal of 0.25 down-weighting factor in the 200411 gsi update). b) TURN ON assimilation of AQUA AIRS and EOS_AMSU-A brightness temperatures Values for numerous satellite/channel combinations are updated based on results of the ongoing GFS-SSI implementation off-line parallel. TEST RESULTS ------------ Old code: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs.200411/ New code: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/ Machine : NCEP IBM (blue) ***NOTE*** Run old2 uses the same code as run "old" (version ncep-gsi-2004_11) with the only exception being the use of the new global_satinfo.txt and global_satangbias.txt files (see Fix File Changes above) GLOBAL T254L64 TEST ------------------- Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_global.sh Max memory old : 497204 Kbytes (48 mpi tasks, blocking=unlimited) old2: 497200 Kbytes new : 513996 Kbytes Wall clock old : 1007.771243 seconds old2: 1006.262499 seconds new : 1011.793407 seconds Output from the first iteration of the minimization old : 0: penalty,grad ,a,b= 1 0 0.409226164549308131E+06 0.737270447721514944E+07 0.415846197458893951E-02 0.000000000000000000E+00 old2: 0: penalty,grad ,a,b= 1 0 0.407441789166940958E+06 0.672127530837542191E+07 0.490077507456235587E-02 0.000000000000000000E+00 new : 0: penalty,grad ,a,b= 1 0 0.407441789166940958E+06 0.672122661060058512E+07 0.490086107864976320E-02 0.000000000000000000E+00 Output from the final iteration of the minimization old : 0: penalty,grad ,a,b= 3 1 0.220481390285436093E+06 0.231680748359994686E+05 0.162007438466449318E-02 0.155903834313775502E+01 old2: 0: penalty,grad ,a,b= 3 1 0.218254369472698600E+06 0.149414057808893995E+05 0.292814877422529953E-02 0.122838802627630805E+01 new : 0: penalty,grad ,a,b= 3 1 0.218254527381437656E+06 0.149411492876785796E+05 0.292822773187458030E-02 0.122839623650184437E+01 REGIONAL NMM BINARY TEST ------------------------ Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_regional_nmm_binary.sh Max memory old : 509088 Kbytes (24 mpi tasks, blocking=unlimited) old2: 509104 Kbytes new : 509232 Kbytes Wall clock old : 567.330985 seconds old2: 558.842468 seconds new : 571.645303 seconds Output from the first iteration of the minimization old : 0: penalty,grad ,a,b= 1 0 0.401061189427129138E+05 0.520079195960925950E+06 0.269958768088938332E-02 0.000000000000000000E+00 old2: 0: penalty,grad ,a,b= 1 0 0.417459794731650545E+05 0.485790670239038009E+06 0.337102066890215520E-02 0.000000000000000000E+00 new : 0: penalty,grad ,a,b= 1 0 0.417459794731650472E+05 0.485790677769222355E+06 0.337102070813298212E-02 0.000000000000000000E+00 Output from the final iteration of the minimization old : 0: penalty,grad ,a,b= 3 1 0.329509225333884096E+05 0.118257860613835626E+06 0.151048232746914821E-02 0.317517630461349731E+00 old2: 0: penalty,grad ,a,b= 3 1 0.340852722479096265E+05 0.107983819646417673E+06 0.137827124210482287E-02 0.506068496072743623E+00 new : 0: penalty,grad ,a,b= 3 1 0.340852703012454003E+05 0.107983740927210281E+06 0.137827176081728128E-02 0.506068117680818275E+00 REGIONAL NMM NETCDF TEST ------------------------ Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_regional_nmm_netcdf.sh Max memory old : 509068 Kbytes (24 mpi tasks, blocking=unlimited) old2: 509060 Kbytes new : 509052 Kbytes Wall clock old : 570.245339 seconds old2: 575.902046 seconds new : 583.502807 seconds Output from the first iteration of the minimization old : 0: penalty,grad ,a,b= 1 0 0.434856108067335008E+05 0.971521955496094888E+06 0.184144347270626682E-02 0.000000000000000000E+00 old2: 0: penalty,grad ,a,b= 1 0 0.442922982479959028E+05 0.771470431850075838E+06 0.249435029546953273E-02 0.000000000000000000E+00 new : 0: penalty,grad ,a,b= 1 0 0.442922982479959101E+05 0.771470438897610176E+06 0.249435037456222390E-02 0.000000000000000000E+00 Output from the final iteration of the minimization old : 0: penalty,grad ,a,b= 3 1 0.354765374049383463E+05 0.208297783683200018E+06 0.101990556933870789E-02 0.108458716386790610E+01 old2: 0: penalty,grad ,a,b= 3 1 0.361960475078076051E+05 0.214005683246707631E+06 0.909712094782592386E-03 0.148463035271751553E+01 new : 0: penalty,grad ,a,b= 3 1 0.361960476567865117E+05 0.214005417525003082E+06 0.909713471571737859E-03 0.148462736362034375E+01 REGIONAL MASS BINARY TEST ------------------------- Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_regional_mass_binary.sh ***Note*** Run old2* uses version ncep-gsi-2004_11 with a bug fix to line 532 of wrf_binary_interface.f90. For more information see Expected Differences d) above Max memory old : 532788 Kbytes (24 mpi tasks, blocking=unlimited) old2 : 532740 Kbytes old2*: 532748 Kbytes new : 532780 Kbytes Wall clock old : 533.858389 seconds old2 : 535.579115 seconds old2*: 539.393215 seconds new : 542.008095 seconds Output from the first iteration of the minimization old : 0: penalty,grad ,a,b= 1 0 0.494185336918565081E+05 0.371636278829111904E+08 0.125371761799441931E-03 0.000000000000000000E+00 old2 : 0: penalty,grad ,a,b= 1 0 0.489908101341518268E+05 0.319963020953230970E+08 0.143758034940895841E-03 0.000000000000000000E+00 old2*: 0: penalty,grad ,a,b= 1 0 0.338927100665839826E+05 0.320291654854559489E+08 0.143801505090933204E-03 0.000000000000000000E+00 new : 0: penalty,grad ,a,b= 1 0 0.338927100665839826E+05 0.320291671463267989E+08 0.143801495166350239E-03 0.000000000000000000E+00 Output from the final iteration of the minimization old : 0: penalty,grad ,a,b= 3 1 0.238995439920104291E+05 0.145185006515976624E+06 0.378141400281192168E-03 0.106623224745615711E+00 old2 : 0: penalty,grad ,a,b= 3 1 0.248096487446142673E+05 0.962703602431149484E+05 0.579303440851839822E-03 0.687401574540842247E-01 old2*: 0: penalty,grad ,a,b= 3 1 0.191782010702939479E+05 0.791311478288523504E+05 0.663740629805496612E-03 0.588467668911551930E-01 new : 0: penalty,grad ,a,b= 3 1 0.191783404350827332E+05 0.795010406050478632E+05 0.656257400536789976E-03 0.590322265961654852E-01 REGIONAL MASS NETCDF TEST ------------------------- Script: /nfsuser/g01/wx20rt/global_gsi/sorc/gsi_cvs/rungsi_regional_mass_netcdf.sh Max memory old : 542840 Kbytes (24 mpi tasks, blocking=unlimited) old2: 542848 Kbytes new : 542884 Kbytes Wall clock old : 497.794331 seconds old2: 480.518360 seconds new : 480.98876 seconds Output from the first iteration of the minimization old : 0: penalty,grad ,a,b= 1 0 0.496156729273343226E+05 0.467930322082468891E+06 0.853369983105661291E-02 0.000000000000000000E+00 old2: 0: penalty,grad ,a,b= 1 0 0.503181793545898545E+05 0.412090074630687071E+06 0.105746467874470573E-01 0.000000000000000000E+00 new : 0: penalty,grad ,a,b= 1 0 0.503181793545898545E+05 0.412090121450347884E+06 0.105746441331344091E-01 0.000000000000000000E+00 Output from the final iteration of the minimization old : 0: penalty,grad ,a,b= 3 1 0.383767115934001122E+05 0.707503608737142931E+05 0.561133566118554410E-02 0.465576783934556315E+00 old2: 0: penalty,grad ,a,b= 3 1 0.376820173705630077E+05 0.268358424515335646E+05 0.616443119558214049E-02 0.361483257652960299E+00 new : 0: penalty,grad ,a,b= 3 1 0.376820116304352341E+05 0.268358440778415425E+05 0.616443254201253042E-02 0.361483313088322322E+00