Skip Navigation Links www.nws.noaa.gov
NOAA logo - Click to go to the NOAA homepage National Weather Service NWS logo - Click to go to the NWS homepage
EMC Logo
Navigation Bar Left Cap Home News Organization
Navigation Bar End Cap

         MISSION / VISION    |    About EMC

EMC > NEMS > RUNNING NEMS
NOAA Environmental Modeling System

Running NEMS



This page is meant to guide you in setting up and running experiments for the different models within NEMS. If you are interested in starting out with regression tests, please refer to 'Getting Started' . Bear in mind that, currently, there is no simplified way of running the NMM-B or GFS individually. It may require changes to the regression test script (RT.sh), the ability to recognize other files involved, creation of your own run scripts, and the creation of your own compatible input files.

Libraries & Dependencies


Currently, NEMS requires the following external libraries to be installed on the system in order to run properly.
  • ESMF versions 3.1.0rp2 and 5.2.0rp1 are supported. Available from the ESMF web site.
  • netcdf - available from the UNIDATA website.
  • The following libraries are from NCEP and can be downloaded from the NEMS trunk (located in EMC's Subversion repository) in /trunk/util/syslib.tar:
    • bacio
    • w3
    • sp
    • nemsio
  • The following will be used only when write_dopost is set to .true. in the configure file:
    • g2 - /nwprod/lib
    • g2tmpl - /nwprod/util/sorc/g2tmpl.fd
    • sigio, sfcio - /nwprod/lib
    • unipost - /climate/save/wx20wa/esmf/nems/unipost_r14561
    • png, jasper - /usrx/local/64bit/lib; jasper is the library to make jpeg2000 packing
The last two libraries listed above may already be installed on your system (Linux), but may not be by default. PNG, in turn, requires zlib, but that is almost always installed, at least on Linux. They are required for compilation of the g2 (GRIB2) library.



Check Out and Build NEMS:


If you are interested in just running regression tests, please refer to 'Regression Tests'. Otherwise, continue on below.

Process to checkout and build NEMS:
ActionActive DirectoryCommand
_______________________________________________________________________________________________________________
Check out NEMS./gpfs/td1/emc/**/save/Firstname.Lastname (Tide)
/gpfs/gd1/emc/**/save/Firstname.Lastname (Gyre)
/scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname (Zeus)
* - 1 or 2
** - global, meso, etc.
svn checkout https://svnemc.ncep.noaa.gov/projects/nems/trunk
_______________________________________________________________________________________________________________
Configure using the version of the ESMF library that will be used (if running RT.sh, this will be done automatically). Make note of machine specific options..../trunk/srcESMF 3.1.0rp2:
./configure 3_wcoss
./configure 3_zeus (for the ESMF 3.1.0r series library)
./configure 3_yellowstone

ESMF 6.3.0r:
./configure 6.3r_nmm_wcoss
./configure 6.3r_gsm_wcoss
./configure 6_nmm_zeus
./configure 6_gsm_zeus
./configure 6_yellowstone

ESMF with reference NUOPC layer (currently ESMF 7.0.0 beta snapshot):
./configure nuopc_zeus
./configure nuopc_gaea
_______________________________________________________________________________________________________________
Source the appropriate module file. Note the machine and ESMF version needed..../trunk/srcsource conf/modules.nems.Gaea_ESMF_NUOPC
source conf/modules.nems.wcoss_ESMF_3
source conf/modules.nems.wcoss_ESMF_630rAPI_gsm
source conf/modules.nems.wcoss_ESMF_630rAPI_nmm
source conf/modules.nems.Zeus_ESMF_310rAPI
source conf/modules.nems.Zeus_ESMF_630rAPI_gsm
source conf/modules.nems.Zeus_ESMF_630rAPI_nmm
source conf/modules.nems.Zeus_ESMF_NUOPC
_______________________________________________________________________________________________________________
Clean any previous compilations..../trunk/srcgmake clean
_______________________________________________________________________________________________________________
Compile and build the core(s) needed (/job/tests/rt.sh will build the default settings - NMM and GFS with GOCART)..../trunk/srcNMM & GSM: gmake nmm_gsm
GEN only: gmake gen
GSM & GEN w/ GOCART & post:gmake gsm_gen_post GOCART_MODE=full
NMM & GSM w/ GOCART: gmake nmm_gsm GOCART_MODE=full
NMM only: gmake nmm
NMM with post: gmake nmm_post
GSM only: gmake gsm
GSM with post: gmake gsm_post
GSM w/ GOCART:gmake gsm GOCART_MODE=full
In order to build NEMS with ocean components, all of the above targets support the optional OCN variable. Supported values are: \"dummy\", \"hycom\", and \"mom5\". Specifying multiple values, e.g. "gmake nmm_gsm OCN=mom5,dummy", is supported.

In order to build NEMS with sea-ice components, all of the above targets support the optional ICE variable. Supported values are: \"dummy\", \"cice\".

Specifying multiple values, e.g. \"gmake nmm_gsm OCN=mom5,dummy,hycom ICE=dummy\", is supported.

The executable, NEMS.x, will appear in /exe.



How To Run NMM-B


Using regression test script on WCOSS and Zeus (coming soon):

Using the NMM-B Launcher (coming soon):


The state of the Launcher on WCOSS is in a state of flux. For reference, please refer to the evolving draft of Launcher documentation.


How To Run NMM-B Nests


The NMM-B has telescoping static and moving nest capability. All domains, whether the uppermost parent or any nest, are functionally equivalent and thus each needs its own configure file. Both 1-way and 2-way interaction between parent and child domains are available. In addition to running nests using the regression test input and configuration (similar to the method mentioned in 'How to Run NMM-B'), one can also follow as specified below.

The following instructions are also available in the README.NMM file.

1-way nesting: [+/-]


2-way nesting: [+/-]


NMM-B output files are written in a gridded format readable by GrADS (.ctl). To understand NMM-B output, please refer to the 'Output' tab.



How to Run GFS


On WCOSS:


Currently, the scripts to run the GFS with NEMS are in flux. Follow the temporary instructions below to run, but bear in mind you must have a Subversion account to follow these directions.

GFS Parallel Scripts:

All script work is being done in the GFS branch, gfs4nems. Make sure to use "--ignore-externals" when checking out the branch, unless you want to check out all of the externals, as well. Also, if you have not been tasked to work on any of the branches mentioned in this section, please do not commit anything back to the branches.

Please note that work will continue to be committed back to this branch, so you will want to follow it with ticket #184. Any questions about the scripts can be directed to Kate Howard.

Config files:

Currently, para_config_T574 in the gfs4nems branch only runs Eulerian. You will want to adjust the BASE_NEMS variable to point to your copy of NEMS and BASEDIR to point to your copy of the gfs4nems branch.

Note the new $NEMS variable near the top. When NEMS=YES, it sets DYNVARS and PHYVARS for NEMS.x and when NEMS=NO, it sets FCSTVARS for global_fcst.

para_config_T1534 will be updated to run semi-Lagrangian soon.

GSM Code:

All model code work is being done in the GSM branch, gsm_slg. All questions about the code should be directed to Moorthi. How to build the current NEMS/GSM code:
  1. Check out the NEMS trunk and make sure NOT to ignore externals. You want to grab the GSM code trhough the external under /src/atmos.
  2. Update the GSM external under /src/atmos to point to Moorthi's branch if you want to run semi-Lagrangian. Leave the external alone if trying Eulerian first.
    1. Type "svn propedit svn:externals"
      (Seek help if you are unsure about how to do this.)
    2. Type "svn update" after changing the external. That will change the contents of the GSM directory under /src/atmos to Moorthi's branch.
  3. To build, do the following under /src:
    1. Type "./configure ESMF_OPTION". (Typing just "./configure" will show options, but you will want 6.3r_gsm_wcoss for the ESMF_OPTION for now.)
    2. Type "source conf/modules.nems".
    3. Type "gmake gsm".
    4. You will get a NEMS.x in the /exe directory. Use that instead of global_fcst.
global_chgres:

The global_chgres code and script within the gfs4nems GFS branch is new and working. You should build and use that copy to convert your sigio ICs to nemsio. There are a number of wrapper scripts floating around that can be used to run chgres offline. Feel free to use one you already have or try this one, /gpfs/td1/emc/global/noscrub/Kate.Howard/para_gfs/data_4_gfs4nems/run_chgres.sh.

Notes:
  1. Attempts have been unsuccessful using global_chgres straight from sigio to nemsio with different starting and ending resolutions (i.e. T1534 sigio to T574 nemsio). You may need to first convert the original sigio file to the resolution you want (e.g. T1534 sigio to T574 sigio, OUTTYP=2, no GRDFMT) and then convert it to nemsio (e.g. T574 sigio to T574 nemsio, OUTTYP=1).
  2. You can NOT currently convert nemsio back to sigio. Keep that in mind!
  3. Attemps have also been unsuccessful in using global_chgres to change nemsio files to other resolutions. The only surefire way, so far, is to follow the steps in note #1.
  4. Make sure to go through the run_chgres.sh script to edit the settings to your preferences. For OUTTYP=1 (nemsio), you need GFSOUT instead of SIGOUT.
  5. For nemsio, the siganl file is named gfsanl. The scripts are set up to look for gfsanl if NEMSIO_IN=.true. and siganl if SIGIO_IN=.true.
  6. The sfcanl file does not change, but needs to be converted as well.
  7. Most likely, nemsio files will be used in binary format. You will notice GRDFMT='bin4' in the CHGRESVARS within the scripts.
nemsio_get:

The nemsio_get utility is used for nemsio files, just as global_sighdr is used for sigio files. You can use the following copy on Tide, /nwprod/ngac.v1.0.0/exec/nemsio_get. All questions about nemsio_get should be directed to Jun Wang.

Status of Eulerian and semi-Lagrangian:

The current NEMS trunk has been successfully used to run Eulerian sigio to nemsio forecasts (SIGIO_IN=.true. and NEMSIO_OUT=.true.) for all supported resolutions under T878. Eulerian nemsio to nemsio (NEMSIO_IN=.true. and NEMSIO_OUT=.true.) has also been successful for T574. The gsm_slg branch has just started to be used to run semi-Lagrangian T1534, with a few glitches that will hopefully be remedied soon.

On Zeus:


In addition to isolating specific GFS regression tests in RT.sh_IN, as mentioned above, one can also use the following method to run GFS on Zeus.

As part of the 2013 NEMS/GFS Modeling Summer School, a suite of scripts and sample initial conditions was created to allow students to run the NEMS GFS. The code is located in /scratch2/portfolios/NCEPDEV/global/save/glopara/Summer_School_2013. Below is information on how to use the code:
  1. Move to your working directory, then use the following command to copy necessary files: 'sh /scratch2/portfolios/NCEPDEV/global/save/glopara/Summer_School_2013'.
  2. Compile NEMS/GFS code:
    • cd Summer_School_2013/Tutorials
    • sh makefile_nems.sh
      • Compile log can be viewed in ../sorc/global_fcst.fd/Compile.log
  3. Make sure your global_fcst executable is located in Summer_School_2013/exec.
  4. Submit an experiment by using 'sh submit_nems.sh' in your /Tutorials directory. By default, an experiment will run at current operational T574 resolution.
    • Make changes to the following corresponding to your account:
      • 'export acc='
      • 'export NEMSDIR='
      • 'export RUNDIR='
  5. Output, by default, can be found in the directory defined by OUTDIR in submit_nems.sh.
Sample input files are provided in /scratch2/portfolios/NCEPDEV/global/noscrub/glopara/Summer_School_2013_ICs.


How to Run FIM (coming soon)






Porting Information


Following is a procedure for defining functional equivalence on different environments:
  1. Select key fields from the import state Fi (may include a selection of initial cases).
  2. Run control run C of application on control environment.
  3. Perturb each Fi in the last significant digit.
    1. Nominally after the 6th decimal digit for 32-bit reals.
    2. For each value Fi(x), construct a distribution.
    3. Gaussian distribution with width being | Fi(x)/1.e6|.
    4. Randomly select perturbation from that distribution.
  4. Run perturbed run P from perturbed import state on control environment.
  5. Select key fields from the export states Cj (control run) and Pj (perturbed run) - may include a selection of output times.
  6. Determine area weight w(x)
  7. Compute the butterfly vector .
  8. Butterfly vector will have units of original export fields.
  9. Now run control run but on different environment D.
  10. Compute difference vector like butterfly vector but using D instead of P.
  11. Compute dimensionless difference butterfly number vector .
  12. For a bit-identical requirement, vj must be zero.
  13. For a functionally equivalent requirement, vj must be largely less than 10.
The presentations listed below show preliminary results from porting parts of NEMS to other computers:
Contributors: Eugene Mirvis, Tom Black, Ed Colon, Jun Wang