NEMS DOCUMENTATION
NEMS User’s Guide

Table of Contents

Revision DREV73964

1. Introduction to NEMS

1.1 Basics

The NOAA Environmental Modeling System (NEMS) is the infrastructure underlying a coupled modeling system that supports predictions of Earth's environment at a range of time scales. Examples of other coupled modeling systems are the Community Earth System Model (CEEM) and the U.K Met Office Unified Model

A model component is a software representation of a physical domain or process, for example sea ice. It is often developed by a team of specialists in that domain. Model coupling is a software representation of feedbacks between physical processes. It involves modifying the exported fields of a component through grid, unit, temporal, and other transformations so that they can be used as the inputs for another component.

NEMS includes infrastructure for coupling model components representing major Earth system domains and processes. External model components have a primary repository that is not at the NCEP Envionmental Modeling Center (EMC). In general, model components are coupled through the NEMS mediator (in other coupled modeling systems this is often called the "coupler"). NEMS also includes some specialized mediators; for example, for space weather. In some cases in NEMS, the model components are coupled "in-line", meaning that they are called directly from another model component instead of having fields sent through the mediator.

NEMS can be assembled into a number of different modeling applications (often shortened to just applications). Modeling applications are associated with a purpose, like medium-range forecasting; a set of model components; and a set of parameters that represent a range of supported options, including grids and resolutions. Different NEMS modeling applications can have different types and numbers of model components. Also, the same physical domain may be represented by different model components in different modeling applications. For example, in some NEMS modeling applications the ocean component may be the HYbrid Coordinate Ocean Model (HYCOM) and in others it may be the Modular Ocean Model (MOM).

This spreadsheet lists anticipated NEMS modeling applications. For each modeling application, the spreadsheet includes the set of model components that is the target for final system delivery, and the set of components in initial and incremental deliveries. Note that this spreadsheet does not include all anticipated configurations! There are many test configurations that are not represented on the spreadsheet. Documentation about each of the NEMS applications and components is in general the responsibility of its original developers. Links to available documentation on NEMS applications and components have been compiled here.

1.2 Tools

Since there are multiple NEMS modeling applications, each with multiple model components, and multiple modes for each model component type (prognostic, prescribed data, etc.), there is a lot of complexity in the NEMS system. It is important to have a consistent way of working with the NEMS code base, and especially of coordinating changes to the NEMS code base.

Modeling applications can be assembled using a tool called the AppBuilder This tool enables each modeling application team to fully control its source code.

Development and analysis of modeling applications in NEMS will require the use of other component modes besides prognostic components. Prescribed data components, stub components that don't send any fields, and other options are needed for technical testing and controlled scientific experimentation. The tool called Component Sets, or Compsets, enables testing of each model application in numerous configurations.

1.3 Infrastructure

NEMS is built using the Earth System Modeling Framework (ESMF) infrastructure software. ESMF provides utilities like generation of interpolation weights and utilities for calendar and timee management, and also wrappers that create a standard component calling interface. This enables model components developed at different sites to be coupled more easily.

The National Unified Operational Prediction Capability (NUOPC) Layer adds additional rules about how ESMF models interact and increases their interoperability. The NUOPC Layer covers aspects from the level of build dependencies, to standardization of initialization phases, all the way to standard names of the exchanged fields. NEMS is an example of a modeling system built using the NUOPC Layer architecture.

2. NEMS Technical Architecture Overview

2.1 Component Architecture

The NEMS architecture is based on an ESMF component hierarchy with the application driver MAIN_NEMS at the top, calling into the NEMS_COMP component which in turn drives the EARTH_COMP component. The EARTH_COMP drives the ATM component (which calls into options GSM, NMMB, or FIM). The architecture allows for multiple EARTH_COMP instances, supporting ensemble applications such as the Global Ensemble Forecast System (GEFS).

Coupled NEMS includes atmosphere, ocean, ice, wave, land, aerosol/chemistry, and hydrologic models, with coupling interface and utilities based on the Earth System Modeling Framework (ESMF). The NEMS applications also utilize intreopereability conventions introduced by the National Unified Operational Prediction Capability (NUOPC).

Key architecture features of the coupled NEMS:

The coupled NEMS version extends the original NEMS version by implementing the EARTH_COMP level as a NUOPC compliant driver that accepts a NUOPC compliant mediator component and NUOPC compliant model components (ATMs, OCNs, ICEs, WAVs). The diagram below shows the architecture of the target coupled NEMS system. Although not shown, recent developments in the mediator are adding separate land and hydrology components as siblings to ATM, OCN, ICE, and WAV.

The specific OCN, ICE, WAV models are shown within clouds outside of the NEMS box to indicate that NEMS handles them as external dependencies. Data exchanges between components are handled by generic NUOPC_Connector components indicated by green arrows. The generic connectors perform basic regrid and redist operations as needed to take field data from one side to the other.

2.2 Build & Run Configuration

The coupled NEMS application is highly configurable. During build-time, i.e. when the NEMS executable is being compiled and linked, choices are made as to which model and mediator components are built into the system. The built-in components become accessible during run-time, allowing the execution of different run configurations without the need to rebuild NEMS.

Often the build and run configurability of NEMS is hidden from the user by application or test scripts. However, there are times where it is useful to understand the technical details on the level of the NEMS executable.

2.2.1 Build Configuration

The NEMS build configuration is accessible through the GNU Make based build system.

The ocean, sea-ice, ionosphere-plasmasphere model, hydraulic model, and wave components that are built into the NEMS executable are picked via the OCN, ICE, IPM, HYD, and WAV variables, respectively. Each of these variables is a comma separated list of models that can be specified on the make command line. The currently available options for each variable are shown below:

For each model type the current non-active instance options are listed below. The definition of these two options is similar to their use in the Community Earth System Model (CESM):

s<model type>: Stub components conform to the NUOPC rules for model components. They do not advertise any fields in their importState or exportState. Their primary use is to test control flow between components in a driver.

x<model type>: Dead components conform to the NUOPC rules for model components. They advertise fields in the importState and exportState that are appropriate for the specific model type. Import fields may be ignored internally. Export fields are filled with data that changes during time stepping, but has no scientific relevance. Their primary use is in coupled systems with other dead components to test the data transfers between components.

All of the variables support the specification of multiple options via comma separated list on the right-hand side. The default target of the NEMS makefile (i.e. no target specified) will print out a list of options with explanation to assist the user with the build configuration.

2.2.2 Run Configuration

During run-time of the NEMS executable, it accesses a file called nems.configure, which it expects to find in the run directory. This file specifies the dynamic component selection, and the exact run sequence to be used. Only models built into the executable at build-time are accessible during run-time. An error will be triggered if an unavailable model is selected in nems.configure. The component selection is based on two variables:

xxx_model:          abc
xxx_petlist_bounds: lower upper

Here "xxx" can be "atm", "ocn", "ice", "ipm", "med". The "abc" stands for the actual instance name, e.g. "gsm" or "mom5". The lower and upper bounds of the petList specification are integer PET numbers.

The specification of the run sequence provides the flexibility needed to cover different coupling scenarios. The format is line based between special tags:

runSeq::
    line1
    line2
    ...
    lineN
::

There are a number of format options for each line:

Here is an example of a run sequence specification with two time scales:

# Run Sequence #
runSeq::
    @7200.0
    OCN -> MED
    MED MedPhase_slow
    MED -> OCN
    OCN
    @3600.0
        MED MedPhase_fast_before
        MED -> ATM
        ATM
        ATM -> MED
        MED MedPhase_fast_after
      @
    @
::

Anything on a line after the "#" symbol is treated as a comment and ignored. Indentation in the formatting does not change the meaning of a line, but is purely used to increase readability.

3. NEMS Directory Structure

The NEMS directory contains the source code and test scripts for the NEMS. Most of the documentation is in the doc subdirectory or in the ../doc/ directory. Most of the files that were in the NEMS have been moved to the application layer, discussed below. Further documentation, specific to the app, is also at the app level.

Within NEMS resides:

At the application level resides these files:

4. Building NEMS

This chapter describes the options for building the NEMSLegacy, and the supported platforms. There are three ways to build: the NEMSAppBuilder in interactive or non-interactive mode, or a manual process. The recommended way to compile the NEMS is to use the NEMSAppBuilder in non-interactive mode. However, all methods are described here. We also provide troubleshooting information at the end of this chapter.

4.1 Build Targets

The list of build targets available for an app is found at the top level of the app in *.appBuilder files. The app-level documentation should have information about the meanings and purpose of each build target.

4.2 Recommended Method: Non-Interactive NEMSAppBuilder

From the top level (directory above NEMS), run the NEMSAppBuilder. This is a build script that knows how to build various apps, and works for more than just the NEMSLegacy. The syntax is:

./NEMS/NEMSAppBuilder (options) app=(app)

Here, the (app) is the selected application as discussed in the Supported Builds and Platforms section. The (options) should be one of the following:

4.3 Interactive NEMSAppBuilder

The NEMSAppBuilder can be run in interactive mode. To do so, simply run the command without any arguments:

./NEMS/NEMSAppBuilder

The NEMSAppBuilder will instruct you further. Note that this method will discard some of the log files, which makes build failures harder to track. Also, it has some errors in its process tracking, and will kill the wrong processes when a build is canceled. Such bugs are why the non-interactive mode is preferred.

4.4 Manual Method: Configure, Clean, Make

It is possible to build all apps via a manual method. This method also makes other, undocumented, features available. Ultimately, the NEMSAppBuilder is simply a wrapper around these manual commands. Before using such manual commands, it is best to talk to code managers to make sure you are building correctly.

The manual method works like this:

cd NEMS/src/ ./configure (method) source conf/modules.nems gmake clean gmake (nems-ver) J=-j2

The (method) is one of the available configurations. Run ./configure help to get a list, or read the configure script. The (nems-ver) is one of the following:

4.5 Troubleshooting Failed Builds

4.5.1 Incomplete Checkout

When there are network problems or high server load, your checkout from the Subversion and Git repositories may fail. This will lead to any number of confusing errors while building. You can continue the checkout process by going to the top level (above the NEMS directory) and running svn update. Repeat that until no more files are updated, and no errors are reported.

4.5.2 Unclean Environment

Setting up your environment incorrectly can lead to problems while building. If you see build issues from a clean, new checkout, this may be the problem. You should remove all module commands from your ~/.*rc files and get a clean, new login shell. Then retry the build.

4.5.3 Unclean Checkout

Another common cause of failed builds is having unintended changes in your source code or build system. To test for this, get a clean, new checkout from the repository and retry.

4.5.4 Unsupported Platform

Some apps only support a few platforms. For example, the NEMSLegacy app is only supported on WCOSS Phase 1 (Gyre/Tide) and NOAA Theia. Attempts to build on other platforms may or may not work.

4.5.5 Simultaneous Builds

Attempting to build multiple times in the same NEMS checkout directory will cause unexpected failures. For example, if you are running the regression test system twice at once, multiple builds will happen at the same time. On Theia, this frequently shows up as a massive, many terabyte, file which cannot be created due to fileset quota limits. Other failure modes are possible.

5. New Test System

The old regression test system has been replaced by a new system. It has a different design that the old one. It has a superset of the capabilities of the old system, but the different design leads to advantages and disadvantages.

Presently, that implementation is available by the NEMS/tests/rtgen script, and two scripts it generates (rtrun, rtreport). For backward compatibility, there is a wrapper "rt.sh" script to prevent users from having to learn a new system if they are only running the regression tests (not modifying them).

5.1 Design and Capabilities

This system works on a different principle than the older one. The old system ran shell scripts specific to each model or test which copied files from outside the NEMS test area and ran external programs to generate some inputs.

The new system has a directory of prepared inputs, has no external dependencies, and simply runs the NEMS executable without any test-specific scripts. In other words, scripts like the exglobal_fcst_nems.sh are no longer used. This makes porting and workflow changes simpler, but has the disadvantage of not testing model workflow scripts. That disadvantage is intentional; the purpose of the NEMS regression tests is to test the NEMS, not model workflow scripts.

5.2 Running the System

This section explains how to run the system in its simplest form. Later sections discuss running subsets of the tests, dependency resolution, and available tests. We provide two methods: a simple way using the rt.sh wrapper, and a more complex way that provides complete control and flexibility.

Simple Method: rt.sh

For backward compatibility, there is an rt.sh script that acts similarly to the old rt.sh. Some aspects are different to give extra flexibility.

To execute in an sh-family shell (sh, bash, ksh, etc.)

cd NEMS/tests
./rt.sh (options) > rt.log 2>&1 &

To execute in a csh-family shell (csh, tcsh):

cd NEMS/tests
./rt.sh (options) >& rt.log &

This will run rt.sh in the background and send all output to the rt.log file. To see the success or failure information, look in the rt.log file.

The (options) specify what is to be run. Common needs are:

Full Test Method

The process of running is:

./NEMS/tests/rtgen   # generates rtrun and rtreport commands
/path/to/USERNAME/rtgen.(ID)/rtrun (options)
/path/to/USERNAME/rtgen.(ID)/rtreport

To use this for a commit to the trunk, one must copy the results to the NEMS/tests directory. This could be done manually, or one could run rt.sh and tell it to skip the rtgen step. To do this, use the rt.sh -r option:

./rt.sh -r (PLATFORM):/path/to/USERNAME/rtgen.(ID)

where (PLATFORM) is "theia" or "wcoss.phase1"

The rest of this section explains the purpose and function of rtgen, rtrun and rtreport.

Step 1: Generate Test Scripts (rtgen)

The first step is to run rtgen. This will generate a set of scripts to run the requested tests. If you do not request any tests, it will run all tests.

./NEMS/tests/rtgen

That command will give you instructions and will log the more important parts of its execution:

11/17 18:42:38Z rtgen-INFO:  Will run all known tests.
11/17 18:42:50Z rtgen-INFO:  Auto-chosen project for job submission is 'cmp'
11/17 18:42:51Z rtgen-INFO:  Auto-chosen ptmp is '/path/to/USERNAME'
11/17 18:42:51Z rtgen-INFO:  Generating workflow with id 23768.
11/17 18:42:55Z rtgen-INFO:  Requested test has been generated.

You need to run the test now. You have three options:

OPTION 1: Put this in your cron: */3 * * * * /path/to/USERNAME/rtgen.23768/rtrun –step –zero-exit \

/path/to/USERNAME/rtgen.23768/rtrun-cron.log 2>&1

OPTION 2: Run this program: /path/to/USERNAME/rtgen.23768/rtrun –loop

OPTION 3: Verbose mode: run this program: /path/to/USERNAME/rtgen.23768/rtrun) -v –loop Adding -n to that command will disable colors.

Step 2: Run the Test (rtrun)

The rtrun command runs the tests until all have succeeded or failed. You have three options for how to run this. The easiest execution option is number 3, which runs on the command line and reports the queue status every few minutes. The path to rtrun will vary, but the command will look something like this:

/path/to/USERNAME/rtgen.23768/rtrun -v --loop

If the colors annoy you, add the -n switch, and if you don't want the queue state, remove the -v switch.

The components of that path are:

The rtrun command will generate output like this:

11/17 00:19:21Z rtrun INFO: check dependencies and submit jobs...
11/17 00:19:22Z rtrun INFO: check status...
11/17 00:19:22Z rtrun INFO: workflow is still running and no jobs have failed.
11/17 00:19:22Z rtrun INFO: sleep 2
11/17 00:19:24Z rtrun INFO: get queue information

Job ID Reserv Queue Procs ST Queue Time Stdout Location


573626 dev 64 R 11/17 00:14 /.../tmp/log/test_gfs_gocart_nemsio.log From bjobs -l -u Samuel.Trahan (age 0 sec.) 11/17 00:19:24Z rtrun INFO: sleep 100

It will keep looping until all jobs have succeeded or failed. If all goes well, the tests will all pass and you will see this message:

11/17 00:21:04Z rtrun INFO: check dependencies and submit jobs...
11/17 00:21:05Z rtrun INFO: check status...
11/17 00:21:05Z rtrun INFO: workflow is complete and all jobs succeeded.

Step 3: Report Results (rtreport)

At that point, you can run rtreport to get a report of the tests. Actually, you can run rtreport at any time. If the tests are not yet complete, it will tell you which ones are complete. It will report all it knows about failed tests too. There are two output formats:

To run:

/path/to/USERNAME/rtgen.23768/rtreport [mode]

Where the optional mode is one of:

The output of txt mode (the default) looks something like this

BUILD nmm.x: SUCCEEDED
BUILD nmm.debug.x: SUCCEEDED
BUILD gsm.x: SUCCEEDED
BUILD gsm_gocart.x: SUCCEEDED
TEST #1: PASS
Test nmm_cntrl starting.
Wed Nov 16 22:51:23 UTC 2016
.../REGRESSION_TEST/NMMB_glob/nmmb_hst_01_bin_0000h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_hst_01_bin_0024h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_hst_01_bin_0048h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_hst_01_nio_0000h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_hst_01_nio_0024h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_hst_01_nio_0048h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_rst_01_bin_0024h_00m_00.00s: bit-for-bit identical
.../REGRESSION_TEST/NMMB_glob/nmmb_rst_01_nio_0024h_00m_00.00s: bit-for-bit identical
TEST PASSED
TEST #2: PASS
Test nmm_nemsio starting.
... information about more tests ...

Rerunning Failed Tests

If a test fails, you can request that it be rerun via the rtrewind command. The command is located in the same directory as rtrun and can be called in two different ways:

/path/to/USERNAME/rtgen.23768/rtrewind -a

/path/to/USERNAME/rtgen.23768/rtrewind job1 [job2 [...]]

The first method requests a rerun of ALL tests and builds while the second requests only certain ones be rerun.

The jobs (job1, job2, ...) are the names from the test suite such as gsm.x or nmm_cntrl. You can optionally include test_ or build_ before the name, as it is printed by the rtreport command.

Running Subsets of the Test Suite

The test suite, as of this writing, has 48 tests and 5 build options. Frequently, you only want to run a few of them. The rtgen script has a simple set arithmetic language for specifying what to run. The subsetting is done by the command line. For example, to run all standard nmm tests, you need to take the intersection of those two sets of tests:

./NEMS/tests/rtgen 'inter(nmm,standard)'

The rtgen will generate a workflow to run just those tests.

Other subsetting operations:

union(nmm,wam) # run all nmm and wam tests minus(gfs,wam) # run all gsm (gfs) tests that are not wam tests {gfs_slg,nmm_cntrl} # run the gfs_slg and nmm_cntrl tests

You can combine multiple operations:

minus(inter(union(gfs,nmm),standard),{gfs_slg,nmm_cntrl})

That will ask rtgen to run all gsm (gfs) and nmm tests that are standard tests, except for gfs_slg and nmm_cntrl.

Despite that, the rtgen will still run the gfs_slg test. Why? Dependency resolution.

Dependency Resolution

Some tests have dependencies, and rtgen will resolve those dependencies automatically, similar to how make works. For example, the gfs_slg_rsthst requires the gfs_slg to run first. Output from gfs_slg is used as input to gfs_slg_rsthst. If you ask rtgen to run gfs_slg_rsthst without running gfs_slg, it will see the dependency and add gfs_slg to your list of tests. The builds are handled the same way. The gfs_slg has a dependency on the build gsm.x, and so rtgen will always add the gsm.x build if you select the gfs_slg test.

List of Available Tests and Sets

The configuration for rtgen is stored in the compsets/all.input file in the app level repository. This is where you specify the available tests and sets of tests.

The top few lines of that file look like this

load 'gsm.input'
load 'nmm.input'
run nmm_cntrl              @ nmm, standard, baseline, nmmglob
run nmm_nemsio             @ nmm,                     nmmglob
run nmm_rest               @ nmm,                     nmmglob
... many more "run" statements ...

The first two lines import the details of the test from other files. The lines beginning with run specify a test to run and the sets it belongs to. The test must be one declared in the other file, as discussed later in this document.

The list of sets after the @ sign are the ones recognized by the subsetting functionality of rtgen.

Note that you can enable tests on only certain platforms by including a comparison operator in the list of subsets:

run gfs_slg_2thread @ gfs, standard, baseline, slg, plat==wcoss.phase1

This line ensures the gfs_slg_2thread is only available on WCOSS Phase 1.

5.3 Work Area Contents

Running the rtgen creates a directory in a scrub area which will contain the generated scripting system, input and output files, logs, and resource usage information. This section documents those files and directories.

Recall that running rtgen creates a directory with a name like this:

/path/to/USERNAME/rtgen.23768

That directory contains the following:

Jobs, Scripts and Ush

These are the three tier NCEP workflow directories and have the usual meanings:

For each test, there is one "jobs" directory file and one "scripts" directory file. The "scripts" directory and "jobs" directory are populated by the tests blocks which will be discussed in great detail in the Test Description Language section. They are generated from the test blocks.

Src, Exec, and Include

The src directory does not contain source code. Instead, it contains two scripts that describe how to build or uninstall the NEMS.x

The install.sh creates executables and modulefiles which are copied into the exec and include directories.

Rocoto Directory

The rtgen makes one file in the rocoto directory. The rtrun will create a second file.

Tmp and Logs

The tmp directory contains all logs and all execution directories for each test.

Scripts rtrun and rtreport

These are discussed in earlier sections. The scripts are generated automatically by rtgen. The rtrun runs Rocoto and the rtreport scans the reports, combining them into one text file.

COM directory

This directory contains one subdirectory for each test with all verified files as described in a test's (criteria)[#criteria] block. It also contains the "report.txt" file with the report of the test success or failure.

5.4 Test Description Language

This chapter discusses the language used by the rtgen tool to describe regression tests and compsets. The language consists of "modules" which are simply a collection of variables and functions. A module has a type: build, test, hash, etc. A set of run commands list which runnable modules should be executed.

Variable Definitions and Modules

The simplest type of module is a hash, which looks like this:

nems_vars={
atm_model='none'
atm_petlist_bounds="-1 -1"
ocn_model='none'
ocn_petlist_bounds="-1 -1"
ice_model='none'
ice_petlist_bounds="-1 -1"
med_model='nems'
med_petlist_bounds="-1 -1"
med_atm_coupling_interval_sec='-1'
med_ocn_coupling_interval_sec='-1'
}

In this example, we have declared a hash called nems_vars which contains several variables, such as atm_model and atm_petlist_bounds. Later on, another module declaration can "use" this module, to import its variables:

nmm_vars_global={
use plat%nmm_dflt
use nems_vars
use common_vars
use nmm_vars
use nmm_aliases
use nmm_uncoupled
GBRG="glob"
CNTL_NAME='NMMB_glob'
}

Values can include variable substitution, which uses a similar syntax as shell, but with different escape characters:

common_vars={
THRD=1
WLCLK=15
GEFS_ENSEMBLE=0
GEN_ENSEMBLE=0
WRITE_DOPOST='.false.'
POST_GRIBVERSION='grib1'
CONF="@[plat%PARMnems]"
}

Here, the CONF variable in the common_vars module has the value of the PARMnems variable in the plat module.

Strings

There are three ways of specifying a string:

If you need to insert a literal @ into the string, you have three options. In these examples, we'll use the multi-line string format:

Embedded Scripts

Most of the scripts required to run the tests are automatically generated, but there are occasional instances when you need to specify specific code. This is done via embed blocks:

embed bash nems_regtest_prep(RUNDIR,modules,CNTL) [[[
mkdir -p "$RUNDIR" "$CNTL"
cd @[RUNDIR]
source "$modules"
export MPI_TYPE_DEPTH=20
export ESMF_RUNTIME_COMPLIANCECHECK=OFF:depth=4
]]]

In this example, we have embedded a bash script called nems_regtest_prep.

Embedded Script

Variables: $ vs. @

In the example script, there are two methods of doing variable substitution:

They have slightly different meanings. In the case of @[RUNDIR], the value of the RUNDIR variable is substituted directly in the generated script. If the variable contained any shell metacharacters, those would be copied verbatim. In the case of $RUNDIR, the bash variable is used instead. That variable's value is set before the code in nems_regtest_prep is run.

Either approach is valid. It is up to the user to decide which one to use.

Platform Detection

The test suite needs to reconfigure certain aspects based on platform; WCOSS vs. Theia vs. GAEA, etc. This is done with platform blocks. These are simply modules with a detect function. After all platforms are defined, an autodetect block selects between them.

Here is an example of a platform. This is the one for Phase 1 of WCOSS.

platform wcoss.phase1 {
use wcoss.common
CPU_ACCOUNT='NAM-T2O'
pex='1'
cores_per_node=32
MPI='LSF'
SHORT_TEST_QUEUE='&SHORTQ;'
LONG_TEST_QUEUE='&LONGQ;'
BUILD_QUEUE='&BUILDQ;'

embed bash detect [[[
# This function is used at PARSE TIME to detect whether we are
# on WCOSS Phase 1.  It must be very fast and low resource
# usage since the parser runs it.
if [[ -d /usrx && -d /global && -e /etc/redhat-release && \
-e /etc/prod ]] ; then
# We are on WCOSS Phase 1 or 2.
if ( ! cat /proc/cpuinfo |grep 'processor.*32' ) ; then
# Fewer than 32 fake (hyperthreading) cpus, so Phase 1.
exit 0
fi
fi
exit 1
]]]
... more wcoss stuff ...
}

Note the embed bash block called detect. This is the bash function that is run to detect whether the script is running on WCOSS Phase 1.

Once all platforms are defined, there is an autodetect block:

autodetect plat (/ wcoss.phase1, theia /)

This will define the plat variable, which is a duplicate of either wcoss.phase1 or theia.

Build Definition

The build blocks define a method of building an executable. They must define three variables and a function:

Here is an example. This builds the GOCART-capable standalone GSM in the NEMSLegacy branch:

build gsm_gocart.x {
use plat
NEMS.x="@[plat%EXECrt]/NEMS_gocart.x"
modules.nems="@[plat%INCrt]/NEMS_gocart.x.modules"
target="@[NEMS.x]"
build=NEMSAppBuilder(NEMS.x="@[NEMS.x]",modules.nems="@[modules.nems]",
OPTS="app=GSM-GOCART")
}

The NEMSAppBuilder function is declared elsewhere. It is used by most of the build definitions to avoid duplication. That function looks like this:

embed bash NEMSAppBuilder(NEMS.x,modules.nems,OPTS)
[[[
mkdir -p "@[plat%EXECrt]" "@[plat%INCrt]"
rm -f "@[NEMS.x]" "@[modules.nems]"
cd @[plat%HOMEnems]

# NOTE: Replace "rebuild" with "norebuild" to disable "gmake clean"
./NEMS/NEMSAppBuilder rebuild $OPTS

cd @[plat%SRCnems]
cp -fp ../exe/NEMS.x "@[NEMS.x]"
cp -fp conf/modules.nems "@[modules.nems]"
]]]

Notice that the four variables we're passing from gsm_gocart.xbuild are in the definition line of NEMSAppBuilder:

embed bash NEMSAppBuilder(NEMS.x,modules.nems,OPTS)
...
build gsm_gocart.x {
...
build=NEMSAppBuilder(NEMS.x="@[NEMS.x]",modules.nems="@[modules.nems]",
OPTS="app=GSM-GOCART")

Tests

A test is a module that defines the following:

This is the test block for the global nmm control. Later text describe the meaning of each part:

# nmm_cntrl test
test nmm_cntrl: nmm.x {
use nmm_vars_global

# Convenience variables:
RUNDIR_ROOT="@[plat%TMPrt]"
RUNDIR="@[RUNDIR_ROOT]/@[TEST_NAME]"
TEST_DESCR="Compare NMMB-global results with previous trunk version"
CNTL="@[plat%BASELINE]/@[CNTL_NAME]"      # Control baseline area
TEST_IN="@[plat%INPUTS]/@[CNTL_NAME]"   # Test-specific input data
COM="@[plat%COMrt]/@[TEST_NAME]"

criteria output {
#    NEMS.x output file --------- comparison - control file or dir
"nmmb_hst_01_bin_0000h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_bin_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_bin_0048h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_nio_0000h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_nio_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_nio_0048h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_rst_01_bin_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_rst_01_nio_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
}

# The prep is run at the top of any job.  It should do such things
# like making directories and loading modules.
prep=nems_regtest_prep(
RUNDIR="@[RUNDIR]",modules="@[nmm.x%modules.nems]",
CNTL="@[CNTL]")

# The execute step runs the program:
spawn execute {
{ "@[nmm.x%NEMS.x]", ranks="@[TASKS]", threads="@[OpenMPThreads]" }
}

filters input {
# work file         operation   input file
"input_domain_01"        .copy. "@[TEST_IN]/test_input_nmmb_global"
"input_domain_01_nemsio" .copy. "@[TEST_IN]/test_input_nmmb_global.nemsio"
"GWD_bin_01"             .copy. "@[TEST_IN]/GWD_bin_01"

"nems.configure"      .atparse. "@[CONF]/nems.configure.@[nems_configure].IN"
"atmos.configure"     .atparse. "@[CONF]/atmos.configure_nmm"

"configure_file_01"   .atparse. "@[CONF]/nmm_conf/nmm_@[GBRG]_conf.IN"
"model_configure"        .copy. "configure_file_01"

"*"                   .copydir. "@[plat%NMM_DATA]"

"VEGPARM.TBL"            .copy. "IGBP_VEGPARM.TBL"
"LANDUSE.TBL"            .copy. "IGBP_LANDUSE.TBL"
"ETAMPNEW_DATA"          .copy. "ETAMPNEW_DATA.expanded_rain"
"fort.28"                .link. "global_o3prdlos.f77"
"fort.48"                .link. "global_o3clim.txt"

"solver_state.txt"       .copy. "@[plat%PARMnems]/solver_state.txt"
"nests.txt"              .copy. "@[plat%PARMnems]/nests.txt"
}
}

Test Dependencies

The first line (after the comment) is this:

test nmm_cntrl: nmm.x {

The : nmm.x indicates that the nmm.x build has to run before the nmm_cntrl can start. The test suite will include that dependency in its Rocoto or ecFlow automation system.

Test Prep

The prep step is a simple script that prepares the environment. In this case, it just runs the nems_regtest_prep, which we discussed earlier:

# The prep is run at the top of any job.  It should do such things
# like making directories and loading modules.
prep=nems_regtest_prep(
RUNDIR="@[RUNDIR]",modules="@[nmm.x%modules.nems]",
CNTL="@[CNTL]")

Note that it refers to @[RUNDIR] and @[CNTL]. Those variables are defined earlier in the same test:

# Convenience variables:
RUNDIR_ROOT="@[plat%TMPrt]"
RUNDIR="@[RUNDIR_ROOT]/@[TEST_NAME]"
TEST_DESCR="Compare NMMB-global results with previous trunk version"
CNTL="@[plat%BASELINE]/@[CNTL_NAME]"      # Control baseline area
TEST_IN="@[plat%INPUTS]/@[CNTL_NAME]"   # Test-specific input data
COM="@[plat%COMrt]/@[TEST_NAME]"

Test Input Filter

This block specifies the input files and how to prepare them. It declares an input variable inside the nmm_cntrl test, which is of type filters:

filters input {
# work file         operation   input file
"input_domain_01"        .copy. "@[TEST_IN]/test_input_nmmb_global"
"input_domain_01_nemsio" .copy. "@[TEST_IN]/test_input_nmmb_global.nemsio"
"GWD_bin_01"             .copy. "@[TEST_IN]/GWD_bin_01"

"nems.configure"      .atparse. "@[CONF]/nems.configure.@[nems_configure].IN"
"atmos.configure"     .atparse. "@[CONF]/atmos.configure_nmm"

"configure_file_01"   .atparse. "@[CONF]/nmm_conf/nmm_@[GBRG]_conf.IN"
"model_configure"        .copy. "configure_file_01"

"*"                   .copydir. "@[plat%NMM_DATA]"

"VEGPARM.TBL"            .copy. "IGBP_VEGPARM.TBL"
"LANDUSE.TBL"            .copy. "IGBP_LANDUSE.TBL"
"ETAMPNEW_DATA"          .copy. "ETAMPNEW_DATA.expanded_rain"
"fort.28"                .link. "global_o3prdlos.f77"
"fort.48"                .link. "global_o3clim.txt"

"solver_state.txt"       .copy. "@[plat%PARMnems]/solver_state.txt"
"nests.txt"              .copy. "@[plat%PARMnems]/nests.txt"
}

Notice that there are four different operations in the middle column:

Local file Operation Remote file or directory
"GWD_bin_01" .copy. "@[TEST_IN]/GWD_bin_01"
"*" .copydir. "@[plat%NMM_DATA]"
"fort.28" .link. "global_o3prdlos.f77"
"atmos.configure" .atparse. "@[CONF]/atmos.configure_nmm"

In the .atparse. variable replacement, only variables from the test's module are replaced. Hence, if you want many variables accessible to .atparse.d files, you need to either declare or use them. The nmm_cntrl test does that at the top of its declaration:

test nmm_cntrl: nmm.x {
use nmm_vars_global

# Convenience variables:
RUNDIR_ROOT="@[plat%TMPrt]"
RUNDIR="@[RUNDIR_ROOT]/@[TEST_NAME]"
TEST_DESCR="Compare NMMB-global results with previous trunk version"
CNTL="@[plat%BASELINE]/@[CNTL_NAME]"      # Control baseline area
TEST_IN="@[plat%INPUTS]/@[CNTL_NAME]"   # Test-specific input data
COM="@[plat%COMrt]/@[TEST_NAME]"

Everything in the nmm_vars_global module will be available plus all six of the declared "convenience variables"

Note that variables with a period (".") or percent ("%") in their name are not yet available. That will be fixed in a later release.

Test Execution

The next step is to actually run the NEMS.x:

# The execute step runs the program:
spawn execute {
{ "@[nmm.x%NEMS.x]", ranks="@[TASKS]", threads="@[OpenMPThreads]" }
}

The columns inside the execute block have these meanings:

Test Verification or Baseline Generation

The last step is to either verify the results or generate the baseline. Both cases are handled by the output criteria block:

criteria output {
#    NEMS.x output file --------- comparison - control file or dir
"nmmb_hst_01_bin_0000h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_bin_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_bin_0048h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_nio_0000h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_nio_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_hst_01_nio_0048h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_rst_01_bin_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
"nmmb_rst_01_nio_0024h_00m_00.00s" .bitcmp. "@[CNTL]"
}

The columns have this meaning:

In verification mode, the comparisons are performed after running NEMS.x

In baseline generation mode, the local file (first column) is copied to the remote location (third column).

6. How to Build and Run a NEMS Application

6.1 Quick Build and Run

6.1.1 Download

Use the following SVN command to download a specific revision of a NEMS application from its development trunk:

svn co -r <REV> 
https://svnemc.ncep.noaa.gov/projects/nems/apps/
<Application>/trunk <Application>

In this command, <REV> stands for a revision number, for example, 73964. In some places on this site, the prefix DREV is used to indicate that the revision is on a development trunk. The prefix R indicates that a revision is on an application trunk. These prefixes should not be included in the command, just the revision number.

Where <Application> appears, it can have one of the following values:

The SVN command will download all of the necessary pieces including constituent components from different repositories. It is possible to access the very latest working revision of the code by omitting the "-r REV" part from the SVN command when downloading by revision number. This may be convenient during development, however, it is not recommended for validation work where it is critical to keep track of the precise revision information.

There are a few ways to find out more about specific revisions and the features that they contain. The links under the Milestone Revisions header on the left navigation bar describe revisions that are documented and tested for particular application milestones. The development tasks completed for each revision are included in the task prioritization spreadsheet.

6.1.2 Build

Change into the <Application> directory created by the SVN command during download. Then execute the NEMS AppBuilder by typing:

./NEMS/NEMSAppBuilder

A terminal based dialog script will guide you through the build process. The end result of the build process is a NEMS executable (./NEMS/exe/NEMS.x) that is configured to run the application.

6.1.3 Run

Below are general instructions on how to run a NEMS application. There may be some special or additional directions for running a particular application configuration (for example, for the cold start in UGCS-Seasonal). Please see the links under specific Milestone Revisions for these instructions.

From within the <Application> directory execute the NEMS CompsetRun tool by typing:

./NEMS/NEMSCompsetRun

This command will automatically run all of the compsets listed in the <Application>.compsetRun file. This file can be edited to change the compsets that are being run. New CompsetRun files can also be created and explicitly specified when calling the NEMSCompsetRun tool. Finally, new application specific compsets can be created and added to the repository.

6.2 NEMS AppBuilder

NEMS is the technical backbone for multiple modeling applications. These modeling applications are multi-component coupled systems that must pull together, in a systematic way, components from different centers and agencies. The NEMS AppBuilder enables users to construct a specific, versioned modeling application from a versioned set of model components and configuration files.

The AppBuilder approach also encourages coordination of changes made to different applications as they get checked back into the NEMS repository. The NEMS AppBuilder introduces two levels of configuration control: the modeling application level and the NEMS level. This split provides the model developer with full version control over all aspects of the modeling application code, while hiding many technical details about the infrastructure at the NEMS level.

Under modeling application control:

Under NEMS control:

6.2.1 Implementation of Version Control

The recommended best management practice for implementing the application version control is with SVN using externals. SVN externals allow the application to point to specific versions of NEMS and specific versions of each of the external components. Only external components actually used by the application need to be referenced.

Following the above recommendation, the proposed UGCS-Seasonal application directory structure under SVN control looks like this:

---- UGCS-Seasonal 
|
+-- UGCS-Seasonal.appBuilder  (application specific AppBuilder file, listing required external components)
|
+-- UGCS-Seasonal.compsetRun  (application specific CompsetRun file, listing compsets to be run)
|
+-- NEMS        (SVN external, e.g. "NEMS -r 56703 https://svnemc.ncep.noaa.gov/projects/nems/branches/NUOPC/development")
|    |
|    + -.... -- NEMS code located here --
|
+-- NEMS/src/atmos/gsm      (SVN external, e.g. "GSM -r 75055 https://svnemc.ncep.noaa.gov/projects/gsm/trunk")
|                   |
|                   + -.... -- GSM code located here --
|
+-- CICE        (SVN external, e.g. "CICE -r 20 https://github.com/feiliuesmf/lanl_cice/trunk)
|    |
|    + -.... -- CICE code located here --
|
+-- MOM5        (SVN external, e.g. "MOM5 -r 5304 https://github.com/feiliuesmf/mom/branches/mom5_5_0_2_nuopc)
|
+ -.... -- MOM5 code located here --

This structure gives full control of versioning of all of the model code to the application: each UGCS-Seasonal version is tied to a specific revision of NEMS, CICE, and MOM5.

A prototype with the proposed directory structure exists at https://svnemc.ncep.noaa.gov/projects/nems/apps/UGCS-Seasonal/trunk. It can be accessed via SVN:

svn co -r 56726 https://svnemc.ncep.noaa.gov/projects/nems/apps/UGCS-Seasonal/trunk UGCS-Seasonal

This checks out everything needed to build the initial revision of the UGCS-Seasonal application prototype from source into a directory called UGCS-Seasonal. Build the NEMS executable for this application by changing into the UGCS-Seasonal directory and follow the steps below.

6.2.1.1 Location of the Modeling Application Directory

The application directory can be located anywhere under revision control. However, most of the NCEP applications are expected to be located on the EMC SVN server. It is very simple to relocate a directory that is under SVN control to a different location at any time (or to rename it), without losing any history. For this reason we have set up a staging area under https://svnemc.ncep.noaa.gov/projects/nems/apps where applications may first be placed before a final location is identified. We recommend creating the canonical trunk, tags, branches directory triplet under each application for consistency. E.g., at this point the future CFS system has this location and directory structure:

/projects/nems/apps/UGCS-Seasonal
|
+--- branches
+--- tags
+--- trunk

At any point in the future the UGCS-Seasonal directory can be renamed and/or moved to a different location on the SVN server. None of the revision history will be lost. This setup provides a very structured approach for early development work, without committing to anything that cannot be changed in the future.

6.2.1.2 Handling External Components

NEMS includes some external components whose primary development repository is outside of EMC. The application level provides the necessary control over which version of a specific external component is used in the modeling application.

External components can be located anywhere under revision control. There is a staging area under https://svnemc.ncep.noaa.gov/projects/nems/external_components where external components may first be placed before a final location is identified (in some cases back at the parent organization). We recommend creating the canonical trunk, tags, branches directory triplet under each application for consistency. E.g., at this point the future XYZ component has this location and directory structure:

/projects/nems/external_components/XYZ
|
+--- branches
+--- tags
+--- trunk

At any point in the future the XYZ directory can be renamed and/or moved to a different location on the SVN server. None of the revision history will be lost. This setup provides a very structured approach for early development work, without committing to anything that cannot be changed in the future.

6.2.2 Interaction with the NEMS AppBuilder

The NEMS AppBuilder is an infrastructure tool that comes with the NEMS system. Within the above shown directory structure the tool is accessible as ./NEMS/NEMSAppBuilder. It is important to call the tool from within the same directory where the application specific AppBuilder description file is located. Here the descriptor file is UGCS-Seasonal.appBuilder with the following contents:

# Climate Forecast System
#
## UGCS-Seasonal NEMS Application Builder file

COMPONENTS=( "GSM" "CICE" "MOM5" )

# CICE
CICE_SRCDIR=$ROOTDIR/CICE
CICE_DIR=$ROOTDIR/CICE-INSTALL

# MOM5
MOM5_SRCDIR=$ROOTDIR/MOM5
MOM5_DIR=$ROOTDIR/MOM5-INSTALL

# Zeus environment
environment_zeus(){
source /etc/profile
module purge
module use ~Gerhard.Theurich/Modulefiles
module load intel mpt/2.06 netcdf/4.1.3-intel esmf/7.0.0bs42-netcdf4.1.3
}

# Gaea environment
environment_gaea(){
source /etc/profile
module swap PrgEnv-pgi PrgEnv-intel
module load netcdf
module use ~Gerhard.Theurich/Modulefiles
module load esmf/7.0.0bs42
}

# Theia environment
environment_theia(){
source /etc/profile
module use ~Gerhard.Theurich/Modulefiles
module load intel impi esmf/7.0.0bs42-theia
}

The following steps show the NEMS AppBuilder in action for the above CFS example:

[~/UGCS-Seasonal]$ ./NEMS/NEMSAppBuilder

The AppBuilder is based on low-level terminal-based Unix utilities for maximum ease of use and portability.

Caption text

Next each of the external components is built. First CICE, then MOM5:

Caption text

Next the NEMS build system is configured:

Caption text

Finally the NEMS executable is being built:

Caption text

The build finishes with a notification:

Caption text

During the building process the NEMS AppBuilder monitors the progress and reports on issues followed by a diagnostic screen. Each diagnostic screen has the option to repeat the step that just had failed or to bail out. Repeating a step can be useful when the reported issue can be corrected in a second terminal and the build process can continue.

If the build process terminates with a problem condition, appBuilder.*log files with build and debug information will be present in the root directory from which the AppBuilder was started.

6.3 NEMS Compsets and CompsetRun

NEMS uses "component sets", or "compsets," to systematically label run configurations. The labels are associated with scripts that pull together all the files and inputs needed to run the specified configurations. Compset labels include which components and mediators are part of the configuration, whether each component is running in a prognostic or test mode, resolutions, and initial conditions. This approach offers a number of benefits:

Compsets were originated by the Community Earth System Model (CESM).

Supported NEMS compsets are listed here.

6.3.1 Running NEMS with Compsets

NEMS compsets are available for revision 52376 and later versions of NEMS. The front-end to using the compset feature is provided by the NEMSCompsetRun script located under the NEMS root directory.

From within a modeling application, e.g., UGCS-Seasonal, the NEMSCompsetRun is typically invoked from the root directory:

./NEMS/NEMSCompsetRun

The script will look for file(s) *.compsetRun in the root directory and execute the compsets listed therein.

The script can also be invoked from within the NEMS directory itself:

./NEMSCompsetRun [COMPSET_LIST_FILE]

or

./NEMSCompsetRun -compset COMPSET_FILE

Running this script without the optional COMPSET_LIST_FILE argument will execute the compsets that are listed in file ./compsets/default.compsetRun under the NEMS directory. An alternative list of compsets can be specified by using the COMPSET_LIST_FILE argument. The format of this file is very simple:

  1. A line starting with # is a comment.
  2. A line starting with DIR= specifies an alternative search directory for individual compset configuration files (explained in more detail below).
  3. Blank lines are ignored.
  4. All other lines are assumed to list a single compset configuration filename at the beginning of the line. The remainder of the line is ignored and can be used for comments.

For example, here is the content of a sample COMPSET_LIST_FILE:

### List of compsets ###
#####################

#DIR=  # optionally set this if not using default ./compsets directory

2013_sbys_nmmb%glob      ! stand-alone global NMM-B
AMIP_sbys_gsm            ! stand-alone GSM - fake example
2009_nems_gsm_cice_mom5   ! NEMS mediator coupled GSM-CICE-MOM5
2011_sbys_gsm%wam        ! stand-alone GSM run as WAM
2011_sbys_gsm%wam%ndsl   ! stand-alone GSM run as WAM with NDSL

For each compset listed, there must be an associated compset configuration file of the same name. By default the ./compsets subdirectory under the NEMS root directory is searched for compset configuration files. The search directory can be changed by using the DIR= directive in the COMPSET_LIST_FILE.The format of the compset configuration files is described below.

Calling NEMSCompsetRun with the -compset COMPSET_FILE option allows to run a single compset without the need to first create a COMPSET_LIST_FILE. The COMPSET_FILE file must be specified with path (absolute or relative) so the compset file can be found by NEMSCompsetRun.

Starting in April, 2016, a more detailed specification for NEMS compsets, below, was introduced. It adds the run duration after an initial condition label, and includes grid resolution as part of the specifier for each component.

cfsr%20150401_1hr_nems%cold_gsm%eul%T126_cice%0.5_mom5%0.5
cfsr%20150401_1hr_nems%cold_gsm%slg%T126_cice%0.5_mom5%0.5
cfsr%20150401_1day_nems_gsm%eul%T126_cice%0.5_mom5%0.5
cfsr%20150401_1day_nems_gsm%slg%T126_cice%0.5_mom5%0.5
cfsr%20150401_30day_nems_gsm%eul%T126_cice%0.5_mom5%0.5
cfsr%20150401_30day_nems_gsm%slg%T126_cice%0.5_mom5%0.5

6.3.2 How to Add a New Compset

A new compset is added to the system by adding a new compset configuration file, either in the default location under ./compsets, or in a different location referenced by the DIR= setting as explained above. The name of the new configuration file is that of the new compset. The existing default compset configuration files under the ./compsets subdirectory can be used as templates for new configurations. The compset configuration files represent the connection between the compset based approach to label different configurations and the existing NEMS run scripts. The files are very simple short pieces of shell scripts, exporting a number of variables that are being used by the existing NEMS run scripts. Writing a new compset thus requires some knowledge about the standard NEMS run scripts.

6.3.2.1 Compset Nomenclature

On the machine level the name of a compset is not dissected or interpreted. Instead, the compset names are simply matched as literal strings against the name of the available compset configuration files. However, there is a nomenclature to be followed when naming compsets. The purpose of this nomenclature is to help a human reader to quickly grasp what a specific compset is testing, and to bring some level of order into the many possible component configurations. The general naming format is:

caselabel[optDescriptor][_optRunDuration]_architecture_model1[opt1[opt2[...[optN]]]]_model2[...]_..._modelN[...]

The "caselabel" may be as simple as a calendar year, like "2009", indicating a specific set of initial conditions that are being used. This label can be used very freely to indicate specifics about the case being run. For the sake of uniformity the convention is to keep the case label below 16 characters, not use any special characters like underscores, and instead compound by camel case. It can take an option, for example the start data and/or time of an initial condition.

The "optRunDuration" specifies the duration of the run specified in the compset.

The "architecture" specifies the fundamental coupling mode. Details of the run sequence are set inside the compset file through the nems_configure variable (see below "Compset Configuration"). The currently available options are:

Architecture options Description
blocked Model components are coupled directly through connectors. All connectors are executed in a single block at the beginning of the coupling cycle, followed by executing all of the model components.
leapfrogModel components are coupled directly through connectors. Connectors and model components are interleaved. As a result, some model components receive forcing data that is at the start time of the next coupling step.
nems Model components are coupled through a central mediator component. Most or all interactions are through the mediator. There may be some direct connections between model components. The default NEMS mediator is used.
sbys Model components are running side-by-side for the same simulation conditions (start/end times), but without any interaction between the individual models. This architecture option also covers the standalone mode for the case where only a single model component is present.
spaceweather Model components are coupled through a central mediator component. Most or all interactions are through the mediator. There may be some direct connections between model components. The NEMS Spaceweather mediator is used.

The "model" labels indicate which specific model instances are being run. In order to reduce redundancies introduced by identical permutations, a specific order of model types is used when constructing compset names:

  1. ATM - Atmosphere
  2. LND - Land
  3. ICE - Sea Ice
  4. OCN - Ocean
  5. WAV - Wave
  6. IPM - Ionosphere-Plasmasphere
  7. HYD - Hydraulic

Several model instances are available for each model type (instances in ** are not yet accessible):

Model type Instance options
ATM satm, xatm, gsm, nmmb
LND slnd, xlnd, lis**
ICE sice, xice, cice
OCN socn, xocn, hycom, mom5
WAV swav, xwav, ww3**
IPM sipm, xipm, ipe
HYD shyd, xhyd, wrfhydro

For each model type the current non-active instance options are listed below. The definition of these two options is similar to their use in CESM:

Only model components that are present in specific compset configurations are listed in the compset name.

The compset nomenclature supports appending option strings after each model component. Each option is introduced by a "%" character. Multiple options are simply concatenated. Each option string should be less than 16 characters long, use no special characters like underscores, and compounded by camel case.

6.3.2.2 Compset Configuration Parameters

Each compset configuration file is a bash script that is sourced within the NEMSCompsetRun script as it iterates through the compset file list. The variables defined in a compset configuration file are referenced by individual components to determine the configuration for that run. The set of runtime configuration variables varies by component. Please refer to the component for more information. A list of variables specific to NEMS is listed below. The compset configuration script is also responsible for setting up the run directory with component specific runtime configuration and data files. Setup functions have been defined in NEMSCompsetRun for some components. Once the runtime configuration is complete please call the appropriate regression test script.

6.3.2.2.1 Compset Variables

TEST_DESCR : Compset description

6.3.2.2.2 NEMS Variables

These variables are passed to the NEMS configuration file located in tests/nems.configure.[NEMS configuration filename].IN and read at runtime.

nems_configure : Name of the NEMS configuration file template - includes the run sequence pattern
atm_model : Atmosphere component ("satm", "xatm", "gsm", "nmmb", "none")
atm_petlist_bounds : Atmosphere component petlist lower and upper bounds
lnd_model : Land component ("slnd", "xlnd", "lis", "none")
lnd_petlist_bounds : Land component petlist lower and upper bounds
ice_model : Sea ice component ("sice", "xice", "cice", "none")
ice_petlist_bounds : Sea ice component petlist lower and upper bounds
ocn_model : Ocean component ("socn", "xocn", "mom5", "hycom", "none")
ocn_petlist_bounds : Ocean component petlist lower and upper bounds
wav_model : Wave component ("swav", "xwav", "ww3", "none"
wav_petlist_bounds : Wave component petlist lower and upper bounds
ipm_model : Space weather component ("sipm", "xipm", "ipe", "none")
ipm_petlist_bounds : Space weather component petlist lower and upper bounds
hyd_model : Hydrology component ("shyd", "xhyd", "wrfhydro", "none")
hyd_petlist_bounds : Hydrology component petlist lower and upper bounds
med_model : Mediator ("nems", "none")
med_petlist_bounds : Mediator petlist lower and upper bounds
atm_coupling_interval_sec : Atmosphere run sequence coupling interval
ocn_coupling_interval_sec : Ocean run sequence coupling interval
coupling_interval_sec : Run sequence coupling interval
coupling_interval_slow_sec : Run sequence slow coupling interval
coupling_interval_fast_sec : Run sequence fast coupling interval

:

6.3.2.2.3 Component Default Variable Functions

export_nmm : Set NMM configuration variables to default values
export_gsm : Set GSM configuration variables to default values
export_fim : Set FIM configuration variables to default values
export_nems : Set NEMS configuration variables to default values

6.3.2.2.4 Component Setup Functions

setup_hycom : Link HYCOM test case runtime configuration and data files to regression test directory
setup_mom5cice : Copy MOM5 test case runtime configuration and data files to regression test directory.
setup_ipe : Link IPE test case runtime configuration and data files to regression test directory.
setup_spaceweather : Link SpaceWeather test case runtime configuration and data files to regression test directory.
setup_wrfhydro : <hydro_namelist_file> <parameter_data_folder> <namelist_hrldas_file>: Link WRFHydro test case
setup_lis <lis_config_file> : Link LIS/Noah test case config and data
setup_ww3 : Link WaveWatch3 test case config and data

6.3.2.2.5 GFS Regression Test Variables

CNTL_DIR : Control run directory. This is the source directory for control run files.
LIST_FILES : Control run file list. Files are compared using the unix cmp utility.
6.3.2.2.6 Run Script
RUN_SCRIPT: Script to be executed after component variables have been set. E.g. rt_gfs.sh, rt_nmm.sh, ...

7. How to Configure a NEMS Application

7.1 Changing Run Time, Queue, Project Number

The wall times are set in the compset, and those files are in NEMS/compsets and the variable name is WLCL, for example:

export WLCLK=30

You can set more than 60 minutes. Change the queue and project number in NEMS/NEMSCompsetRun, in the section like this:

elif [ $MACHINE_ID = yellowstone ]; then
export DATADIR=/glade/p/work/theurich/NEMS-Data
export ACCNR=P35071400
# export ACCNR=UCUB0024                                                                             
export QUEUE=small

7.2 Changing Run Directory

All outputs go into the run directory, which goes to a unique directory (e.g. under /glade/scratch/${user}). The directory will have a name like rt_*. When you run the ./NEMS/NEMSCompsetRun, you will see output like this,

RUNDIR:
'/scratch3/NCEPDEV/stmp1/Anthony.Craig/rt_128577/20150401short_nem_gsm_cice_mom5'.

7.3 Model Output Control

This is the directory where things are run and where the output data will be. The default is hourly output from all parts of the model and is a lot of data. You will want to reduce that if you are running 30 or more days. To reduce the output, in your nems.configure settings, edit this file, NEMS/tests/nems.configure.med_atm_ocn_ice.IN

and set

DumpFields = false

everywhere DumpFields is set. That will turn off output from the mediator and all the caps. Or you can selectively turn of some of the output in the different caps and mediator.

To change the GSM output frequency, edit your compset, for example NEMS/compsets/20150401long_nems_gsm_cice_mom5.

Change to:

export FHZER=6
export FHOUT=6

FHZER is the GSM accumulated fields zeroing frequency.

FHOUT is the GSM output frequency in hours.

To change the MOM output frequency, edit

NEMS/NEMSCompsetRun

and under setup_mom5, change from 1 hourly to 6 hourly output by changing

cp ${OCN_INPUT}/diag_table.1hr ${dst_dir}/diag_table

to

cp ${OCN_INPUT}/diag_table.6hr ${dst_dir}/diag_table

To change the CICE output history frequency, the ice_in namelist needs to be modified.

Edit ice_in_mine in $SOMEDIR.

histfreq = ‘m’,’d’,’h’,’x’,’x’
histfreq_n = 1,1,6,1,1,1

To add a second daily stream and a third 6-hourly stream. Change the ‘d’ or ‘h’ to ‘x’ if you do not want the extra streams. Also, you need to tell the ice what variables go on what stream.

For example:

f_hi = ‘mdhxx’
f_aice = ‘mdhxx’
f_Tsfc = ‘mdxxx’

Where ice thickness (hi) and concentration (aice) will go on all three streams, while Tsfc will only go on the first two (monthly and daily) streams. This will create history files under the ‘history’ subdirectory of the form:

iceh.YYYY-MM.nc or iceh.YYYY-MM-DD.nc, etc

You can edit the NEMS/NEMSCompsetRun file to make sure it is copying your namelist.

cp $SOMEDIR/ice_in_mine ${dst_dir}/ice_in

7.4 Changing Restart Frequency

The restart frequency is controlled in each model separately.

To modify the MOM restart frequency, edit restart_interval in the NEMS configure file, NEMS/tests/nems.configure.med_atm_ocn_ice.IN. restart_interval is set in seconds.

To modify the GSM restart frequency:

The GSM does not currently support true restart from restart files. Instead, it is able to restart from the forecast output history files. This is controlled by the variable “NDAYS” specified in the compset. For instance, if a restart after 30 days is required, set NDAYS to 30 and run the forecast. Two files from this run are needed to restart the GSM, sigf720 and sfcf720. These will be read by the GSM when the run is setup for a restart. When true restart from restart file capability works in GSM, restart file frequency will be controlled by the variable “FHRES”, specified in the compset. (For now, FHRES must be set to a number greater than “NDAYS * 24”, otherwise the model will not finish successfully.)

The CICE restart output frequency is also controlled by the namelist (ice_in). Edit your copy of the namelist: Edit $SOMEDIR/ice_in_mine

Change the variables:

dumpfreq = ‘s’
dumpfreq_n = 21600

To have 4 x daily restarts written. Acceptable units for dumpfreq are ‘s’, ‘d’, ‘m’, and ‘y’. You can edit the NEMS/NEMSCompsetRun file to make sure it is copying your namelist. cp $SOMEDIR/ice_in_mine ${dst_dir}/ice_in

The mediator is currently setup to write restarts at the end of the run and to overwrite the restarts in the run directory. To write out restarts during the run, edit the mediator restart_interval in the NEMS configure file, NEMS/tests/nems.configure.med_atm_ocn_ice.IN. restart_interval is set in seconds. Restarts written during the run with have a timestamp prepended to the filename.

7.5 Changing PET Counts (PE layout)

To change pe counts, do the following. First, edit the compset (ie. NEMS/compsets/cfsr%20150401_1day_nems_gsmslgT126_cice%0.5_mom5%0.5) and change these lines:

export TASKS=136
export atm_petlist_bounds="0 23"     
export ocn_petlist_bounds="24 55"  
export ice_petlist_bounds="56 75"   
export med_petlist_bounds="76 135"

Those specify the start and end task ID for each component as well as the total number of TASKS required.

If you have changed the CICE pe count, then you will need to modify the file CICE/comp_ice.backend and change the following lines and rebuild the model from scratch.

setenv NTASK    20      # total number of processors
setenv BLCKX    72      # x-dimension of blocks ( not including )
setenv BLCKY    205 # y-dimension of blocks (  ghost cells  )

If you have changed the MOM pe count, then you will need to modify the mom namelist input file, input.nml and set

layout = 8,4
io_layout = 1,4

If you have changes the GSM pe count, then **<information needed>="">** If you have changed the MEDIATOR pe count, the mediator will adapt automatically.

8. How to Modify a NEMS Application

8.1 Changing the run sequence

During run-time of the NEMS executable, it accesses a file called nems.configure, which it expects to find in the run directory. This file specifies the dynamic component selection, and the exact run sequence to be used. Only models built into the executable at build-time are accessible during run-time. An error will be triggered if an unavailable model is selected in nems.configure. The component selection is based on two variables:

xxx_model:         abc

Xxx_petlist_bounds:   lower upper

Here "xxx" can be "atm", "ocn", "ice", "ipm", "med". The "abc" stands for the actual instance name, e.g. "gsm" or "mom5". The lower and upper bounds of the petList specification are integer PET numbers.

The specification of the run sequence provides the flexibility needed to cover different coupling scenarios. The format is line based between special tags:

runSeq::
line1
line2
...
lineN
::

There are a number of format options for each line:

Here is an example of a run sequence specification with two time scales:
#Run Sequence#
runSeq::
    @7200.0
        OCN -> MED
        MED MedPhase_slow
        MED -> OCN
        OCN
        @3600.0
        MED MedPhase_fast_before
        MED -> ATM
        ATM
        ATM -> MED
        MED MedPhase_fast_after
        @
    @
::

Anything on a line after the "#" symbol is treated as a comment and ignored. Indentation in the formatting does not change the meaning of a line, but is purely used to increase readability.

8.2 Adding a model grid with a different resolution

Changing resolutions in coupled applications is far from an automated process at this point. There are a few reasons for this. One is that each of the components has its own procedure for changing grids, and there is no common interface. There is a constraint that ocean and sea ice grids are the same, which requires a utility to generate ice grids from the ocean grid. Additional utilities are required to generate appropriate initial conditions for all the components on the new grids. Finally, the land masks used by atmosphere, ocean, and sea ice components need to be consistent. This is achieved by having the mediator interpolate the land mask used by the ocean and ice components to the atmosphere grid.

The sequence of steps looks like this:

  1. For any component that requires a grid change, use its native procedure to incorporate the new grid. If the ocean resolution changes, generate a new sea ice grid to match the ocean grid.
  2. For any component that requires a grid change, generate new initial conditions consistent with the grid.
  3. Create a standalone (in NEMS, called side-by-side) compset for the component if one does not exist. Verify correct operation of any component with a new grid and initial conditions running the standalone compset. Create a compset for the target coupled configuration.
  4. Run the coupled configuration compset through initialization in order to generate a mask file on the atmosphere grid that is consistent with ocean and ice mask and grids.
  5. Use the atmosphere mask file to generate correct initial condition files for the atmosphere using the chgres and orographic utilities.

A run can now be performed with the changed resolution, using the compset for the coupled configuration.

The steps are explained in more detail below.

1. Change component grids using their native procedures.

FV3 (Information is needed from FV3 team).

GSM (Information is needed from GSM team).

MOM

  1. Use MOM5 build in ocean grid generation utility “MOM5/src/preprocessing/generate_grids/ocean/ocean_grid_generator.csh” to generate ocean grid specify file (grid_spec.nc) with user’s own resolution setting.
  2. Generate ocean initial conditions.
  3. Copy grid_spec.nc file and ocean initial conditions to INPUT directory.

CICE

  1. CICE will use the same grid as the ocean component.
  2. CICE requires a grid_file and a kmt_file.
  3. The grid file contains the latitudes and longitudes of the corner points as well as the lengths of the North and East faces of the gridcells and the rotation angle from the curvilinear grid to true latitude-longitude grid.
  4. The kmt_file contains the land/ocean mask where 1s are ocean and 0s are land.
  5. A Fortran code (generateCICEfromFMSgridspec.F90) will take the MOM grid definitions and generate the grid and kmt file for CICE.

2. Generate new initial conditions for any components with new grids.

FV3 (Information is needed from FV3 team).

GSM (Information is needed from GSM team).

MOM

Use “cvtR4p0To5p1” utility to convert NCEP CFSv2 initial conditions (MOM4 based) to UGCS compatible initial conditions (MOM5 based).

CICE

  1. Use the NCAR Command Language (NCL) script to take an initial state from CFS and make it CICE compatible.
  2. ncl convert_cice5.ncl
  3. Expects ice_model.res.nc as input and output is cice5_model.res.nc.

3. Use or create side-by-side (standalone) compsets for any components with new grids and verify correct operation. Create a compset for the target coupled configuration.

This sheet shows the compsets that already exist. (The ones for T574 and T1534 GSM do not) To create a new compset, see the section called How to Add a New Compset. To use CompsetRun, see How to Build & Run.

4. Generate a land mask for the atmosphere.

Masks from the ocean/ice models are interpolated to the atmosphere grid using a local area conservation scheme in the NEMS mediator. The result is a fractional land mask file, field_med_atm_a_land_mask.nc. The mediator sends this mask to the GSM. Grid cells with values equal to 1 can be considered "all land". Grid cells with values between 0 and 1 will eventually be part land, part ocean/ice. However, the GSM atmosphere cannot currently handle cells that are part land, part ocean/ice and must create a binary mask representation using a GSM cutoff parameter. This is hardcoded in the routine GSM/phys/do_physics_one_step.f in the following place:

!   -> Mask
    fldname='land_mask'
    findex = QueryFieldList(ImportFieldsList,fldname)
    if (importFieldsValid(findex) .and.
&   importData(1,1,findex) > -99999.0) then
    do j = 1, lats_node_r
        do i = 1, lonr
            aoi_fld%slimskin(i,j) = 1.0
        if (importData(i,j,findex) < 0.01) then
            aoi_fld%FICEIN(i,j) = 0.0
            aoi_fld%slimskin(i,j) = 3.0
        endif
        enddo
    enddo

Any cells with a value greater than the cutoff parameter will be land. Currently (for revisions after DREV64441) GSM sets anything greater than 0.01 to land. In GSM, the points that are associated with ocean and ice following the binary land mask and cutoff parameter receive and use ocean and ice exchange field data from those components.

To generate the field_med_atm_a_land_mask.nc file, you must run the coupled system. The mediator will generate that file during initialization. The models must be configured to run on the grids and masks of interest. The initial conditions and other scientific aspects of the model are not important at this point. In this case, it's best if no mediator restarts are used to initialize the system. The goal is to set up a run with the models on the correct grids and masks, to have the coupled system initialize, to have the mediator generate conservative mapping files between grids, for the mediator to interpolate the ocean mask to the atm grid conservatively, and to have that data written to a file. This will all happen during initialization of the system. The model can abort once initialization is complete.

In the previous step, a compset that included the coupled target components and grids was created. Use CompsetRun to run this compset through initialization following How to Build and Run. You may need to setup new PET layouts and decompositions for the new grids. And you need to carefully check the mediator output grid files after this short initial run is complete to verify that the grids in the mediator are consistent with the new grids defined by the models. After the compset runs, you should have the field_med_atm_a_land_mask.nc file in your run directory.

5. Run chgres and orographic utilities to get appropriate input files for GSM.

Once the land mask for the atmosphere is available, it is necessary to run the chgres and orography utilities in order to get appropriate input files for GSM. The inputs generated by chgres are siganl and sfcanl, which are initial conditions for the atmospheric model. The siganl file contains the sigma level spherical harmonic spectral coefficients in a binary file of the surface pressure, orography, and remaining model 3D dependent variables and tracers. The sfcanl file contains surface and land surface files in physical space (Gaussian grid) used by the physics component.

In the directory of the files is an ieee file and a GrADS control file that reads and plots the ieee file. The second record of the ieee file is the sea-land mask made by the code. The utility chgres uses the sea-land mask (slm) grib file output made by the orography code, if it is present, with the chgres SLMASK exported variable pointing to the Gaussian grib slm file which in turn was made from landmast.txt input to the orography code from the field_med_atm_a_land_mask.nc netcdf file. In the absence of the export variable SLMASK being set explicitly, the sea-land mask Gaussian file will attempt to be found by chgres in the model fix fields directory and thus be made from the USGS 30" elevations, and the UMD 30" ocean/lake/land mask file. The same can be expected for the orography file if the export variable OROGRAPHY is not set to a new orography file or the file is not present. The cutoff rule parameter may be changed with recompilation of the orography code.

The sea land mask cutoff parameter (what is the actual parameter name and where is it located?) used to create the set of mask files received on 10/5/15 is believed to be 0, so any cells with value greater than 0 were considered land. This is what has been used to run DREV64441.

After generating the 10/5/15 files, the sea land mask cutoff parameter was set to 0.5, so any cells with a value greater than 0.5 in the fractional netcdf ocean file would be considered land. A new set of mask files was received on 11/17/15 corresponding to the 0.5 cutoff parameter.

Additional files generated by the orography utility and chgres are:

File Format:

The headers for the signanl/sfcanl file inputs are of the form:

sighdr siganl.gfs.2015040100
jcap
126
lonb
384
latb
190
idate
2015041500
stop
global_sighdr ENDING …

and

sfchdr sfcanl.gfs.2015040100
lonb
384
latb
190
idate
2015041500

8.2.1 Potential improvements

Generating atmosphere land mask offline. The entire "run the coupled system through initialization" process could be substituted with a preprocessing step that leverages ESMF to generate a conservative mapping file offline to interpolate the ocean mask to the atm grid offline. That would require the ability to generate ESMF compatible grid files offline for any grids of interest, including corner points in a way that the grid information is identical to the information in the models. Right now, we are using the coupled system to take care of this preprocessing step.

Consistency improvement. It would be preferable if the atmosphere did not choose or determine mask values. The mediator would determine ocean/ice and land grid cells, and send appropriate data. Ideally the mask used by the mediator would be automatically consistent with the mask used for chgres and orography file generation.

Fractional cells. The atmospheric model should be able to handle grid cells that are divided into ice/ocean/land portions.

9. How to Create a NEMS Application

In order to use the AppBuilder, a “project” directory and repository at EMC needs to be set up with an AppBuilder configuration file. This can be done either with existing EMC projects or new projects. The approach is outlined below.

9.1 How to create a *new* NEMS application that uses AppBuilder

We want to create a project ABC that does not exist yet. An EMC contact first needs to set up a new repository/directory on the EMC server: /projects/ABC

# comments are okay

NEMS -r 123456 https://svnemc.ncep.noaa.gov/projects/nems/trunk

WW3 -r 234567 https://svnemc.ncep.noaa.gov/projects/ww3/branches/esmf2/model

MOM5 -r 456 https://github.com/feiliuesmf/mom/branches/mom5_5_0_2_nuopc

MOM5_CAP -r 12 https://github.com/feiliuesmf/nems_mom_cap/trunk

CICE -r 65 https://github.com/feiliuesmf/lanl_cice/trunk

CICE_CAP -r 32 https://github.com/feiliuesmf/lanl_cice_cap/trunk

./NEMS/compsets/*
./NEMS/NEMSCompsetRun -compset 
./NEMS/compsets/20150401short_nems_gsm_cice_mom5

People can add to this global collection of compsets, or can create compsets locally. For local compsets, we recommend adding a compset directory to the current directory (projects/ABC/trunk/). [This page discusses compset syntax]().

9.2 How to create a NEMS application that uses AppBuilder for an existing EMC project

We recommend creating a new directory that sits at the same level as the current trunk, e.g. transition_trunk, and treating that transition_trunk as the current directory. Associated directories might be called transition_branches, transition_tags, etc. Eventually these should be renamed to trunk, branches, tags once the transition is complete and the previous trunk can be removed.