NEMS Reorganization and New Commit Process

Executive Summary

The current structure of the NEMS repository makes it difficult for NEMS applications to keep up to date with updates to the NEMS framework. This reorganization moves files out of the framework's repository that should be versioned elsewhere. As a consequence, it will be possible for regression testing of coupled applications. To this end, we have a new commit process for the NEMS framework.


The reorganization moves many files from the NEMS Framework level to the Application level. This allows applications to keep up with the NEMS framework trunk, and to configure test suites on an application-by-application basis. We also rewrite the test system to allow coupled testing and rearrange documentation.

We refer to three Subversion repository areas within this document:

NEMS Framework

NEMS Applications


Directory Structure Before Reorganization

A typical NEMS application contained the following:

The contents of the NEMS framework before the reorganization was:

The reorganization moves files that are application-specific to the application level, moves the GSM and NMM components to the application level, and replaces the regression testing implementation with a more flexible one that allows coupled regression tests. The next section will explain the new regression test system in more detail. This section describes the new directory structure.

All paths are relative to the top of an app checkout.

What is Moved Moved From
NGGPS main page
GFS model

What is Moved Moved From Moved To
Test configuration NEMS/compsets compsets
Test configuration NEMS/tests compsets
Test suite logic NEMS/tests NEMS/tests/
  • produtil
  • rtgen, rtreportimpl
  • (simple wrapper)
Modulefile Selection *.appBuilder shell functions
Make Configuration NEMS/conf/configure*conf
Old test suite NEMS/tests oldtests
Build logs (NEMSAppBuilder deletes them) logs
Documentation Scattered, some non-existent doc - application-specific
NEMS/doc - framework docs

The NEMS application will look like this after the reorganization:

The NEMS framework will look like this after the reorganization:

New Commit Process

The removal of NEMSLegacy means there is no way to test NEMS framework changes unless we use a new test system and commit process. The new system and process are detailed here.

NEMS Application Test System

All applications will have a list of compsets that use the new compset runner. This new compset runner does not use any model-specific workflow scripts and minimizes per-computer configuration changes. This is accomplished by using the produtil.testing Python package to generate the needed test scripts from a simple description of the test to be run. Rocoto or ecFlow are used to automate the execution of those generated scripts.

The test system requires the maintenance of the following information for each test or compset:

  1. A set of input data.
  2. NEMS execution information (MPI ranks, OpenMP thread count, etc.)
  3. List of files to validate.

From that information, a bash-based test system is generated in a temporary area. A Rocoto or ecFlow workflow description is generated to run the test suite. For users that are not modifying tests, and only want to run them, there is an ? script that mimics the old test system.

Application Tiers

Applications are divided into three tiers: Supported, Diverged, and Unsupported. Supported applications will be listed in a readme file in the apps area.

Supported - changes to the NEMS framework cannot break these applications. To be supported:

Diverged - The NEMS framework updates are allowed to break these applications.
It is the responsibility of the application developer to modify their application as needed to adapt to NEMS framework changes. However, developers of Diverged applications can still request a commit to the NEMS trunk. To be a Diverged application:

Unsupported - These are applications that do not satisfy the requirements for being Diverged. They are moved out of the projects/nems/apps directory into projects/nems/apps/unsupported.

Note that the projects/nems/apps contains directories that fit into a fourth category:

Future - these are placeholder directories that will become apps.


This section only deals with Supported applications. Diverged, Unsupported, and Future applications have no restriction on how they maintain their application, nor any requirement to maintain working compsets, but they cannot request the NEMS Framework Team run their application's compsets.

Application Team Responsibilities

  1. Provide an application branch or trunk that:
    a. Has at least one compset that always works and uses the new compset runner.
    b. Has a NEMS external pointing to the head of the NEMS Framework trunk
  2. Modify or add compsets in advance of NEMS Framework commits, in collaboration with developers that propose the NEMS Framework commit.
  3. If the application needs to run on a new platform, application team must port it.
  4. Ensure the application uses a reasonably up-to-date version of its components.
  5. Provide a list of people, in priority order, to perform these duties.

NEMS Framework Team Responsibilities:

  1. Maintain the test system implementation (NEMS/tests) on all NEMS platforms.
  2. Email all NEMS developers upon any commit to the NEMS Framework trunk.
  3. Run compsets for Supported applications in a clean test area when requested.
  4. Automatically run compsets for all Supported applications on a regular basis.
  5. Provide a list of people, in priority order, to perform these duties.

NEMS Commit Process

This section deals with the commit process to the NEMS Framework. Note that the only applications we refer to here are Supported applications. Updates to the NEMS Framework are allowed to break applications that are not Supported.

As a reminder, this is the NEMS Framework Trunk:

Branches go in the "branches" subdirectory. Here is where the Example branch would be:

STEP 1: The developer makes a NEMS Trac ticket and uses it to track the status of development of their branch. When the branch is ready to commit, they note that in the ticket.
STEP 2: The developer announces to the NEMS list an intent to commit, description of the changes, the NEMS Ticket number, and branch information.
STEP 3: The NEMS Framework Team tests the NEMS change with all applications
STEP 4: The NEMS development community evaluates the code change. This includes:

STEP 5: If needed, the developer works with the Application teams to update apps to support the NEMS Framework change. This may be an iterative process, and may require the NEMS Framework Team to rerun the compsets.
STEP 6: The NEMS Framework Team re-runs the compsets one final time.
STEP 7: The NEMS Framework Team updates the NEMS Framework Trunk, as well as any external documentation or other resources.