Skip Navigation Links www.nws.noaa.gov
NOAA logo - Click to go to the NOAA homepage National Weather Service NWS logo - Click to go to the NWS homepage
EMC Logo
Navigation Bar Left Cap Home News Organization
Navigation Bar End Cap

         MISSION / VISION    |    About EMC

EMC > NEMS > GETTING STARTED
NOAA Environmental Modeling System

Regression Tests



Introduction:


This page will help get started with the basics and provide what you need to know to run NEMS regression tests.

Before continuing, some information:
  • Links to the right provide navigation to specific sections.
  • This page is for internal NCEP users, i.e., those who can access the R&D (Zeus) or WCOSS machines (Tide/Gyre), and have a Subversion account.
    • If you are working on Jet or Bluefire, you can find more information here.
  • This page assumes you are new to using NEMS and running NEMS regression tests.
  • If at any time you are confused and cannot find the information that you need please, email for help.




Purpose of Regression Tests:


Regression testing attempts to uncover errors by partially retesting a modified program, and its intent is to provide a general assurance that no additional errors were introduced into the model.

What do we test?
  • Bit-identical result reproducibility
  • Different domain decomposition
  • Threading
  • Different physics options
  • Timing (only NMM-B, but will eventually also be done for GFS)
  • Different compilations
  • And many more...
Currently, we do not test forecast verification (scores and skills).

The NEMS regression test suite is separated into two sets, FULL and STANDARD. The STANDARD set should be run before any repository commit. It takes about 1.5 hours to finish and consists of 8 NMM-B tests, 3 GFS tests, 1 GEFS, and 1 WAM test. Those tests are highlighted in red in the regression test details section below. The FULL set is run from a cron once a week on Saturday to make sure that no code was broken from any recent commits. It takes about 7 hours to finish and has 48 tests and 5 different compilations (ESMF3, ESMF3 with POST, ESMF5, NMM-B only, and GFS only). There are also additional tests to determine compatibility with NUOPC.

Return to top




NEMS Regression Test Details:


Currently, there are three models that can be run under the NEMS framework: the Global Forecast System (GFS), Global Ensemble Forecast System (GEFS), and the B-grid version of the Nonhydrostatic Multiscale Model (NMM-B), none of which are the operational versions.

There are several compilations in the FULL set of tests, including the following:
  • Compiled with ESMF3.1.0r, all cores and GOCART.
  • Compiled with ESMF3.1.0r, all cores and POST.
  • Compiled with ESMF5.2.0r API, all cores.
  • Compiled with ESMF3.1.0r, only NMM-B.
  • Compiled with ESMF3.1.0r, only GFS.
Original NEMS regression test data is located in directories similarly to the following schematic:



The method to run the regression tests for these models is the same, but each model has its own set of regression tests. Following is a basic schematic to visualize the process.



Within RT.sh_IN, there are options at the beginning of the file to allow you to skip specific tests. Uncomment any of the skip options to skip those tests. Some options include SKIP_NMM, SKIP_GFS, SKIP_GFS_POST. For the full list of options, refer to RT.sh_IN.

Below, you will find more specific information about the various regression tests that can be run in NEMS, all of which are run under the first compilation mentioned in the above list (ESMF3.1.0r, all cores, GOCART). The tests highlighted in red are run in the STANDARD set. Based on trunk version 30777.

NMM-B:

Unless otherwise stated, all NMM-B regression tests use 6x5 compute tasks, 1 thread, and opnl (operational) physics.
    1 - Baseline global to compare with previous trunk version: free fcst, pure binary input.
    2 - Baseline global with NEMSIO input: free fcst, NEMSIO input.
    3 - Global restart run: restart, pure binary input.
    4 - Global restart run with NEMSIO input: restart, NEMSIO input.
    5 - Global with different domain decomposition: 3x5 compute tasks with single thread, opnl physics, free fcst, pure binary input.
    6 - Global with multiple threading: 6x5 compute tasks with 2 threads, opnl physics, free fcst, pure binary input.
    7 - Global with GFS physics: GFS physics, free fcst, pure binary input.
    8 - Baseline regional to compare with previous trunk version: free fcst, pure binary input.
    9 - Baseline regional with NEMSIO input: free fcst, NEMSIO input.
    10 - Regional restart run: restart, pure binary input.
    11 - Regional restart run with NEMSIO input: restart, NEMSIO input.
    12 - Regional with different domain decomposition: 3x5 compute tasks with single thread, opnl physics, free fcst, pure binary input.
    13 - Regional with multiple threading: 6x5 compute tasks with 2 threads, opnl physics, free fcst, pure binary input.
    14 - Regional with GFS physics: GFS physics, free fcst, pure binary input.
    15 - Regional with selected GFS physics schemes: selected GFS physics, free fcst, pure binary input.
    16 - Regional with separate species advection ON: 6x5 compute tasks with 1 thread, NEMSIO input.
    17 - Regional restart with separate species advection ON: 6x5 compute tasks with 1 thread, free fcst, pure binary input.
    18 - Regional with selected GFS physics schemes (WSM6, GFDL): 6x5 compute tasks with 1 thread, selected GFS physics, free fcst, pure binary input.
    19 - Regional with selected GFS physics schemes (WSM6, RRTM): 6x5 compute tasks with 1 thread, selected GFS physics, free fcst, pure binary input.
    20 - Regional with selected GFS physics schemes (SAS, ZHAO, RRTM): 6x5 compute tasks with 1 thread, selected GFS physics, free fcst, pure binary input.
    21 - Regional NMM-B with THOMPSON microphysics and RRTM radiation: 6x5 compute tasks with 1 thread, THOMPSON and RRTM, free fcst, pure binary input.
    22 - Regional with precipitation adjustment on: free fcst, pure binary input.
    23 - Regional writing time series: free fcst, pure binary input.
    24 - Regional with static nesting: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, pure binary input, compute tasks:
    • upper parent: 2x3
    • child 1: 4x8
    • child 2: 2x4
    • grandchild: 7x10
    25 - Regional restart with static nesting: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, pure binary input, compute tasks:
    • upper parent: 2x3
    • child 1: 4x8
    • child 2: 2x4
    • grandchild: 7x10
    26 - Using RRTM radiation (static nests) : regional parent with two children and one grandchild, single thread, test physics, free fcst, pure binary input, compute tasks:
    • upper parent: 2x3
    • child 1: 4x8
    • child 2: 2x4
    • grandchild: 7x10
    27 - Using RRTM radiation (restart static nests): regional parent with two children and one grandchild, single thread, test physics, free fcst, pure binary input, compute tasks:
    • upper parent: 2x3
    • child 1: 4x8
    • child 2: 2x4
    • grandchild: 7x10
    28 - Using RRTM threading: 2 threads, test physics, free fcst, pure binary input.
    29 - Regional static nests with digital filter: regional parent with one child and one grandchild, single thread, opnl physics, free fcst, NEMSIO binary input, compute tasks:
    • upper parent: 2x2
    • child 1: 3x5
    • grandchild: 6x7
    30 - Regional moving nests: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO and pure binary input, compute tasks:
    • upper parent: 8x8
    • child 1: 2x6
    • child 2: 2x6
    • grandchild: 5x6
    31 - Regional moving nests (restart): regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO input, compute tasks:
    • upper parent: 8x8
    • child 1: 2x6
    • child 2: 2x6
    • grandchild: 5x6
    32 - Regional moving nests with generational task assignments: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO and pure binary input, compute tasks:
    • upper parent: 5x18
    • child 1: 5x9
    • child 2: 5x9
    • grandchild: 10x9
    33 - Regional 2-way moving nests with generational task assignments: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO and pure binary input, compute tasks:
    • upper parent: 5x18
    • child 1: 5x9
    • child 2: 5x9
    • grandchild: 10x9
    34 - Regional 2-way moving nests with generational task assignments (restart): regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO and pure binary input, compute tasks:
    • upper parent: 5x18
    • child 1: 5x9
    • child 2: 5x9
    • grandchild: 10x9

GFS:

GFS in NEMS is not the same as the operational GFS model. There are big differences between the two, although their results are almost the same due to using the same science component. All of the GFS tests run on T62, 64 levels on global domain, 4 soil layers, diabatic run on reduced grid. Restart files are created every 24 hours.

    1 - Baseline to compare with previous trunk version: Runs on 32 pe, 30 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
    2- Compare adiabatic results with previous trunk version: Runs on 32 pe, 30 forecast tasks, with single thread, quilting, no digital filter.
    3 - With 2-copy option: Runs on 32 pe, 30 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics have (allocated) their own copy of model variables, restarts from 24 hours, integrates to 48 hours, output at every 6 hours.
    4 - With different decomposition: Runs on 60 pe, 58 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
    5 - With multiple threads: Runs on 16 pe, 12 forecast tasks and 2 write group/2 processors per write group, with 2 threads, quilting, no digital filter, dynamics and physics share one copy of model variables, restarts from 24 hours, integrates to 48 hours, output at every 6 hours.
    6 - 32 processors, 1 thread, with frequent output: Runs on 32 pe, with single thread, output every 4 timesteps.
    7 - With single processor with no quilting: Runs on 1 pe with single thread, nsout=1.
    8 - With multiple threads & output at every 2 time steps: Runs on 16 pe with 16 tasks (16 forecast, 1 write), 2 threads, no quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 2 time steps.
    9 - With 48 processors: With single thread, no quilting, no digital filter, dynamics and physics share one copy of model variables, restarts from 24 hours, integrates to 48 hours, output every 2 time steps.
    10 - With 32 processors and digital filter: Runs on 32 pe with 32 tasks (30 forecast, 2 write), with single thread, quilting, digital filter on reduced grid, dynamics and physics share one copy of model variables, integrates 24 hours, output every 6 hours.
    11 - With 16 processors and digital filter: Runs on 16 pe with 16 tasks (12 forecast, 2x2 write), 2 threads, quilting, digital filter on reduced grid, dynamics and physics share one copy of model variables, restarts from 24 hours, integrates to 48 hours, output every 6 hours.
    12 - With digital filter, Hyb 2 loop: Runs on 16 pe with 12 compute tasks, 2x2 write, 2 threads, quilting, HYB 2 loop digital filter on reduced grid.
    13 - With digital filter restart, Hyb 2 loop: Runs on 16 pe with 16 compute tasks (12 forecast, 2x2 write), 2 threads, quilting, HYB 2 loop digital filter on reduced grid.
    14 - WAM enthalpy on general hybrid 2 loop:150 levels, 32 compute tasks, single thread, no quilting
    15 - GFS with NEMSIO: without post quilting.
    16 - GFS_GOCART with NEMSIO: without post quilting.
    17 - Semi-Lagrangian: 96 tasks, T574L64, 24hr fcst, adiabatic, no digital filter.
    18 - Semi-Lagrangian with digital filter: 96 tasks, T574L64, 24hr fcst, adiabatic, digital filter.
    19 - Concurrency GEFS: T190 resolution, 28 vertical levels, run on 4 ensemble members (one member for the control with no initial perturbation added, 3 ensemble run members with initial perturbations), using 64 CPU tasks on 1 node, runs 24 hours, and at every 6 hours couples all ensemble members together, adds the stochastic perturbations to represent the model uncertainties, the re-sets up the initial conditions for the next 6 hours.

FIM

    Compare FIM results with previous trunk version, only FIM compiled: 40 compute tasks, 1 thread, 1 node.

Tests for post:

    1 - NMM-B regional with post on quilt: free fcst, pure binary input.
    2 - Baseline GFS to compare with previous trunk version: Runs on 32 pe, 30 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
    3 - GFS with GOCART: Runs with 64 compute tasks, reproduces the Q4Y12 NGAC production except for G2 products.

Tests with NUOPC:

    1 - Baseline global NMM-B: free fcst, pure binary input.
    2 - GSM with stepable Run() driven in 12h intervals: 30 compute tasks, with single thread.
    3 - GSM <-> Mediator <-> DummyOCN: from data with 12h coupling interval, exchanging dummy fields.
    4 - GSM <-> Mediator <-> MOM5: 12h coupling interval, exchanging dummy fields.
    5 - GSM + HYCOM: side-by-side from data with 6h intervals.
    6 - GSM + MOM5: side-by-side from data with 2h intervals.

Following are the lists of tests included in the other compilations under the FULL set of tests:

Compiled with ESMF 5.2.0r API, all cores
  • Compare NMMB-global results with previous trunk version
  • Compare GFS results with previous trunk version
  • Concurrency GEFS, stochastic perturbations, 4 members, T190L28
Compiled with ESMF 6.3.0r API, all cores
    Compare NMMB-global results with previous trunk version
  • Compare GFS results with previous trunk version
  • Compiled with ESMF3, only NMM-B core
    • Compare NMMB-global results with previous trunk version
    Compiled with ESMF3, only GFS core
    • Compare GFS results with previous trunk version

    Return to top




    Directories and Regression Test Scripts:


    * - 1 or 2
    ** - global, meso, nems, etc

    Recommended Directories In Which To Store NEMS:


    /gpfs/td1/emc/**/save/Firstname.Lastname (Tide)
    /gpfs/gd1/emc/**/save/Firstname.Lastname (Gyre)
    /scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname (Zeus)



    Location Of Regression Test Run Scripts:


    /gpfs/td1/emc/**/save/Firstname.Lastname/trunk/job/regression_tests (Tide)
    /gpfs/gd1/emc/**/save/Firstname.Lastname/trunk/job/regression_tests (Gyre)
    /scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname/trunk/job/regression_tests (Zeus)



    Files in Regression Test Directory:

    .../job/regression_tests  Sampling of scripts used in the NEMS regression tests
      Chem_Registry.rcUsed to control basic properties of the GOCART and StratChem Grid Components; specifically, it
      • selects which constituents to simulate.
      • selects the number of bins for each constituent.
      • specifies variable names and units for each constituent.
      Compile_*.logResults of the compilation.
      detect_machine.shDetects the machine being used.
      MAPL.rcMinimum MAPL resource file.
      RT.sh_INScript to run regression tests; performs the following:
      • cleans and compiles model cores.
      • runs regression tests.
      RT.shDriver to run RT.sh_IN.
      RegressionTests.logResults of the regression tests.
      rt_nmm.shMakes configure and run files for NMM-B.
      nmm_llLoadleveler script to run NMM-B regression tests.
      rt_gfs.shFor stand-alone GFS tests, makes configure and run files for GFS.
      gfs_fcst_runLoadleveler script to run GFS and GEFS regression tests.
    Return to top




    Running the Standard Set of Regression Tests:


    The following provides information about how to run the NEMS regression tests.

    This is relatively simple to do, although it will take a few hours to complete. The 'STANDARD' set is most likely what you would run, and is required before a commit to the NEMS trunk. The 'FULL' set is on a cron, set to run every Saturday, and contains full listthe of regression tests and various compilations.

    ActionActive DirectoryCommand
    Run 'STANDARD' set of regression tests /gpfs/td1/emc/**/save/Firstname.Lastname/trunk/job/regression_tests (Tide)
    /gpfs/td1/emc/**/save/Firstname.Lastname/trunk/job/regression_tests (Tide)
    /scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname/trunk/job/regression_tests (Zeus)

    * - 1 or 2
    ** - global, meso, nems, etc.
    'sh RT.sh'
    Create baseline(s)Same as aboveUse one argument (i.e. nmm, gfs, gen, or all)
    e.g. 'RT.sh nmm'
    Run 'FULL' set of regression testsSame as above'RT.sh full test'

    Details on whether or not each test finished successfully will be found in RegressionTest.log.


    Return to top




    Understanding Output:


    Output from the NEMS regression tests is written to your /ptmp directory (i.e. /gpfs/s4/scrub/ptmp/userid, /gpfs/c4/scrub/ptmp/userid, or /scratch*/portfolios/NCEPDEV/**/ptmp/Firstname.Lastname (* - 1 or 2, ** - global, meso, nems, etc)). NOTE: To view output from regression tests and not have it erased upon completion, make sure the following line in RT.sh_IN is commented out, 'rm -rf ${RUNDIR_ROOT}'.

    NMM-B output files are written in a gridded format readable by GrADS (.ctl). GFS files are written in Grib format and can easily be processed into the same format as the NMM-B files. See the above tab, 'GFS Post' for more information regarding how to process GFS output.

    For a description of the various output variables, please reference 'Output'.

    Return to top




    Notes:


    CLEAN UP:
    Please be aware that files in the /ptmp directory are scrubbed after they are 10 days old. If you want to keep specific files, they should be moved to another directory that is not scrubbed on a regular basis (i.e. /global/save, /meso/save, /u/userid, etc.)

    PATH:
    Some scripts assume that "." is included in the users PATH. Check for this if unexpected errors occur. (The error msg isn't always clear).

    Return to top


  • Contributors: Ratko Vasic, Jun Wang