PURPOSEWorkshop on the

USE OF ENSEMBLES IN DATA ASSIMILATION

April 13-14, 1999, NCEP, Camp Springs, MD

*Return
to EMC ensemble home page*

8:30-9:00 Refreshments

*INTRODUCTION*

9:00 Louis Uccellini, Director,
NCEP:

Welcoming Remarks

9:05 Steve Lord, Acting Director, EMC:

Purpose and expectations of the workshop

9:15 John Derber:

Basic requirements for the
use of ensembles to define background error covariances

9:45 Discussion

**ENSEMBLE-BASED ANALYSIS SCHEMES**

Chair: Dale Barker

10:00 Peter Houtekamer:

Prospects
for an operational ensemble Kalman filter

10:30 Break

10:45 Jeff Anderson:

Application of a fully
non-linear filter and Monte Carlo techniques to ensemble data assimilation
in intermediate models

11:15 Craig Bishop:

A Survey of Ensemble Kalman Filters

11:45 Discussion

12:00 Lunch brought in

13:00 Jan Barkmeijer:

Recent
developments in the ECMWF Ensemble Prediction System

13:30 Martin Ehrendorfer (Presented
by J. Barkmeijer):

Kalman filtering and ensemble prediction

13:45 Chris Snyder:

Dynamics
and statistics of forecast errors in a quasi-geostrophic model

14:00 Tom Hamill:

A
Combination 3DVAR-Ensemble Kalman Filter Approach to Data Assimilation

14:15 Discussion

14:30 Break

**ENSEMBLES SUPPORTING EXISTING ANALYSIS SCHEMES**

Chair: Jeff Anderson

14:45 Dale Barker:

The
specification and use of synoptically-dependent background errors in 3DVAR
using information from an Error Breeding Cycle

15:15 Eugenia Kalnay:

Some ideas on the possible
use of bred vectors in data assimilation

15:45 Jeff Whitaker:

Spread/analysis
error relationships in a simple model

16:05 Zoltan Toth:

Estimating
analysis uncertainty using the NCEP global ensemble

16:35 Discussion

17:00 Workshop Report Structure - Discussion/Report
leader: Kerry Emanuel

Suggested topics
Working group leaders
Rapporteurs

a) Required ensemble characteristics
Jan Barkmeijer
Istvan Szunyogh

b) Use of ensembles with existing
analysis schemes
Wan-Shu Wu

c) Ensemble-based analysis schemes
Craig Bishop
Milija Zupanski

17:30 Adjourn for day

18:30 Optional dinner

*Wednesday, April 14*

8:00 Refreshments

**ENSEMBLES SUPPORTING EXISTING ANALYSIS SCHEMES
(Cont.)**

Chair: Eugenia Kalnay

8:30 Jim Purser:

Ensemble
guidance in defining adaptive covariances

8:50 Milija Zupanski

Plans for using fully
flow-dependent background error covariance information in the NCEP regional
4DVAR data assimilation system

9:10 Discussion

9:30 Break

10:00 Summary of presented material (Report
Leader and Rapporteurs)

Writing assignments established

11:00 Writing (In 2-3 groups)

12:00 Lunch brought in

13:00 Writing continues (In 2-3 groups)

14:00 Break

14:30 Discussion: Review and Revise Group Reports; Finalize Workshop Report

16:00 Adjourn

Jeff Anderson, GFDL, Princeton, NJ

Stephen Anderson, Metron, Reston, VA

Dale Barker, UK Met. Office, Bracknell, UK

Jan Barkmeijer, ECMWF, Reading, England

Craig Bishop, Pennsylvania State University,
State College, PA

Steve Cohn, NASA GSFC, Greenbelt, MD

Kerry Emanuel, MIT, Boston, MA

Brian Etherton, Pennsylvania State University,
State College, PA

Peter Houtekamer, AES, Dorval, Canada

Tom Hamill, NCAR, Boulder, CO

Eugenia Kalnay, University of Oklahoma, Norman,
OK

Sharan Majumdar, Pennsylvania State University,
State College, PA

Zhao-Xia Pu, USRA - NASA GSFC, Greenbelt,
MD

Chris Snyder, NCAR, Boulder, CO

Jeff Whitaker, CDC, Boulder, CO

*
From NCEP:*

John Derber

Geoff Dimego

Steve Lord

Hua-Lu Pan

Dave Parrish

Jim Purser

Istvan Szunyogh

Zoltan Toth

Steve Tracton

Wan-Shu Wu

Dusanka Zupanski

Milija Zupanski

Application of a Fully Non-Linear Filter and Monte
Carlo Techniques

to Ensemble Data Assimilation in Intermediate
Models

A probabilistic approach to the fully non-linear
filtering problem is

developed in the context of the problem of data
assimilation for

atmospheric and oceanic models. The goal of this
method is to produce a

probability sample of the state of a dynamical
system that is

consistent with a set of temporally discrete
observations. This sample

can then be used as initial conditions for ensemble
forecasts which

themselves approximate probability samples of
the forecast state of the

system. Both the state of the assimilating model
and the set of

observations available at a given instant of
time are treated formally

as random variables. The state of the system
given a new set of

observations can then be computed, after an application
of Bayes' rule,

as a convolution of the conditional probability
densities associated

with the new observations and the prior density
generated from a

knowledge of the model's dynamics and all previous
observations.

Traditionally, this problem has been simplified
through linearization

leading to the Kalman-Bucy filter. However, in
the approach discussed

here, a Monte Carlo (ensemble) approach is used
to sample the prior

density and an expanded Monte Carlo sample is
generated. This expanded

Monte Carlo sample can then be convolved with
the conditional

distribution from the new observations. An updated
probability sample

of the state of the system is then generated
by subsampling the

convolution of the expanded Monte Carlo sample.
A number of interesting

issues related to applying Monte Carlo methods
in this context are

addressed.

Presentations at previous meetings have shown
results in low order models.

This talk will quickly review the method and
its use of ensembles. The

method will then be extended for application
in models with many degrees

of freedom and results presented for a spectral
barotropic vorticity

model on the sphere and a grid-point PE model.

The Specification and Use of Synoptically-Dependent
Background Errors

in 3DVAR using information from an Error Breeding
Cycle

The specification and use of synoptically-dependent background errors in 3DVAR using information from an Error Breeding Cycle

A study is currently under way at the UKMO to
use 3D synoptically-dependent

background error modes (SBEMs) within 3DVAR.
Current background errors are

`static', derived via the `NMC' method.
An error-breeding cycle is used to

provide SBEMs which are used in 3DVAR via a new
control variable and cost

function. Details of the methodology and early
results will be presented.

Recent developments in the ECMWF Ensemble Prediction System

Perturbations used in ensemble forecasting ask
for a careful computation.

One of the conditions they should satisfy is
that their statistics

resemble what is known of the analysis error.
Prelimenary results on the

EPS performance will be presented of singular
vectors computed

with a Reduced Rank Kalman Filter. Such singular
vectors are constrained

by the analysis error covariance matrix at initial
time. This is achieved

by using the Hessian of the full 4D-Var costfunction.
Also the use of different

analyses in the EPS or the construction a so-called
consensus analysis

will be discussed.

A Survey of Ensemble Kalman Filters

A variety of ensemble Kalman filters are tested by performing

OSSEs on an idealized barotropic vorticity equation model.

For fixed (non-adaptive) observational networks, the effectiveness
of

the schemes is sensitive to the manner in which the ensemble perturbations

are generated. Important issues that one must consider when creating

an ensemble for data assimilation include the following:

(a) the ensemble perturbations need to be

well resolved by the observational network

in order to avoid spurious representations

of the part of the vector subspace

not represented in the ensemble;

(b) spurious long distance correlations need to be avoided; and

(c) perturbations which obey linear dynamics at the assimilation time

may linearly combine into a likely prediction error more readily than

perturbations which are profoundly non-linear at the assimilation time.

Interestingly, ensemble generation schemes such as those that accurately
sample

the PDF or those that "optimally" estimate the leading eigenvectors
and

eigenvalues of the prediction error covariance matrix do not necessarily

have the desirable characteristics mentioned above. In our particular
OSSEs,

ensemble generation techniques optimized for the PDF or prediction
error

variance perform badly as a basis for data assimilation.

We found that the most effective ensemble Kalman filter is one in which
the

initial perturbations are

based on the eiegnvectors of an isotropic correlation matrix scaled
by

the square root of their eigenvalues. This choice of initial perturbations

yields evolved perturbations with the three desirable characteristics
mentioned

above. This Kalman filter significantly out performs 3-D Var with isotropic

error covariances.

In order to avoid the problem of diminishing ensemble spread,

all of our Kalman filters (except that which is based

on the Houtekamer and Mitchell (1998) scheme) include some sort

of on-line estimation of prediction error covariance parameters

(cf Dee 1995, MWR). This on-line estimation is made easier by an ensemble

transformation approach that also obviates the need to invert large
matrices.

Kalman filtering and ensemble prediction

Martin Ehrendorfer (Presented by J. Barkmeijer)

A full Kalman filter has been developed in the context of a 3-level

quasi-geostrophic model. Results of 3D-Var cycling experiments with

this full Kalman filter are presented. In particular, the impact of

observations on the structure of singular vectors and error covariance

matrices will be discussed.

A Combination 3DVAR-Ensemble Kalman Filter

Approach to Data Assimilation

Thomas M. Hamill and Chris Snyder

We present an alternative methodology for ensemble-based

data assimilation, based on existing Perturbed Observation

and Ensemble Kalman Filter techniques. Our technique

assumes forecast error covariances can be modeled to be

a linear combination of time-invariant (3DVAR), spectrally

diagonal covariances and flow-dependent (Ensemble Kalman

Filter) covariances. This design has several appealing

characteristics. The 3DVAR term eliminates

rank deficiency problems; no cutoff radius is assumed, so

analyses are spatially smooth; and the technique works even

for limited-size ensembles, as the ratio of the two

background error terms may be adjusted, consistent with

ensemble size. Results from perfect model simulations

will be presented.

Prospects for an operational ensemble Kalman filter

An ensemble of short-range forecasts can provide
the flow-dependent

covariances of the forecast error, needed by
the Kalman filter.

The finite ensemble size causes the estimated
correlations to be noisy.

To filter small forecast-error correlations associated
with remote

observations, a Schur (termwise) product of the
covariances of the

forecast error and a correlation function with
local support is used.

To solve the Kalman filter equations, the observations
are organized

into batches which are assimilated sequentially.
The ensemble of

background fields is updated at each step, and
thus provides a measure

of the improving quality of the background fields
as more and more

batches of observations are assimilated. For
each batch, a Cholesky

decomposition method is used to solve the linear
system of equations.

Observations from several regions of the globe
may be selected for a

single batch, such that information from different
regions

has zero correlation due to the Schur product.
The linear system

then becomes block diagonal.

A prototype sequential filter has been developed
for atmospheric

data assimilation. Application in real time would
appear to be

feasible.

Some ideas on the possible use of bred vectors in data assimilation

I will briefly review several potential uses such as a) minimization
of the

distance between observations and the first guess along the space of
the bred

vectors (Kalnay and Toth, 1994); b) use of the ensemble spread to increase
the

weight of the observations in areas with large short range ensemble
spread (Pu

et al, W&F, 1997); c) use of the ensemble and the quasi-inverse
of the linear

model to identify "the best regional initial conditions" given the

verification

of a short range ensemble forecast.

Ensemble guidance in defining adaptive covariances

By the application of spatial smoothing filters along

generalized grid directions it is possible to synthesize

quite generally anisotropic covariance operators within a

3-dimensional variational analysis. Each is locally characterized

by a symmetric ``aspect tensor'' defining the scale of coherence

in different directions. We speculate on methods by which the

information available in an ensemble may be incorporated into the

process of estimating the aspect tensor in an objective and

dynamically adaptive way.

Dynamics and statistics of forecast errors in a quasi-geostrophic model.

Results from QG model. Time scale
for collapse

of specified initial ensemble to "the attractor",
characteristics

of perturbations on the attractor.
Instantaneous statistics of

analysis and forecast errors for 3DVAR,
in particular influence of

past dynamics through projection of errors
onto leading Lyapnov

subspace. Singular vectors for approximate
"analysis error covariance

norm" and their differences from energy
SV's.

Estimating analysis uncertainty using the NCEP global ensemble

Estimated errors in short-range forecasts and analyses will be compared
with variance and covariance information derived from the NCEP global ensemble
forecast system at short lead time. Targeted data collected in the FASTEX,
NORPEX, CALJET and Winter Storm Reconnaissance 1999 field programs will
be used to evaluate the quality of numerical guidance products.

Spread/Analysis error relationships in a simple model

The relationship between analysis error and analysis spread is examined

in the simple 40 variable model described by Lorenz and Emanuel (Feb
1

1998, JAS). Two types of analysis ensembles are used: an optimal
scheme

based on the ensemble Kalman filter, and a suboptimal scheme with

flow-independent forecast error covariances (similar to 3DVAR).
The

ability of these schemes to estimate the quality of the analysis (as

measured by ensemble spread) is investigated as a function of the number

of observations, the ensemble size, and the size of model and

observation error.

Plans for using fully flow-dependent background error covariance information in the NCEP regional 4DVAR data assimilation system

A fully cycled low rank 4DVAR system will be presented from the theoretical
point of view.

The issue of preconditioning in 4DVAR, and its relation to the low
rank approximation, will

be addressed in more detail. A "forecast ensemble" formulation of this
4DVAR algorithm,

especially attractive because it does not use the adjoint model, will
be shown as well.

Finally, some basic assumptions involved in the low rank 4DVAR
system will be discussed.

Beyond two overhead projectors, the meeting room will also be equipped
with a computer supporting electronic presentations (Corel, Microsoft,
etc).