EHRATM Overview

The EHRATM project had its origins in the initial CTBTO-sponsored exploratory work of Gerald Klinkl and Michael Schoeppner, which delivered a fully-functional WRF/FlexpartWRF pipeline.  Based on user experiences with this software, design goals for an enhanced version were defined, and implementation proceeded as an entirely new product.  EHRATM has been implemented primarily as 1) a software stack that provides user-level abstraction to access the low-level operations associated with the complexities of running individual NWP and ATM components; 2) a distribution installation system that automates the retrieval, configuration, compilation and installation of the WRF and FlexpartWRF distributions in a predictable “same way every time” manner; and 3) a domain validation and plotting tool that helps a user create an acceptable domain as one of the inputs for a workflow.  Detailed user and technical information can be found on all of these starting at the top of the Sphinx documentation HTML tree, <BASE_URL>/docs/index.html.

The layered software stack that encompasses the core of EHRATM is built according to well-accepted software engineering principles to push tedious implementation details low in a call-tree, providing a high level of abstraction at the top so that developers can build applications with efficiency, with some confidence that the underlying software is stable and well-tested (consider the relative ease with which we can use cartopy to plot complex data over various projections by making just a handful of high-level function calls).   Starting from the bottom of the stack, as each layer is implemented, it is built with a standard, non-changing application program interface (API) so that programs which call these functions from above will have reasonable confidence that the API will always be the same, and that any changes made in the underlying functions will be fully tested and provide the same behavior to the calling functions.

At the bottom of the EHRATM software stack, the nwpservice layer provides low-level, direct access to the WPS, WRF and FlexpartWRF program components so that calling functions need only adhere to the API defined for nwpservice components and not have to worry about many of the tedious details (typically via a shell command line)  associated with setting up, configuring for a specific set of data, executing, and staging output data.  Each of the components in this layer is a standalone Python class, which means that they can be invoked in isolation as long as expected input is provided.  This not only allows for “plug ‘n play” flexibility from the perspective of upper layers, but greatly facilitates rigorous testing of each component, and makes it relatively straightforward to experiment with (or debug) the coding within a component, testing it afterwards to ensure that it continues to provide the interface and behavior that upper layers are counting on.  With this base layer for accessing the NWP and ATM model components it is possible to build a variety of layers above it for many different purposes.  Additionally, one can consider adding more components at a later date - data assimilation, for example.  One could build a fully functional workflow simply by chaining together the standalone components of the nwpservice layer and, in fact, we have tested it in this capacity.

In our case, since we are aiming for a rather targeted application, we built the ehratm layer to serve as an interface to an even higher layer, abstracting many of the low-level details involved in calling the nwpservice components.  The ehratm layer, then, provides a higher level of abstraction to application programs.  Although it could be built in many different ways, we chose to continue the paradigm of loosely-coupled components.  So, at this layer we continue to work with components that can run in isolation for safe and efficient development and testing, as well as flexible use by higher-level applications.

Note that so far, there has been no workflow implied in anything that we built.  It is simply a collection of plug ‘n play components that can be arranged in any way that a programmer would like to create a variety of NWP and ATM applications.  This is where our targeted top-level ehratmworkflow layer comes in.   It relies on the fact that the robust components it needs are available in the underlying ehratm layer and provides the “plumbing” needed to arrange the ehratm components as dictated by a user-supplied workflow namelist (wfnamelist).  The ehratmworklow layer contains a large collection of wfnamelists which can be tested all at once in batch mode to ensure that they all work.  As each one is invoked, it is providing additional testing of the underlying ehratm and nwpservice layers.  The following provides a simple illustration that all of this abstraction has ultimately provided to the user, who simply invokes ehratmwf.py, supplying workflow requirements in a namelist.

Another fundamental subsystem is the nwpinstall component. This is a set of “cookbook” programs that will install WPS, WRF and Flexpart WRF in a standard way, the same way each time. With this programmatic installation, those who install these programs are assured that the configuration, build and installs are done exactly according to specifications - they don’t need to worry about forgetting a step, using a wrong compiler option or environment variable, or many other things that happen when people are installing complex software. The nwpinstall component creates the distributions using - as much as possible - the methods recommended by the model maintainers, but also adds in minor code changes where necessary (for example, correcting Flexpart WRF so that it outputs release location in latlon rather than map coordinates). Finally, the nwpinstall component creates “helper” scripts that the nwpservice components can count on using.

Finally, the separate domaincheck project consists of classes and an executable that, given a domain definition file, makes sure it can be processed error-free, check for various WRF constraints that might cause problems down the line, and optionally plots the nests in the native projection (polar, lambert, mercator) as well as latlon, giving the user the ability to “proof” their domain before trying it in a workflow. It also outputs a suggested extent boundary for meteorology files (ECMWF MARS).