Technical notes on the WPS/WRF installation program

Overview

This section addresses some of the underlying details of the WPS/WRF installation component. It is assumed to be of interest to developers and those with a need or interest in going beyond “cookbook” usage.

The core of the installation process is the wrfinstall.py found in each distribution directory. For example, nwp_install/install-wrf/wrfv4.3_2022-04-25/wrfinstall.py. Although the program is almost identical in each of the distributions, there are typically some minor differences - that’s why each distribution has its own copy. Compile flags may be slightly different if one distribution is intended to have features that differ from another. Particularly in an experimental distribution, certain source files may come from non-standard places, or require some text modifications before being built.

In general, the wrfinstall.py will be executed after ensuring that the environment is set up (see Setup of WPS/WRF installation requirements), specifying the name of the base installation directory and the location of the geog files

$ ./wrfinstall.py --baseinstalldir ~/WRFV4.3-Distribution --geogdatadir /home/ctbtuser/tempgeog

With a successful distribution install in place, users will be able to create any number of independent virtual WPS/WRF environments by using the wrfusersetup.py script within the distribution. For example, one can create a virtual WPS/WRF environment named MyNewCase

$ /home/ctbtuser/WRFV4.3-Distribution/wrfusersetup.py MyNewCase

and this will result in a local directory, MyNewCase consisting of the same WPS and WRF directories one would expect in a typical WPS/WRF runtime environment. Additionally, a link to the geog data is created. This directory may then be modified to create a custom WRF simulation.

$ tree -L 1 -F MyNewCase
MyNewCase
├── GEOG_DATA -> /home/ctbtuser/WRFV4.3-Distribution/GEOG_DATA/
├── WPS/
└── WRF/

3 directories, 0 files

Most of the contents of these directories are symbolic links into the installed distribution, though certain files, like namelists, are copies, allowing users to edit for their particular case. Usage of this environment is described in more detail in Technical notes on using a virtual WPS/WRF environment.

A brief walkthrough of wrfinstall.py

The program wrfinstall.py, found in each distribution install directory, is customised for a specific distribution to be installed. When a new distribution is to be incorporated into this nwp_install environment, this is the file that needs to be modified for the specifics of the distribution, so this section tries to highlight the areas of importance.

Just before the main() function is the first opportunity to customise the program

WRF_GIT = "https://github.com/wrf-model/WRF"
WRF_GIT_BRANCH = "release-v4.3"

WPS_GIT = "https://github.com/wrf-model/WPS"
WPS_GIT_BRANCH = "release-v4.3"

DEFAULT_TESTRUN_ROOTDIR = '/tmp'

We specify the location and branch for the WPS and WRF repositories. Additionally, since a successful installation will conclude with a runtime test, we specify the root directory where temporary test run directories will be created.

main()

In short, the main() function checks a lot of environment variables and expected filesystems, calls wrf_install() to install WRF (some of the libs in the WRF installation are prerequisites for compiling WPS), calls wps_install() to install WPS, creates a wrfusersetup.py script for this new installation, and then calls run_tests() to verify that the model preprocessing and simulation will run. It does not try to verify results, but simply reports whether execution of the pieces was successful from an operating system perspective.

I don’t see anything in this function that people would really want to customise. However, people who dig into this should be sure that they continue to use the same directory name prefixes (e.g. GEOG_DATA, test_case_metdata, etc.). I think that if you want to create a new distribution your best bet is to simply copy one of the existing ones and then modify carefully.

wrf_install()

We start by looking at the --forceinstall flag to the script. If this is False (the default), then if expected executables are already present, no installation is performed. This is useful in testing/debugging the program where we might not want to wait for a full installation, or risk overwriting a working installation, each time we call the script. If it is set to True then any existing installation will be removed, and we will start from scratch.

Next, we retrieve the source code from the specified (environment variables) git repository. Currently a git clone is the only way to retrieve the source, but this is how NCAR has set things up these days. A different retrieval method would require code modification.

Then, we come to the distribution-specific code. Modifying this requires commands that programmatically edit some of the configuration files, and this can get tedious and detailed. It’s probably best to get someone knowledgeable in the WRF system to help with this. From one version of WRF to another, there are likely to be some minor modifications in the configuration files that require customised changes here.

Finally, the compile command is executed. If you’re doing this for the first time on a new version of WRF, it’s probably going to fail, but you can look through the logging statements and find out where the compilation was performed, and then try to make things work manually, and then adjust the configuration files as necessary. If compilation completed without error, we check that the expected files (see the expected_executables list defined within this function) exist, and if so, we consider this to be a success.

wps_install()

This behaves very much like wrf_install(). After checking the --forceinstall flag and expected files and directories, we retrieve the source from the specified git repository and branch and customise configuration files for our specific needs. As with wrf_install() I think it’s best to rely on someone who has a background in building and using WPS/WRF in order to understand what changes might need to be made.

After all of the configuration, we compile, and if there are no failures, we ensure that expected executables exist.

run_tests()

This is set up so that we can run a list of full WPS/WRF simulations, but currently there is only one test, nam_nodak_test(). NAM is the North American Mesoscale Model, and NODAK is North Dakota (where I was born!). This is just a simple test to ensure that we can take some archived test input data and configure and execute a WPS/WRF simulation, ultimately producing the expected WRF output files. If these files are present, it’s highly likely (but not definitive), that our installation is good.

If the tests generate an error, then logging output will allow us to find the run directory and try to debug the program manually. Again, this is where somebody experienced with WPS/WRF would be of great value. Someone without this experience might struggle a lot.

The wrfusersetup.py script

Within the main() function are some lines of code that set up the wrfusersetup.py script. To review, this is the script that a general user will run to create a fully functional virtual WPS/WRF environment in their local directory (see the Overview section). This is a relatively generic Python code that I have used for many years to allow many users access to a centrally-installed WRF distribution (many users install their own versions, which leads to high disc usage and often a lot of confusion).

Because this script is based on the creation of many symbolic links, and the copying of some files, into a new user directory, it is somewhat specific to the WPS/WRF version being used. One version might have a slightly different set of files used for the execution environment than another version. So, somebody that wants to develop a new distribution within this nwp_install environment will need to go through some of the details and modify where necessary. Again, an experienced WPS/WRF user would be valuable for this.

Summary

The wrfinstall.py script has been outlined above. Each WPS/WRF distribution supported by this nwp_install environment has its own install directory, containing a set of environment variable definitions, the wrfinstall.py script customised for the desired distribution, and a wrfusersetup.py script customised for the desired distribution. In general, once the distribution directory files have been set up, it should be possible to install the distribution in a specified, centralised directory, and repeated installations should all be identical, leaving little room for questions about whether certain configuration files were set up correctly.

Once the centralised distribution is set up, users may run the distributions’s wrfusersetup.py script to create one more more independent virtual WPS/WRF run environments within their own filesystems.