================================================================= Installation ================================================================= This document provides a general overview and demonstration of the installation process for the various EHRATM components. Deeper technical details are available in the **nwpinstall** documentation. Here, we focus primarily on the installation of WPS/WRF and Flexpart WRF in a way that will be compatible with the EHRATM system components. Then, we cover the configuration of a couple of "globals" files within the repository source code. The examples here show the distributions being installed in ``/dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/``, and the ``develop-freeze-v1.0`` tag in the ``high-res-atm`` gitlab repository, has been configured to use these distributions. Overview ========= The *nwp_install* component is a scripted environment for a reproducible installation of specified versions of WPS/WRF and Flexpart WRF. The scripts will install a specified version the same way, every time, providing a standard installation that users can count on. Originally built in support of the EHRATM project, this component stands indepdendent and can be used for a variety of standalone WPS/WRF and Flexpart WRF activities. The overall vision of this component, as illustrated below, is that multiple versions of WPS/WRF and Flexpart WRF may be installed in a static environment, and users have the flexibility to use their version of choice, all within their own user space, and they may work with multiple instances simultaneously. .. |distro_vision| image:: nwp_install_distribution_vision.png :scale: 60 % |distro_vision| The distributions created through these install processes provide a static collection of fully-compiled and installed WPS, WRF and Flexpart WRF environments (at a later date, it may contain more). Through their use (typically through copying and/or linking) we may be confident that we are using fully-tested, operational environments for a wide variety of applications. Users or developers who want different distributions can simply install them in a similar manner, storing them in different locations. With this approach, we can have a collection of stable distributions of different versions, and experimental distributions. Again, more detail is provided in the **nwpinstall** documentation. ---- The following demonstrations were performed in ``/dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/`` and these are the distributions utilized by the develop-freeze-v1.0- tag of the high-res-atm repository. Installation of WPS/WRF ======================== We start by creating the directory for WRF distributions .. code-block:: bash $ mkdir WRFDistributions GEOG_DATA --------- The WRF simulations are initialised with a lot of static, gridded data - things like topography, land use, leaf area index, etc. This data is available in various resolutions from coarse to fine, and, when fully installed, takes up 30-50+ GBytes of disc space. In order to maintain a versatile WRF modeling environment, it makes sense to make all of this available on the local computing system. It doesn’t make sense to try to store it within the nwpinstall component, nor does it make sense to download and install it for each new WRF distribution. For the most part, this data is identical from one WRF version to another, though sometimes newer versions may use some newly-available datasets. So, it’s generally necessary to install this geog data one time, and then specifying its location when installing a new WRF distribution,  The geog data is available at the WRF site, at https://www2.mmm.ucar.edu/wrf/users/download/get_sources_wps_geog.html. It can be confusing to understand what needs to be downloaded, so the following provides a working example of one-time installation of the datasets needed (so far) for the WRF distributions installed in this component .. code-block:: bash $ cd WRFDistributions $ mkdir geog $ wget http://www2.mmm.ucar.edu/wrf/src/wps_files/geog_complete.tar.gz $ tar xzvf geog_complete.tar.gz --strip-components=1 -C geog $ du -csk geog 51336536        geog 51336536        total Once you’re sure that installation was correct (you might wait until you’ve had a successful test of a WRF distribution) it’s safe to remove the downloaded tar file. The newer WPS/WRF versions need a couple of extra datasets, installed as follows .. code-block:: bash $ wget https://www2.mmm.ucar.edu/wrf/src/wps_files/albedo_modis.tar.bz2 $ wget https://www2.mmm.ucar.edu/wrf/src/wps_files/maxsnowalb_modis.tar.bz2 $ cd geog $ tar xjvf ../albedo_modis.tar.bz2 $ tar xjvf ../maxsnowalb_modis.tar.bz2 ---- WPS/WRF ------- Within the repository ``nwpsintall/install-wrf`` directory are a set of subdirectories for each of the supported distributions. A new distribution can be made by creating a similar directory (sometimes just copying another one will suffice) and editing as needed. In this case, we will use ``nwpsinstall/install-wrf/wrfv4.3_2022-04-25/``. Before doing anything, we need to set some environment variables, sourcing ``setup_install_env.sh``. The settings are standard for CTBTO-like systems and should not need any modifying by you .. code-block:: bash #!/bin/bash # Set of system-specific (mostly) environment variables used by the # installation routines to compile, setup and test a WRF distribution ################################################################ # These have been tested in the CTBTO-like CentOS 7 VM # and are the same on devlan ################################################################ export MPIF90=/usr/lib64/openmpi/bin/mpif90 export MPICC=/usr/lib64/openmpi/bin/mpicc # This is needed for running post-install test cases export MPIRUN=/usr/lib64/openmpi/bin/mpirun # Note that the netcdf.mod file in CentOS 7 seems not to be in same place # as typical netcdf includes export NETCDF=/usr export NETCDF_MOD_INC=/usr/lib64/gfortran/modules export NETCDF_classic=1 export JASPERLIB=/usr/lib64 export JASPERINC=/usr/include # Needed for some of the WPS utilities (e.g. plotgrids.exe) export NCARG_ROOT=/usr/lib64/ncarg So, before running the installation script, the environment can be setup as follows, within the desired distribution directory. Note that the system Python 2.7.5 is sufficient for all this. No need to worry about Conda environments yet .. code-block:: bash $ . setup_install_env.sh Once the geog files and the system environment variables have been set up correctly, it should be possible to run the installation program to build and install the distribution in (for example) ``WRFDistributions/`` as follows .. code-block:: bash $ ./wrfinstall.py --baseinstalldir /dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/WRFDistributions/WRFV4.3 --geogdatadir /dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/WRFDistributions/geog 2023-11-25 18:59:43,761 - INFO - wrfinstall.py:main:56 --> Starting install... . . . 2023-11-25 19:11:57,247 - INFO - wrfinstall.py:nam_nodak_test:753 --> Apparent successful completion of test... This script retrieves the source codes, configures, builds, and performs simple tests of functionality for WRF, then WPS. It doesn’t test for correct answers, but just tests that the basic components run. .. code-block:: bash $ ls -lF /dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/WRFDistributions/WRFV4.3 total 32 lrwxrwxrwx. 1 morton consult 71 Nov 25 18:59 GEOG_DATA -> /dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/WRFDistributions/geog/ drwxr-xr-x. 8 morton consult 4096 Nov 25 19:10 WPS/ drwxr-xr-x. 22 morton consult 4096 Nov 25 19:00 WRF/ -rwxr-xr-x. 1 morton consult 17957 Nov 25 19:10 wrfusersetup.py* The entire installation process on **dls011** took approximately 12 minutes. Within the ``WRFV4.3`` directory is a fully-functional WPS/WRF environment. The ``wrfusersetup.py`` program was created in the installation process and serves as a tool within the EHRATM system to create standalone WPS/WRF run directories. You could use this yourself at this point - more details are in the **nwpinstall** documentation. ---- Installation of Flexpart WRF ============================= In a similar way we can install Flexpart WRF in ``/dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/`` .. code-block:: bash $ mkdir FlexWRFDistributions So, in the ``nwpinstall/install-flxwrf/flxwrfv3.3_2023-11-25/`` directory Again, just using system Python 2.7.5 default .. code-block:: bash $ . setup_install_env.sh $ ./flxwrfinstall.py --baseinstalldir /dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/FlexWRFDistributions/flexwrfv3.3 2023-11-25 19:48:27,148 - INFO - flxwrfinstall.py:main:75 --> Starting install... . . . 2023-11-25 19:51:35,436 - INFO - flxwrfinstall.py:sicily_2hr_test:816 --> All expected output files are present 2023-11-25 19:51:35,436 - INFO - flxwrfinstall.py:sicily_2hr_test:818 --> Apparent successful completion of sicily_2hr_test... and upon completion we can view the new distribution, noting that it has also install **flexpart2ctbt** in the ``srm/`` directory, and has created the ``flxwrfusersetup.py`` program that allows us to create standalone run environments, like ``wrfusersetup.py``. .. code-block:: bash $ ls -lF /dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0/FlexWRFDistributions/flexwrfv3.3 total 44 drwxr-xr-x. 4 morton consult 28672 Nov 25 19:49 flexwrf_code/ -rwxr-xr-x. 1 morton consult 6003 Nov 25 19:49 flxwrfusersetup.py* drwxr-xr-x. 3 morton consult 4096 Nov 25 19:48 srm/ ---- Configuration of EHRATM Defaults ================================ At this point, we have a complete system for running WPS/WRF and Flexpart WRF, and it's useful to consider that this is all the knowledgeable user really needs. The EHRATM system is merely a set of tools to allow us to use it all more efficiently for a wide variety of scenarios. The **high-res-atm** repository, when accessed through the tag ``develop-freeze-v1.0-`` should be set up to default values that will allow a user to immediately begin operations. However, several of the values should be discussed, particularly for those who might want to install things in different locations. There are three files within the **high-res-atm** system that define some global (or semi-global) values. I would have liked to have had time to make this cleaner, perhaps having these files up higher in the directory tree, or having a simple text file for setting the files, but this is how it is for now. In ``packages/nwpservice/src/nwpservice/defaults.py`` we have a single definition for setting a log level. Most users will have no interest in this. ``packages/ehratm/src/ehratm/syspaths.py`` is the most important one. What's in there now should work for these demonstrations, but I am the only one who has tested any of this so far. You should look in the **devlan** section of the file, and note the following definitions .. code-block:: bash REPOSITORY_ROOTDIR = '/dvlscratch/ATM/morton/git/high-res-atm' DISTRIBUTIONS_ROOTDIR = '/dvlscratch/ATM/morton/EHRATM-develop-freeze-v1.0' These are the locations of the gitlab repository (set at the ``develop-freeze-v1.0-`` tag) and the WPS/WRF and Flexpart WRF distributions discussed above. Both of these are read-only filesystems, so it should be safe for a user to stick with these. Obviously, new installations will require a change in here. In the same ``syspaths.py`` we have .. code-block:: bash self._WORKING_SCRATCH_ROOTDIR = '/tmp' self._WORKFLOW_ROOTDIR = '/tmp' ``WORKFLOW_ROOTDIR`` is where all of the simulations take place, and where all of the input/output files for a workflow are located, so it tends to take up a lot of space. The ``/tmp`` directory is typically not a good place for this, but it is known to be writable by everybody, so it is the default. As demonstrated in the :doc:`New User HowTo ` document, there is an entry in the wfnamelist, ``workflow_rootdir`` that allows one to override the default to something within their own user directory. ``WORKING_SCRATCH_ROOTDIR`` is little-used, so the ``/tmp`` default location should not be a problem. But, power users who create their own installations may want to go ahead and set it to something in their own filesystems. Finally, in ``ehratm/src/ehratm/defaults.py`` we have a set of global constants, many of which can be overridden elsewhere. The ones that might be most important include ``WRF_SPINUP_HOURS``, which can be overriden in the wfnamelist, and ``MAX_MPI_TASKS``. This is currently set to 16 in order to keep a user from blindly using too many cores on a devlan system. There is no way to override this, so someone interested in more MPI tasks would need to clone their own copy of the repository, set it to the ``develop-freeze-v1.0-`` tag, and then modify this value. ---- Summary ======== That's it. Although a lot was said above, any new installation is a matter of * Cloning the repository and setting to the ``develop-freeze-v1.0-`` tag * Installing the distributions as described above * Configuring several of the EHRATM default values as described above. After that, one could go to the :doc:`New User HowTo ` and try some workflows. .. toctree::