Technical notes on using a virtual Flexpart WRF environment
Overview
The Flexpart WRF distribution created through the process outlined in Flexpart WRF Install provides a static collection of a fully-compiled and installed Flexpart WRF environment (at a later date, it may contain more). Through its use (typically through copying and/or linking) we may be confident that we are using a fully-tested, operational environment for a wide variety of applications. Users or developers who want different distributions can simply install them in a similar manner, storing them in different locations. With this approach, we can have a collection of stable distributions of different versions, and experimental distributions.
Within the Flexpart WRF distribution is the custom flxwrfusersetup.py program, which is used to create a new Flexpart WRF environment in a user directory. This environment consists of a directory named by the user, with a large number of links, and in some cases, copies, to files in the static distribution. This local environment may then be modified and used for custom Flexpart WRF activities without affecting the primary distribution. In this way, a user can create one or more separate Flexpart WRF environments for independent simulations.
Use of the flxwrfusersetup.py program is relatively simple - the user invokes it with a single argument of the name of the directory (and an optional path to it) that should be used for the simulation. The directory name provided must be writable by the user, of course, and if it already exists then it will not proceed. This policy is implemented to ensure that users don’t accidentally overwrite a simulation directory that might have a lot of important data in it.
The following demonstration goes through the process of creating a single virtual Flexpart WRF environment and running all of the preprocessing and simulation stages within it. This is the way a typical Flexpart WRF user would interact with the model components. Although not important to understand at this point, EHRATM processes will create a single virtual WPS/WRF/FlexpartWRF environment for each preprocessing and simulation stage. The reason for this is to support flexibility in a structure of loosely-coupled components. In this way, users are able to create custom workflows and not be restricted as one might in a single virtual environment. Additionally, users who need to do do some low-level debugging or experimentation can do so in a very restricted environment without the complexities of a full model run.
Creation of a Flexpart WRF run environment for custom simulation
In these examples, I’m assuming a Flexpart WRF distribution has been created (using guidance provided in Technical notes on using a virtual WPS/WRF environment) in /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFV3.3/ and that I’m starting with /dvlscratch/ATM/morton/ as my cwd. First, I run flxwrfusersetup.py with no argument, getting the usage message, then I run it with the MyFirstTest argument, indicating that I want a local Flexpart WRF distribution to be created in that directory:
$ pwd
/dvlscratch/ATM/morton
$ FlexWRFDistributions/FlexWRFv3.3/flxwrfusersetup.py
Usage: ./flxwrfusersetup.py <DomainName>
or
/path/to/flxwrfusersetup.py /path/to/<DomainName>
* Creates dir <DomainName> in specified directory
$ FlexWRFDistributions/FlexWRFv3.3/flxwrfusersetup.py MyFirstFlexWRFTest
FLXWRF src distribution directory: /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code
Creating domain: MyFirstFlexWRFTest
Setting up FLEXWRF run dir: MyFirstFlexWRFTest/flexwrf_rundir
Once complete, I can take a quick look in this new directory
$ tree -L 2 MyFirstFlexWRFTest
MyFirstFlexWRFTest
└── flexwrf_rundir
├── AVAILABLE2.sample
├── AVAILABLE3.sample
├── flexwrf.input.backward1.sample
├── flexwrf.input.backward2.sample
├── flexwrf.input.forward1.sample
├── flexwrf.input.forward2.sample
├── flexwrf_mpi -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/flexwrf_mpi
├── flexwrf_omp -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/flexwrf_omp
├── flexwrf_serial -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/flexwrf_serial
├── IGBP_int1.dat -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/IGBP_int1.dat
├── OH_7lev_agl.dat -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/OH_7lev_agl.dat
├── README_FIRST.txt.sample
├── surfdata.t -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/surfdata.t
└── surfdepo.t -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/surfdepo.t
1 directory, 14 files
The Flexpart WRF directory is fully functional, ready for use to set up and run a simulation. Note that flexwrf.input* are copies of files, not linked to the original execution. In the following, I demonstrate how this new run directory can be used to manually set up and execute a simple Flexpart WRF simulation using existing wrfout* files. My assumption is that the reader is experienced in running Flexpart WRF simulations, so I don’t bother to explain the steps in any detail.
Note, however, that the demonstration uses NetCDF Flexpart WRF output format, allowing for easier visualisation for the purposes of the demo. This is not aligned with the general EHRATM use where we will generally use Flexpart binary formats, but this shouldn’t detract from the purpose of the following demo, which is to simply go through the mechanics of using this environment to run Flexpart WRF.
Demonstration of the newly-created Flexpart WRF run environment
For this example, we will arbitrarily choose to run a simple model over the Anchorage region of Alaska for 18 hours, using the wrfout that was produced in the met data from the WPS/WRF example of Technical notes on using a virtual WPS/WRF environment. These wrfout files are
$ ls -l wrfout*
-rw-r--r--. 1 morton consult 19332628 Sep 21 01:52 wrfout_d01_2021-02-21_00:00:00
-rw-r--r--. 1 morton consult 19332628 Sep 21 01:52 wrfout_d01_2021-02-21_01:00:00
-rw-r--r--. 1 morton consult 19332628 Sep 21 01:53 wrfout_d01_2021-02-21_02:00:00
.
.
.
-rw-r--r--. 1 morton consult 19332628 Sep 21 02:00 wrfout_d01_2021-02-21_17:00:00
-rw-r--r--. 1 morton consult 19332628 Sep 21 02:01 wrfout_d01_2021-02-21_18:00:00
The steps we’ll take (in the newly created run directory)
Make a subdirectory metdata and inside that make links to the wrfout files
Create an AVAILABLE file
Edit the namelist file, flxp_input.txt
Run it, and quickly view the output
All of this is performed on dls011 in /dvlscratch/ATM/morton/MyFirstFlexWRFTest/. We assume that the wrfout files are available in /dvlscratch/ATM/morton/MyFirstTest/WRF/
$ cd flexwrf_rundir
and create the links to the wrfout files
$ mkdir metdata; cd metdata
$ ln -sf /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout* .
$ ls -l
lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_00:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_00:00:00
lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_01:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_01:00:00
.
.
.
lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_17:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_17:00:00
lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_18:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_18:00:00
$ cd ..
Then, create the AVAILABLE file
XXXXXX EMPTY LINES XXXXXXXXX
XXXXXX EMPTY LINES XXXXXXXX
YYYYMMDD HHMMSS name of the file(up to 80 characters)
20210221 000000 'wrfout_d01_2021-02-21_00:00:00' ' '
20210221 010000 'wrfout_d01_2021-02-21_01:00:00' ' '
20210221 020000 'wrfout_d01_2021-02-21_02:00:00' ' '
20210221 030000 'wrfout_d01_2021-02-21_03:00:00' ' '
20210221 040000 'wrfout_d01_2021-02-21_04:00:00' ' '
20210221 050000 'wrfout_d01_2021-02-21_05:00:00' ' '
20210221 060000 'wrfout_d01_2021-02-21_06:00:00' ' '
20210221 070000 'wrfout_d01_2021-02-21_07:00:00' ' '
20210221 080000 'wrfout_d01_2021-02-21_08:00:00' ' '
20210221 090000 'wrfout_d01_2021-02-21_09:00:00' ' '
20210221 100000 'wrfout_d01_2021-02-21_10:00:00' ' '
20210221 110000 'wrfout_d01_2021-02-21_11:00:00' ' '
20210221 120000 'wrfout_d01_2021-02-21_12:00:00' ' '
20210221 130000 'wrfout_d01_2021-02-21_13:00:00' ' '
20210221 140000 'wrfout_d01_2021-02-21_14:00:00' ' '
20210221 150000 'wrfout_d01_2021-02-21_15:00:00' ' '
20210221 160000 'wrfout_d01_2021-02-21_16:00:00' ' '
20210221 170000 'wrfout_d01_2021-02-21_17:00:00' ' '
20210221 180000 'wrfout_d01_2021-02-21_18:00:00' ' '
and edit flxp_input.txt
=====================FORMER PATHNAMES FILE===================
./
./metdata/
./AVAILABLE
=============================================================
=====================FORMER COMMAND FILE=====================
1 LDIRECT: 1 for forward simulation, -1 for backward simulation
20210221 000000 YYYYMMDD HHMISS beginning date of simulation
20210221 180000 YYYYMMDD HHMISS ending date of simulation
3600 SSSSS (int) output every SSSSS seconds
3600 SSSSS (int) time average of output (in SSSSS seconds)
180 SSSSS (int) sampling rate of output (in SSSSS seconds)
999999999 SSSSS (int) time constant for particle splitting (in seconds)
180 SSSSS (int) synchronisation interval of flexpart (in seconds)
10. CTL (real) factor by which time step must be smaller than tl
10 IFINE (int) decrease of time step for vertical motion by factor ifine
1 IOUT 1 concentration, 2 mixing ratio, 3 both, 4 plume traject, 5=1+4
0 IPOUT particle dump: 0 no, 1 every output interval, 2 only at end
0 LSUBGRID subgrid terrain effect parameterization: 1 yes, 0 no
0 LCONVECTION convection: 3 yes, 0 no
3600. DT_CONV (real) time interval to call convection, seconds
1 LAGESPECTRA age spectra: 1 yes, 0 no
0 IPIN continue simulation with dumped particle data: 1 yes, 0 no
0 IFLUX calculate fluxes: 1 yes, 0 no
0 IOUTPUTFOREACHREL CREATE AN OUPUT FILE FOR EACH RELEASE LOCATION: 1 YES, 0 NO
0 MDOMAINFILL domain-filling trajectory option: 1 yes, 0 no, 2 strat. o3 tracer
1 IND_SOURCE 1=mass unit , 2=mass mixing ratio unit
2 IND_RECEPTOR 1=mass unit , 2=mass mixing ratio unit
0 NESTED_OUTPUT shall nested output be used? 1 yes, 0 no
0 LINIT_COND INITIAL COND. FOR BW RUNS: 0=NO,1=MASS UNIT,2=MASS MIXING RATIO UNIT
1 TURB_OPTION 0=no turbulence; 1=diagnosed as in flexpart_ecmwf; 2 and 3=from tke.
0 CBL SCHEME 0=no, 1=yes. works if TURB_OPTION=1
1 SFC_OPTION 0=default computation of u*, hflux, pblh, 1=from wrf
0 WIND_OPTION 0=snapshot winds, 1=mean winds,2=snapshot eta-dot,-1=w based on divergence
0 TIME_OPTION 1=correction of time validity for time-average wind, 0=no need
1 OUTGRID_COORD 0=wrf grid(meters), 1=regular lat/lon grid
1 RELEASE_COORD 0=wrf grid(meters), 1=regular lat/lon grid
2 IOUTTYPE 0=default binary, 1=ascii (for particle dump only),2=netcdf
1000 NCTIMEREC (int) Time frames per output file, only used for netcdf
0 VERBOSE VERBOSE MODE,0=minimum, 100=maximum
=====================FORMER AGECLASESS FILE==================
2 NAGECLASS number of age classes
7200 SSSSSS (int) age class in SSSSS seconds
999999 SSSSSS (int) age class in SSSSS seconds
=====================FORMER OUTGRID FILE=====================
-157.00 OUTLONLEFT geograhical longitude of lower left corner of output grid
58.0 OUTLATLOWER geographical latitude of lower left corner of output grid
24 NUMXGRID number of grid points in x direction (= # of cells )
28 NUMYGRID number of grid points in y direction (= # of cells )
1 OUTGRIDDEF outgrid defined 0=using grid distance, 1=upperright corner coordinate
-141.0 DXOUTLON grid distance in x direction or upper right corner of output grid
63.0 DYOUTLON grid distance in y direction or upper right corner of output grid
3 NUMZGRID number of vertical levels
100.0 LEVEL height of level (upper boundary)
1000.0 LEVEL height of level (upper boundary)
20000.0 LEVEL height of level (upper boundary)
=====================FORMER RECEPTOR FILE====================
0 NUMRECEPTOR number of receptors
=====================FORMER SPECIES FILE=====================
1 NUMTABLE number of variable properties. The following lines are fixed format
XXXX|NAME |decaytime |wetscava |wetsb|drydif|dryhenry|drya|partrho |parmean|partsig|dryvelo|weight |
AIRTRACER -999.9 -9.9E-09 -9.9 -9.9E09 -9.99 29.00
=====================FORMER RELEEASES FILE===================
1 NSPEC total number of species emitted
0 EMITVAR 1 for emission variation
1 LINK index of species in file SPECIES
1 NUMPOINT number of releases
20210221 000000 ID1, IT1 beginning date and time of release
20210221 120000 ID2, IT2 ending date and time of release
-149.9000 XPOINT1 (real) longitude [deg] of lower left corner
61.2000 YPOINT1 (real) latitude [deg] of lower left corner
-149.9000 XPOINT2 (real) longitude [deg] of upper right corner
61.2000 YPOINT2 (real) latitude [DEG] of upper right corner
1 KINDZ (int) 1 for m above ground, 2 for m above sea level, 3 pressure
100.000 ZPOINT1 (real) lower z-level
1000.000 ZPOINT2 (real) upper z-level
500000 NPART (int) total number of particles to be released
.1000E+04 XMASS (real) total mass emitted
box1 NAME OF RELEASE LOCATION
and run it
$ ./flexwrf_serial flxp_input.txt
the input file used is flxp_input.txt
Pre-generating random numbers
RANLUX LUXURY LEVEL SET BY RLUXGO : 3 P= 223
RANLUX INITIALIZED BY RLUXGO FROM SEEDS 254 0 0
Calling set_mt19937
Calling new (mts(0))
Calling init (mts(0),iseed)
Generating additional streams
.
.
.
61200 SECONDS SIMULATED: 499999 PARTICLES: Uncertainty: 0.000 0.000 0.000
time needed 216.584000
64800 SECONDS SIMULATED: 499999 PARTICLES: Uncertainty: 0.000 0.000 0.000
time needed 191.740005
timemanager_serial: End of simulation reached, closing output file
CONGRATULATIONS: YOU HAVE SUCCESSFULLY COMPLETED A FLEXPART_WRF MODEL RUN!
note the two NetCDF output files
$ ls -l *.nc
-rw-r--r--. 1 morton consult 49979 Mar 10 04:10 flxout_d01_20210221_010000.nc
-rw-r--r--. 1 morton consult 65751 Mar 10 03:30 header_d01.nc
and use ncview to sample the concentration from the newly-created flxout_d01_20210221_010000.nc. Note that I have placed a copy of the ncview binary in the repo, in misc/utilities/ncview so for convenience I assign the path to an environment variable
$ export NCVIEW=/dvlscratch/ATM/morton/git/high-res-atm/misc/utilities/ncview
$ $NCVIEW flxout_d01_20210221_010000.nc
We can also review the topography from header_d01.nc output file
$ $NCVIEW header_d01.nc
Summary
For someone used to running Flexpart WRF, this was probably boring, but for the less experienced user, hopefully it serves as a little bit of guidance, and demonstrates how we can quickly create and execute a custom Flexpart WRF simulation driven by already-existing wrfout data (whose production was presented in Technical notes on using a virtual WPS/WRF environment. ).
After the demo described above, we could create more local run directories for a variety of simulations. Perhaps we just want to try the same simulation with different parameterisations. One approach would be to create different run directories for each case and, in each, go through the above steps with slightly different versions of flxp_input.txt, each using different parameters.