=========================================================== Technical notes on using a virtual Flexpart WRF environment =========================================================== Overview ======== The Flexpart WRF distribution created through the process outlined in :doc:`../index` provides a static collection of a fully-compiled and installed Flexpart WRF environment (at a later date, it may contain more). Through its use (typically through copying and/or linking) we may be confident that we are using a fully-tested, operational environment for a wide variety of applications. Users or developers who want different distributions can simply install them in a similar manner, storing them in different locations. With this approach, we can have a collection of stable distributions of different versions, and experimental distributions. Within the Flexpart WRF distribution is the custom *flxwrfusersetup.py* program, which is used to create a new Flexpart WRF environment in a user directory. This environment consists of a directory named by the user, with a large number of links, and in some cases, copies, to files in the static distribution. This local environment may then be modified and used for custom Flexpart WRF activities without affecting the primary distribution. In this way, a user can create one or more separate Flexpart WRF environments for independent simulations. Use of the *flxwrfusersetup.py* program is relatively simple - the user invokes it with a single argument of the name of the directory (and an optional path to it) that should be used for the simulation. The directory name provided must be writable by the user, of course, and if it already exists then it will not proceed. This policy is implemented to ensure that users don't accidentally overwrite a simulation directory that might have a lot of important data in it. The following demonstration goes through the process of creating a single virtual Flexpart WRF environment and running all of the preprocessing and simulation stages within it. This is the way a typical Flexpart WRF user would interact with the model components. Although not important to understand at this point, EHRATM processes will create a single virtual WPS/WRF/FlexpartWRF environment for each preprocessing and simulation stage. The reason for this is to support flexibility in a structure of loosely-coupled components. In this way, users are able to create custom workflows and not be restricted as one might in a single virtual environment. Additionally, users who need to do do some low-level debugging or experimentation can do so in a very restricted environment without the complexities of a full model run. ---- Creation of a Flexpart WRF run environment for custom simulation ================================================================= In these examples, I'm assuming a Flexpart WRF distribution has been created (using guidance provided in :doc:`../../WrfInstall/TechNotes/DemoVirtualUsage`) in */dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFV3.3/* and that I'm starting with */dvlscratch/ATM/morton/* as my cwd. First, I run *flxwrfusersetup.py* with no argument, getting the usage message, then I run it with the *MyFirstTest* argument, indicating that I want a local Flexpart WRF distribution to be created in that directory: .. code-block:: bash $ pwd /dvlscratch/ATM/morton $ FlexWRFDistributions/FlexWRFv3.3/flxwrfusersetup.py Usage: ./flxwrfusersetup.py or /path/to/flxwrfusersetup.py /path/to/ * Creates dir in specified directory $ FlexWRFDistributions/FlexWRFv3.3/flxwrfusersetup.py MyFirstFlexWRFTest FLXWRF src distribution directory: /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code Creating domain: MyFirstFlexWRFTest Setting up FLEXWRF run dir: MyFirstFlexWRFTest/flexwrf_rundir Once complete, I can take a quick look in this new directory .. code-block:: bash $ tree -L 2 MyFirstFlexWRFTest MyFirstFlexWRFTest └── flexwrf_rundir ├── AVAILABLE2.sample ├── AVAILABLE3.sample ├── flexwrf.input.backward1.sample ├── flexwrf.input.backward2.sample ├── flexwrf.input.forward1.sample ├── flexwrf.input.forward2.sample ├── flexwrf_mpi -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/flexwrf_mpi ├── flexwrf_omp -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/flexwrf_omp ├── flexwrf_serial -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/flexwrf_serial ├── IGBP_int1.dat -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/IGBP_int1.dat ├── OH_7lev_agl.dat -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/OH_7lev_agl.dat ├── README_FIRST.txt.sample ├── surfdata.t -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/surfdata.t └── surfdepo.t -> /dvlscratch/ATM/morton/FlexWRFDistributions/FlexWRFv3.3/flexwrf_code/data/surfdepo.t 1 directory, 14 files The Flexpart WRF directory is fully functional, ready for use to set up and run a simulation.  Note that *flexwrf.input\** are copies of files, not linked to the original execution.  In the following, I demonstrate how this new run directory can be used to manually set up and execute a simple Flexpart WRF simulation using existing *wrfout\** files.  My assumption is that the reader is experienced in running Flexpart WRF simulations, so I don’t bother to explain the steps in any detail.   Note, however, that the demonstration uses NetCDF Flexpart WRF output format, allowing for easier visualisation for the purposes of the demo. This is not aligned with the general EHRATM use where we will generally use Flexpart binary formats, but this shouldn't detract from the purpose of the following demo, which is to simply go through the mechanics of using this environment to run Flexpart WRF. ---- Demonstration of the newly-created Flexpart WRF run environment ================================================================ For this example, we will arbitrarily choose to run a simple model over the Anchorage region of Alaska for 18 hours, using the *wrfout* that was produced in the met data from the WPS/WRF example of :doc:`../../WrfInstall/TechNotes/DemoVirtualUsage`. These *wrfout* files are .. code-block:: bash $ ls -l wrfout* -rw-r--r--. 1 morton consult 19332628 Sep 21 01:52 wrfout_d01_2021-02-21_00:00:00 -rw-r--r--. 1 morton consult 19332628 Sep 21 01:52 wrfout_d01_2021-02-21_01:00:00 -rw-r--r--. 1 morton consult 19332628 Sep 21 01:53 wrfout_d01_2021-02-21_02:00:00 . . . -rw-r--r--. 1 morton consult 19332628 Sep 21 02:00 wrfout_d01_2021-02-21_17:00:00 -rw-r--r--. 1 morton consult 19332628 Sep 21 02:01 wrfout_d01_2021-02-21_18:00:00 The steps we’ll take (in the newly created run directory) * Make a subdirectory *metdata* and inside that make links to the *wrfout* files * Create an *AVAILABLE* file * Edit the namelist file, *flxp_input.txt* * Run it, and quickly view the output All of this is performed on *dls011* in */dvlscratch/ATM/morton/MyFirstFlexWRFTest/*. We assume that the *wrfout* files are available in */dvlscratch/ATM/morton/MyFirstTest/WRF/* .. code-block:: bash $ cd flexwrf_rundir and create the links to the *wrfout* files .. code-block:: bash $ mkdir metdata; cd metdata $ ln -sf /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout* . $ ls -l lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_00:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_00:00:00 lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_01:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_01:00:00 . . . lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_17:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_17:00:00 lrwxrwxrwx. 1 morton consult 69 Mar 9 19:58 wrfout_d01_2021-02-21_18:00:00 -> /dvlscratch/ATM/morton/MyFirstTest/WRF/wrfout_d01_2021-02-21_18:00:00 $ cd .. Then, create the *AVAILABLE* file .. code-block:: bash XXXXXX EMPTY LINES XXXXXXXXX XXXXXX EMPTY LINES XXXXXXXX YYYYMMDD HHMMSS name of the file(up to 80 characters) 20210221 000000 'wrfout_d01_2021-02-21_00:00:00' ' ' 20210221 010000 'wrfout_d01_2021-02-21_01:00:00' ' ' 20210221 020000 'wrfout_d01_2021-02-21_02:00:00' ' ' 20210221 030000 'wrfout_d01_2021-02-21_03:00:00' ' ' 20210221 040000 'wrfout_d01_2021-02-21_04:00:00' ' ' 20210221 050000 'wrfout_d01_2021-02-21_05:00:00' ' ' 20210221 060000 'wrfout_d01_2021-02-21_06:00:00' ' ' 20210221 070000 'wrfout_d01_2021-02-21_07:00:00' ' ' 20210221 080000 'wrfout_d01_2021-02-21_08:00:00' ' ' 20210221 090000 'wrfout_d01_2021-02-21_09:00:00' ' ' 20210221 100000 'wrfout_d01_2021-02-21_10:00:00' ' ' 20210221 110000 'wrfout_d01_2021-02-21_11:00:00' ' ' 20210221 120000 'wrfout_d01_2021-02-21_12:00:00' ' ' 20210221 130000 'wrfout_d01_2021-02-21_13:00:00' ' ' 20210221 140000 'wrfout_d01_2021-02-21_14:00:00' ' ' 20210221 150000 'wrfout_d01_2021-02-21_15:00:00' ' ' 20210221 160000 'wrfout_d01_2021-02-21_16:00:00' ' ' 20210221 170000 'wrfout_d01_2021-02-21_17:00:00' ' ' 20210221 180000 'wrfout_d01_2021-02-21_18:00:00' ' ' and edit *flxp_input.txt* .. code-block:: bash =====================FORMER PATHNAMES FILE=================== ./ ./metdata/ ./AVAILABLE ============================================================= =====================FORMER COMMAND FILE===================== 1 LDIRECT: 1 for forward simulation, -1 for backward simulation 20210221 000000 YYYYMMDD HHMISS beginning date of simulation 20210221 180000 YYYYMMDD HHMISS ending date of simulation 3600 SSSSS (int) output every SSSSS seconds 3600 SSSSS (int) time average of output (in SSSSS seconds) 180 SSSSS (int) sampling rate of output (in SSSSS seconds) 999999999 SSSSS (int) time constant for particle splitting (in seconds) 180 SSSSS (int) synchronisation interval of flexpart (in seconds) 10. CTL (real) factor by which time step must be smaller than tl 10 IFINE (int) decrease of time step for vertical motion by factor ifine 1 IOUT 1 concentration, 2 mixing ratio, 3 both, 4 plume traject, 5=1+4 0 IPOUT particle dump: 0 no, 1 every output interval, 2 only at end 0 LSUBGRID subgrid terrain effect parameterization: 1 yes, 0 no 0 LCONVECTION convection: 3 yes, 0 no 3600. DT_CONV (real) time interval to call convection, seconds 1 LAGESPECTRA age spectra: 1 yes, 0 no 0 IPIN continue simulation with dumped particle data: 1 yes, 0 no 0 IFLUX calculate fluxes: 1 yes, 0 no 0 IOUTPUTFOREACHREL CREATE AN OUPUT FILE FOR EACH RELEASE LOCATION: 1 YES, 0 NO 0 MDOMAINFILL domain-filling trajectory option: 1 yes, 0 no, 2 strat. o3 tracer 1 IND_SOURCE 1=mass unit , 2=mass mixing ratio unit 2 IND_RECEPTOR 1=mass unit , 2=mass mixing ratio unit 0 NESTED_OUTPUT shall nested output be used? 1 yes, 0 no 0 LINIT_COND INITIAL COND. FOR BW RUNS: 0=NO,1=MASS UNIT,2=MASS MIXING RATIO UNIT 1 TURB_OPTION 0=no turbulence; 1=diagnosed as in flexpart_ecmwf; 2 and 3=from tke. 0 CBL SCHEME 0=no, 1=yes. works if TURB_OPTION=1 1 SFC_OPTION 0=default computation of u*, hflux, pblh, 1=from wrf 0 WIND_OPTION 0=snapshot winds, 1=mean winds,2=snapshot eta-dot,-1=w based on divergence 0 TIME_OPTION 1=correction of time validity for time-average wind, 0=no need 1 OUTGRID_COORD 0=wrf grid(meters), 1=regular lat/lon grid 1 RELEASE_COORD 0=wrf grid(meters), 1=regular lat/lon grid 2 IOUTTYPE 0=default binary, 1=ascii (for particle dump only),2=netcdf 1000 NCTIMEREC (int) Time frames per output file, only used for netcdf 0 VERBOSE VERBOSE MODE,0=minimum, 100=maximum =====================FORMER AGECLASESS FILE================== 2 NAGECLASS number of age classes 7200 SSSSSS (int) age class in SSSSS seconds 999999 SSSSSS (int) age class in SSSSS seconds =====================FORMER OUTGRID FILE===================== -157.00 OUTLONLEFT geograhical longitude of lower left corner of output grid 58.0 OUTLATLOWER geographical latitude of lower left corner of output grid 24 NUMXGRID number of grid points in x direction (= # of cells ) 28 NUMYGRID number of grid points in y direction (= # of cells ) 1 OUTGRIDDEF outgrid defined 0=using grid distance, 1=upperright corner coordinate -141.0 DXOUTLON grid distance in x direction or upper right corner of output grid 63.0 DYOUTLON grid distance in y direction or upper right corner of output grid 3 NUMZGRID number of vertical levels 100.0 LEVEL height of level (upper boundary) 1000.0 LEVEL height of level (upper boundary) 20000.0 LEVEL height of level (upper boundary) =====================FORMER RECEPTOR FILE==================== 0 NUMRECEPTOR number of receptors =====================FORMER SPECIES FILE===================== 1 NUMTABLE number of variable properties. The following lines are fixed format XXXX|NAME |decaytime |wetscava |wetsb|drydif|dryhenry|drya|partrho |parmean|partsig|dryvelo|weight | AIRTRACER -999.9 -9.9E-09 -9.9 -9.9E09 -9.99 29.00 =====================FORMER RELEEASES FILE=================== 1 NSPEC total number of species emitted 0 EMITVAR 1 for emission variation 1 LINK index of species in file SPECIES 1 NUMPOINT number of releases 20210221 000000 ID1, IT1 beginning date and time of release 20210221 120000 ID2, IT2 ending date and time of release -149.9000 XPOINT1 (real) longitude [deg] of lower left corner 61.2000 YPOINT1 (real) latitude [deg] of lower left corner -149.9000 XPOINT2 (real) longitude [deg] of upper right corner 61.2000 YPOINT2 (real) latitude [DEG] of upper right corner 1 KINDZ (int) 1 for m above ground, 2 for m above sea level, 3 pressure 100.000 ZPOINT1 (real) lower z-level 1000.000 ZPOINT2 (real) upper z-level 500000 NPART (int) total number of particles to be released .1000E+04 XMASS (real) total mass emitted box1 NAME OF RELEASE LOCATION and run it .. code-block:: bash $ ./flexwrf_serial flxp_input.txt the input file used is flxp_input.txt Pre-generating random numbers RANLUX LUXURY LEVEL SET BY RLUXGO : 3 P= 223 RANLUX INITIALIZED BY RLUXGO FROM SEEDS 254 0 0 Calling set_mt19937 Calling new (mts(0)) Calling init (mts(0),iseed) Generating additional streams . . . 61200 SECONDS SIMULATED: 499999 PARTICLES: Uncertainty: 0.000 0.000 0.000 time needed 216.584000 64800 SECONDS SIMULATED: 499999 PARTICLES: Uncertainty: 0.000 0.000 0.000 time needed 191.740005 timemanager_serial: End of simulation reached, closing output file CONGRATULATIONS: YOU HAVE SUCCESSFULLY COMPLETED A FLEXPART_WRF MODEL RUN! note the two NetCDF output files .. code-block:: bash $ ls -l *.nc -rw-r--r--. 1 morton consult 49979 Mar 10 04:10 flxout_d01_20210221_010000.nc -rw-r--r--. 1 morton consult 65751 Mar 10 03:30 header_d01.nc and use *ncview* to sample the concentration from the newly-created *flxout_d01_20210221_010000.nc*. Note that I have placed a copy of the *ncview* binary in the repo, in *misc/utilities/ncview* so for convenience I assign the path to an environment variable .. code-block:: bash $ export NCVIEW=/dvlscratch/ATM/morton/git/high-res-atm/misc/utilities/ncview $ $NCVIEW flxout_d01_20210221_010000.nc .. |conc| image:: FlexWRFConc.png :scale: 70 % |conc| We can also review the topography from *header_d01.nc* output file .. code-block:: bash $ $NCVIEW header_d01.nc .. |topo| image:: FlexWRFHeaderTopography.png :scale: 70 % |topo| ---- Summary ======= For someone used to running Flexpart WRF, this was probably boring, but for the less experienced user, hopefully it serves as a little bit of guidance, and demonstrates how we can quickly create and execute a custom Flexpart WRF simulation driven by already-existing *wrfout* data (whose production was presented in :doc:`../../WrfInstall/TechNotes/DemoVirtualUsage`. ). After the demo described above, we could create more local run directories for a variety of simulations. Perhaps we just want to try the same simulation with different parameterisations. One approach would be to create different run directories for each case and, in each, go through the above steps with slightly different versions of *flxp_input.txt*, each using different parameters. .. toctree::