Compare revisions

fe69fe8e · 9a12001e · 528e99cd · e5dc526d · fb8835b7 · eb086bc3
--- a/.gitignore
+++ b/.gitignore
+bin/summa_actors
+build/cmake/build
+build/summa
+build/summa-sundials
+build/.vscode
+.vscode/settings.json
+.DS_Store
+laugh_tests
+build/summa_sundials
+build/summa_original
+build/source/testing/containers/output_container/main
+bin/submission_script.sh
+build/summa_kinsol
+bin/summa_kinsol
+build/flibs-0.9/
+build/cmake/build_be/
+build/cmake/build_kinsol/
+bin/data_be.csv
+bin/data_kinsol.csv
+bin/submission_copern.sh
+bin/summa_be
+bin/data_kinsol_state_vec.csv
+bin/plot_resuduial.py
+bin/state.png
+bin/submission_script_array.sh
+build/source/testing/containers/output_container/out.txt
+build/summa_old
+bin/config.json
+bin/summa_sundials
+build/cmake_build/
--- a/build/source/engine/access_write.f90
+++ b/build/source/engine/access_write.f90
--- a/README.md
+++ b/README.md
-# SUMMA-Actors: Structure for Unifying Multiple Modeling Alternatives with Actors
+# **NOTICE**
+**This repository has been frozen. SUMMA-Actors project development has moved to: https://github.com/uofs-simlab/Summa-Actors**
-SUMMA-Actors is a modified version of the already existing SUMMA software that can be 
+# SUMMA-Actors: Structure for Unifying Multiple Modeling Alternatives with Actors
-found [here](https://github.com/CH-Earth/summa#readme). SUMMA-Actors is a modification 
-of SUMMA that uses the Actor Model to increase scalability and fault-tolerance. The actor which is known as the basic unit of concurrent computation is at the heart of this 
-software. SUMMA-Actors is built using the [C++ Actor Framework](https://github.com/actor-framework/actor-framework). 
-## Compiling Summa-Actors
+SUMMA-Actors is a modified version of the already existing SUMMA software that can be found [here](https://github.com/CH-Earth/summa#readme). SUMMA-Actors uses the Actor Model to increase scalability and fault-tolerance. It is built using the [C++ Actor Framework](https://github.com/actor-framework/actor-framework). 
-SUMMA-Actors is written in C++ and FORTRAN and can be compiled with a C++ and FORTRAN 
-compiler from the GNU Compiler Collection. We have compiled SUMMA-Actors with the 
+## Documentation
-following compilers:
+A more in-depth documentation can be found [here](https://git.cs.usask.ca/numerical_simulations_lab/actors/Summa-Actors/-/wikis/home). SUMMA-Actors depends on many files from the [original SUMMA repo](https://github.com/CH-Earth/summa). Below is a quick start guide for compiling and running SUMMA-Actors, please consult our wiki for more in-depth documentation. Consider creating an issue for any missing documentation, asking questions, or providing suggestions.
+## Preliminaries
+SUMMA-Actors is meant to wrap around the existing SUMMA implementation. Therefore you will need to clone the SUMMA repo into the SUMMA-Actors build directory. SUMMA-Actors works with the master branch of SUMMA, which can be found [here](https://github.com/CH-Earth/summa). Below the directory structure is explained in more detail and the steps to compiling SUMMA-Actors are explained.
+## Directory Structure
+SUMMA-Actors is set up with the following sub-directories, we will consider the top level Summa Actors directory the `root_dir`:
+ - bin
+ - build
+   - includes
+   - source
+   - summa (https://github.com/KyleKlenk/summa/tree/summa-actors, summa-actors branch)
+   - CMakeLists.txt
+ - utils
+ - README.md
+First clone Summa-Actors to your workstation. Then cd into `build/` and clone `summa` into Summa-Actor's build directory as folder summa.
+## Compiling SUMMA-Actors
+### Dependencies
+SUMMA-Actors has only one additional dependency, the [C++ Actor Framework](https://github.com/actor-framework/actor-framework), specifically the [0.18.6 
+release](https://github.com/actor-framework/actor-framework/archive/refs/tags/0.18.6.tar.gz).
+The following steps can be used to install the C++ Actor Framework on a Linux system:
+```bash
+wget https://github.com/actor-framework/actor-framework/archive/refs/tags/0.18.6.tar.gz
+tar -xzf 0.18.6.tar.gz
+cd actor-framework-0.18.6/
+./configure --prefix=/path/to/install
+cd build
+make # [-j]
+make install # [as root if necessary]
+```
+Additional dependencies required by both SUMMA and SUMMA actors are:
 * g++
 * gfortran
-SUMMA-Actors depends on the following Libraries:
 * [NetCDF-Fortran](https://github.com/Unidata/netcdf-fortran)
 * [OpenBLAS](https://github.com/xianyi/OpenBLAS)
- * [C++ Actor Framework](https://github.com/actor-framework/actor-framework) 
+ * [SUNDIALS V6.6](https://github.com/LLNL/sundials/releases/tag/v6.6.0) (optional)
-Once the following libraries have been installed SUMMA-Actors can be compiled in 
+### Steps To Compile
-one of two ways. The first way is to modify the makefile directly and the second 
+ ```
-is to invoke the makefile by shellscript:
+ git clone https://git.cs.usask.ca/numerical_simulations_lab/actors/Summa-Actors.git
+ cd Summa-Actors/build
- 1. Makefile:
+ git clone -b summa-actors https://github.com/KyleKlenk/summa.git
-  Changes need to be made to the following variables in the Makefile:
+ cmake -B cmake_build -S . -DCMAKE_BUILD_TYPE=Build_Type
-    - F_MASTER = directory/above/build
+ cmake --build cmake_build --target all -j
-    - FC = gfortran
+ ```
-    - CC = g++
+Available build types are:
-    - INCLUDES = Path/to/netcdf/includes
+  * BE or BE_Debug: Builds for local machine
-    - LIBRARIES = Path/to/netcdf/lib & Path/to/openblas
+  * BE_Cluster or BE_Cluster_Debug: See below section (Modules for Alliance Machines)
-        -lnetcdff -lopenblas
-    - ACTORS_INCLUDES = $INCLUDES & Path/to/CAF/includes
+### Modules for Alliance Machines
-    - ACTORS_LIBRARIES = $LIBRARIES & PATH/to/CAF/lib & PATH/to/summa.so
+To compile on Alliance machines the following modules can be loaded:
-        -lcaf_core -lcaf_io -lsumma -lopenblas -lnetcdff
+```bash
+module load StdEnv/2020
-  Once these changes have been made SUMMA-Actors can be called with `make` from
+module load gcc/9.3.0
-  the build/ directory.
+module load openblas/0.3.17
+module load netcdf-fortran/4.5.2
-  Note: SUMMA is compiled as a shared library (libsumma.so) and the main program 
+module load caf
-  will need to know where this library is located in order to properly link it.
+```
-  By compiling both libsumma.so and summaMain in the same directory (build/) should be 
-  enough. However if there are issues, specifing where libsumma.so will be compiled can be done 
-  by adding the path to `ACTORS_LIBRARIES` in the makefile this should rectify the issues.
+## Running SUMMA-Actors
+SUMMA-Actors can be run in two modes, distributed and non-distributed. The distributed mode used to create ad-hoc clusters. The command line arguments are kept as close to the original SUMMA as possible. However, there are options not yet available in SUMMA-Actors that are in the original SUMMA. Here is a full list of the available and unavailable options:
- 1. ShellScript:
+```   
-  In the build directory exists a example_compile.sh script that can be modified.
+Usage: summa_actors -m master_file [-g startGRU countGRU] [-c config_file] [-b backup_server] [-s server_mode]
-  This is usually used for HPC computing environments and includes which modules 
+  Available options:
-  to load for Compute Canada or the University of Saskatchewan's Copernicus. 
+    -m, --master:         Define path/name of master file (can be specified in config)
-  example_compile.sh contains instructions on what parts to modify and how to 
+    -g, --gru:            Run a subset of countGRU GRUs starting from index startGRU 
-  invoke the makefile from the script.
+    -c, --config:         Path name of the Summa-Actors config file (optional but recommended)
+        --gen-config:     Generate a config file
-  Once the shellscript has been modified running it with `source your_script.sh` will compile 
+    -b, --backup-server:  Start backup server, requires a server and config_file
-  SUMMA-Actors.
+    -s, --server-mode:    Enable server mode
+    -h, --help:           Print this help message
+  Unimplemented Options:
-After SUMMA-Actors is compiled you should be left with a libsumma.so and a summaMain file 
+    -n --newFile          Define frequency [noNewFiles,newFileEveryOct1] of new output files
-in your build directory.
+    -s --suffix           Add fileSuffix to the output files
+    -h --hru              Run a single HRU with index of iHRU
-It is important to set the `LD_LIBRARY_PATH` environment variable before attempting to run 
+    -r --restart          Define frequency [y,m,d,e,never] to write restart files
-summa. This variable needs to point to the location of libsumma.so. If you compiled using the 
+    -p --progress         Define frequency [m,d,h,never] to print progress
-shellscript using the `source` command and following the directions within the example_compile.sh 
+    -v --version          Display version information of the current build
-this should already be set for you. 
+```
+Instructions for using each mode are provided below, if you just want to skip ahead and get started using the default options.
-## Running Summa-Actors
+### Non-Distributed Mode (Similar to original SUMMA)
-Once the binary is compiled it can be run like the following example command:
+Using SUMMA-Actors in non-distributed mode is like running the normal SUMMA. The difference is that HRUs will run concurrently in SUMMA-Actors and maximize resource use. Usage is very close to SUMMA, and the same input files are required. In depth documentation for configuring a SUMMA run can be found [here](https://summa.readthedocs.io/en/latest/).
-./summaMain -g 1 -n 10 -c /path/to/config/directory --config-file=/path/to/actors/config/file
+Usage is as follows: ./summa_actors -m master_file [-g startGRU countGRU] [-c config_file]
- -g = starting index of the first GRU to compute
+The master_file for SUMMA-Actors can either be defined explicitly on the command line or it can be included in the new to SUMMA-Actors config_file. The config_file is optional but it is highly recommended as it does allow users to fine tune how SUMMA-Actor will perform. We are working to automate these aspects but for now its best to define it for the domain and compute environment in use. Using `./summa_actors --gen-config` will generate a configuration file that can then be filled in. See the section on config_file in this readme for more information. 
- -c = number of grus to run
- -m = path to the file manager
- OPTIONAL: --config-file = /path/to/config/file
+### Distributed Mode
-  This config file specifies to the C++ Actor Framework how many threads to use when executing the program. If left out the C++ Actor Framework will automatically set this value based on your system.
+Using SUMMA-Actors in distributed mode allows SUMMA-Actors to solve batches and dymically assign them to nodes and reassign them in the event of node failures.
-#### Config File ####
+To use this feature there are 3 actors that can be spawned to cluster nodes together and add additional redundancy to the system. All settings are to be defined in the config_file for this mode. In the config file the user must set `distributed_mode` to true, define a `port`, and configure the domain to run. The domain is defined by how many total GRUs the user wishes to compute and the number of GRUs that should be assembled into a batch. Finally, the `server_list` should be set to the hostname of the backup_servers that are started.
-SUMMA-Actors settings can be modified from a JSON file provided in config/Summa_Actors_Settings.json.
-There are three types of actors that can be configured:
+To start the system one start the server first with `./summa_actors -c -s`
-  * SummaActor
+Backup servers can be added with `./summa_actors -c -b`
-    - OutputStructureSize = The number of timesteps in which an HRU can hold before needing to contact 
+Clients can simply be added with `./summa_actors -c`
-    the file_access_actor to write the data to a file.
-    - maxGRUPerJob = The number of GRUs that will be attemtpted to run at once. For example, if this value 
+NOTE: Each system will need a copy of the forcing data or input data, or the data should be in a singular location like on a cluster system.
-    is set to 500 and you invoke the program with ./summaMain -g 1 -n 1000 -c /path/to/config/directory. 
-    SUMMA-Actors will only spawn 500 actors at a time and compute all 1000 in two batches.
+### Config File
-    Both of the above setting control the amount of RAM SUMMA-Actors uses. Larger numbers can cause your 
+The config file is a JSON file that is used to configure SUMMA-Actors. It is highly recommended to use a config file as it allows the user to fine tune how SUMMA-Actors will perform. Using `./summa_actors --gen-config` will generate a configuration file that can then be filled in. Below is a list of the available options and their descriptions.
-    job to run out of memory. We have found that setting both to 500 uses around 20GB of RAM for reference.
+#### Distributed_Settings
-  * JobActor
+The distributed settings are used to configure the distributed mode of SUMMA-Actors. The following options are available:
-    - FileManagerPath = Path the the fileManager.txt file needed by SUMMA. This is remained relativley 
+  - distributed_mode: (true/false) Enables distributed mode
-    unchanged from the original version of SUMMA. With two additions. An example file is provided in the 
+  - servers_list: (list of strings) List of hostnames for backup servers
-    config/ directory called fileManager_example.txt
+  - port: (int) Port to use for communication
-    - outputCSV = Boolean value for if you would like individual HRU run-time statsicts when they complete.
+  - total_hru_count: (int) Total number of HRUs in the entire domain
-    - csvPath = The path that the csv file will be written to.
+  - num_hru_per_batch: (int) Number of HRUs to assemble into a batches
-  * HRUActor
+#### Summa_Actor
-    - printOutput = Boolean value for if you would like each HRU to print information on where it is in 
+The Summa_Actor settings are used to restrict how many HRU actor can run at once. The following options are available:
-    its computation. ie. what timestep it is on and some other timing information.
+  - max_gru_per_job: (int) Maximum number of HRUs that can be run at once
-    - outputFrequency = The frequency in which you would like an HRU printing to stdout. The number specified is the interval in timesteps in which an HRU will print. Note: Lower numbers can see decreased performance as stdout will begin to lag the more that needs to be printed.
+#### Job_Actor
+The Job_Actor settings can specify the file manager path so it does not need to be specified on the command line, and the maximum number of times to attempt an HRU before giving up. The following options are available:
+  - file_manager_path: (string) Path to the file manager file
+  - max_run_attempts: (int) Maximum number of times to attempt an HRU before giving up
+#### File_Access_Actor
+The File_Access_Actor settings are used to configure the file access actor. The following options are available:
+  - num_partitions_in_output_buffer: (int) Number of partitions in the output buffer
+  - num_timesteps_in_output_buffer: (int) Number of timesteps in the output buffer
+#### HRU_Actor
+The HRU_Actor settings are used to configure the HRU actor. The following options are available:
+  - print_output: (true/false) Print output to the screen
+  - output_frequency: (int) Frequency to print output to the screen
+  - dt_init_factor: (int) Factor to multiply dt_init by
+  - rel_tol: (float) Relative tolerance for the HRU actor (Requires Sundials)
+  - abs_tol: (float) Absolute tolerance for the HRU actor (Requires Sundials)
 ## Credits
-The inital implementation of SUMMA is credited to the inital publications below. These 
+The initial implementation of SUMMA is credited to the initial publications below. These 
 publications can be found in [Water Resources Research](http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1944-7973).
 * Clark, M. P., B. Nijssen, J. D. Lundquist, D. Kavetski, D. E. Rupp, R. A. Woods, J. E. Freer, E. D. Gutmann, A. W. Wood, L. D. Brekke, J. R. Arnold, D. J. Gochis, R. M. Rasmussen, 2015a: A unified approach for process-based hydrologic modeling: Part 1. Modeling concept. _Water Resources Research_, [doi:10.1002/2015WR017198](http://dx.doi.org/10.1002/2015WR017198).<a id="clark_2015a"></a>

--- a/actorsChanges.txt
+++ b/actorsChanges.txt
+driver
+/summaActors_alarms.f90 -- remove
+/summaActors_globalData.-- could combine and remove, renamed for now
+/summaActors_type.f90   -- could combine and remove, renamed for now, adds a new function, but don't use both
+/summaActors_util.f90   -- could combine and remove, renamed for now, adds a new function, but don't use both
+/summaversion.inc       -- remove
+dshare
+/csv_file_1d.f90 -- remove?? not seeing it used anywhere?
+/csv_file_2d.f90 -- remove?? not seeing it used anywhere?
+/csv_file.f90   -- remove?? IS this only in kinsol? do we need?
+/data_types.f90 -- has a lot
+/globalData.f90 -- remove with ACTORS_ACTIVE adds forcing stuff and tmZoneOffsetFracDay, fracJulDay, yearLength not as global
+/varLookup.f90  -- remove with ACTORS_ACTIVE uses a iso_c binding call
+engine
+can remove sundials folder shouldn't need with SUNDIALS_ACTIVE
+/sundials/coupled_em.f90 -- remove with ACTORS_ACTIVE needs fracJulDay, yearLength
+/sundials/mDecisions.f90 -- remove with ACTORS_ACTIVE called with iso_c binding
+/sundials/t2enthalpy.f90 -- remove
+/alloc_fileAccess.f90 -- new
+/allocspace[Actors].f90 -- this is a replacement, rename, and see how changes CHECK IF NECESSARY!
+/check_icondActors.f90-- could combine and remove, renamed for now, adds a new function, but don't use both
+/checkStruc.f90 -- remove
+/coupled_em.f90 -- remove shouldn't need the non-sundials version
+/derivforce.f90 -- remove with ACTORS_ACTIVE tmZoneOffsetFracDay needs to be variable not global
+/ffile_info.f90 -- very different, call this ffile_infoActors?
+/mDecisions.f90 -- remove shouldn't need the non-sundials version
+/read_dimension.-- remove, part of read_attrb, - some deallocation changes, use this in all (not actually used in actors, ??)
+ - also Actors assumes don't have aspect variable .. why?
+/read_pinit.f90 -- remove
+/vegPhenlgy.f90 -- remove with ACTORS_ACTIVE needs fracJulDay, yearLength
+SUMMA_FILEACCESS_INTERFACE = \
+		output_structure.f90 \
+		cppwrap_fileAccess.f90 \
+		read_attrb.f90 \
+		read_force.f90 \
+		read_param.f90 \
+		read_icondFromStructure.f90 \
+ARE THESE ALL SORT OF REPEATS?? DO WE EVEN USE ALL OF THEM? specifically read_icondFromStructure ... what is that ... do we call its
+Can we give them the same names as the regular summa files if they are repeats, so we know to change them??
+changed calls in run_oneHRU so can use (modded) currently in actors /derivforce.f90 /coupled_em.f90 /vegPhenlgy.f90 as is
+hookup
+/ascii_util.f90          -- use Kyle's version everywhere!! (so remove)
+/summaActors_FileManager.-- could combine and remove, renamed for now??
+lapack    -- remove
+/Makefile -- remove
+/README   -- remove
+netcdf
+/def_output.f90   -- has a lot ... use Kyle's? Hard to tell which is Actors specific and which is just changed
+/netcdf_util.f90  -- remove
+/read_icondActors.-- has a lot ... could combine and remove, renamed for now,
+/writeOutput.f90  -- part of modelwrite.f90 -- renamed to modelwrite so can replace, and then IS THIS VERSION BETTER can we change the other version and put a few compiler directives in?
+/writeRestart.f90 -- remove, same in modelwrite, part of it
--- a/build/CMakeLists.txt
+++ b/build/CMakeLists.txt
+cmake_minimum_required(VERSION 3.10 FATAL_ERROR)
+project(summa_actors LANGUAGES CXX Fortran)
+enable_language(C)
+SET (CMAKE_Fortran_COMPILER  gfortran)
+include(FortranCInterface)
+FortranCInterface_VERIFY(CXX)
+get_filename_component(PARENT_DIR "${CMAKE_CURRENT_SOURCE_DIR}" DIRECTORY)
+set(EXEC_DIR ${PARENT_DIR}/bin) # set the output directory for executables
+SET(F_MASTER ${PARENT_DIR}/build/summa)
+# Add options for build type
+set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${EXEC_DIR})
+set(CMAKE_CONFIGURATION_TYPES BE BE_Debug BE_Cluster BE_Cluster_Debug)
+# Set Compiler Options
+if(CMAKE_BUILD_TYPE MATCHES Debug)
+    message("\nSetting Debug Options\n")
+    add_compile_definitions(DEBUG)
+    set(FLAGS_NOAH -g -O0 -fbacktrace -fbounds-check -ffree-form -ffree-line-length-none -fmax-errors=0 -fPIC -Wfatal-errors ${FLAGS_OPT})
+    set(FLAGS_ALL  -g -O0 -fbacktrace -fbounds-check -ffree-line-length-none -fmax-errors=0 -fPIC -Wfatal-errors -cpp ${FLAGS_OPT})
+    set(FLAGS_CXX  -g -O0 -fbounds-check -Wfatal-errors -std=c++17 ${FLAGS_OPT})
+else()
+    message("\nSetting Release Options")
+    set(FLAGS_NOAH -O3 -ffree-form -ffree-line-length-none -fmax-errors=0 -fPIC -Wfatal-errors ${FLAGS_OPT})
+    set(FLAGS_ALL  -O3 -ffree-line-length-none -fmax-errors=0 -fPIC -Wfatal-errors -cpp ${FLAGS_OPT})
+    set(FLAGS_CXX  -O3 -Wfatal-errors -std=c++17 ${FLAGS_OPT})
+endif()
+find_package(CAF COMPONENTS core io REQUIRED)
+set(CAF_INCLUDES ${CAF_INCLUDE_DIRS})
+set(CAF_LIBRARIES CAF::core CAF::io)
+set(EXEC_NAME summa_be)
+if (CMAKE_BUILD_TYPE MATCHES Cluster)    
+    find_package(OpenBLAS REQUIRED)
+    # Set include directories
+    set(INCLUDES $ENV{EBROOTNETCDFMINFORTRAN}/include ${netCDF_INCLUDES} ${OpenBLAS_INCLUDES})
+    set(LIBRARIES SUMMA_NOAHMP  ${OpenBLAS_LIBRARIES} -lnetcdff)
+    set(INC_ACTORS 
+        ${CAF_INCLUDES} 
+        ${PARENT_DIR}/build/includes/global 
+        ${PARENT_DIR}/build/includes/summa_actor 
+        ${PARENT_DIR}/build/includes/job_actor 
+        ${PARENT_DIR}/build/includes/file_access_actor 
+        ${PARENT_DIR}/build/includes/hru_actor)
+    set(LIB_ACTORS 
+        ${CAF_LIBRARIES}
+        -lnetcdf 
+        -lcaf_core 
+        -lcaf_io)
+else()
+    # Set include and library for Fortran portion of the code
+    set(INCLUDES "/usr/include" "/usr/local/include")
+    set(LIBRARIES -lnetcdff -lopenblas SUMMA_NOAHMP)
+    # Set include and library for C++ portion of the code
+    set(INC_ACTORS CAF::core CAF::io ${INCLUDES}
+        "${PARENT_DIR}/build/includes/global"
+        "${PARENT_DIR}/build/includes/summa_actor"
+        "${PARENT_DIR}/build/includes/job_actor"
+        "${PARENT_DIR}/build/includes/file_access_actor"
+        "${PARENT_DIR}/build/includes/hru_actor")
+    link_directories("/usr/local/lib")
+    set(LIB_ACTORS CAF::core CAF::io -lopenblas -lnetcdff -lnetcdf)
+endif()
+# Define directories that contains source code
+set(DRIVER_DIR ${F_MASTER}/build/source/driver)
+set(DSHARE_DIR ${F_MASTER}/build/source/dshare)
+set(ENGINE_DIR ${F_MASTER}/build/source/engine)
+set(HOOKUP_DIR ${F_MASTER}/build/source/hookup)
+set(NETCDF_DIR ${F_MASTER}/build/source/netcdf)
+set(NOAHMP_DIR ${F_MASTER}/build/source/noah-mp)
+# Define Actors specific directories
+set(ACTORS_DIR ${PARENT_DIR}/build/source/actors)
+set(FILE_ACCESS_DIR ${ACTORS_DIR}/file_access_actor)
+set(JOB_ACTOR_DIR   ${ACTORS_DIR}/job_actor)
+set(HRU_ACTOR_DIR   ${ACTORS_DIR}/hru_actor)
+# NOAHMP modules
+set(NOAHMP
+    ${NOAHMP_DIR}/module_model_constants.F
+    ${NOAHMP_DIR}/module_sf_noahutl.F
+    ${NOAHMP_DIR}/module_sf_noahlsm.F
+    ${NOAHMP_DIR}/module_sf_noahmplsm.F)
+# Free versions of numerical recipes utilities for NOAH-MP modules
+set(NRUTIL
+    ${ENGINE_DIR}/f2008funcs.f90
+    ${ENGINE_DIR}/nr_utility.f90
+    ${ENGINE_DIR}/nrtype.f90)
+# Free versions of numerical recipes procedures for SUMMA modules
+set(NRPROC
+    ${ENGINE_DIR}/expIntegral.f90
+    ${ENGINE_DIR}/spline_int.f90)
+# Hook-up modules
+set(HOOKUP
+    ${HOOKUP_DIR}/ascii_util.f90
+    ${HOOKUP_DIR}/summaFileManager.f90)
+# Data modules
+set(DATAMS
+    ${DSHARE_DIR}/data_types.f90
+    ${ACTORS_DIR}/global/actor_data_types.f90
+    ${DSHARE_DIR}/flxMapping.f90
+    ${DSHARE_DIR}/get_ixname.f90
+    ${DSHARE_DIR}/globalData.f90
+    ${DSHARE_DIR}/multiconst.f90
+    ${DSHARE_DIR}/outpt_stat.f90
+    ${DSHARE_DIR}/popMetadat.f90
+    ${DSHARE_DIR}/var_lookup.f90)
+# Utility modules
+set(UTILMS
+    ${ENGINE_DIR}/matrixOper.f90
+    ${ENGINE_DIR}/mDecisions.f90
+    ${ENGINE_DIR}/snow_utils.f90
+    ${ENGINE_DIR}/soil_utils.f90
+    ${ENGINE_DIR}/time_utils.f90
+    ${ENGINE_DIR}/updatState.f90)
+# NetCDF routines
+set(NETCDF
+    ${NETCDF_DIR}/def_output.f90
+    ${NETCDF_DIR}/netcdf_util.f90
+    ${NETCDF_DIR}/read_icond.f90)
+# Preliminary modules
+set(PRELIM
+    ${ENGINE_DIR}/allocspace.f90
+    ${ENGINE_DIR}/check_icond.f90
+    ${ENGINE_DIR}/checkStruc.f90
+    ${ENGINE_DIR}/childStruc.f90
+    ${ENGINE_DIR}/conv_funcs.f90
+    ${ENGINE_DIR}/convE2Temp.f90
+    ${FILE_ACCESS_DIR}/ffile_info.f90
+    ${ENGINE_DIR}/read_pinit.f90
+    ${ENGINE_DIR}/read_attrb.f90
+    ${ENGINE_DIR}/paramCheck.f90
+    ${ENGINE_DIR}/pOverwrite.f90
+    ${ENGINE_DIR}/sunGeomtry.f90
+    ${ENGINE_DIR}/read_param.f90)
+    # Model run support modules
+set(MODRUN
+    ${ENGINE_DIR}/canopySnow.f90
+    ${ENGINE_DIR}/derivforce.f90
+    ${ENGINE_DIR}/getVectorz.f90
+    ${ENGINE_DIR}/indexState.f90
+    ${ENGINE_DIR}/layerMerge.f90
+    ${ENGINE_DIR}/layerDivide.f90
+    ${ENGINE_DIR}/qTimeDelay.f90
+    ${ENGINE_DIR}/snowAlbedo.f90
+    ${ENGINE_DIR}/snwCompact.f90
+    ${ENGINE_DIR}/tempAdjust.f90
+    ${ENGINE_DIR}/updateVars.f90
+    ${ENGINE_DIR}/var_derive.f90
+    ${ENGINE_DIR}/volicePack.f90)
+# Solver main modules
+set(SOLVER
+    ${ENGINE_DIR}/bigAquifer.f90
+    ${ENGINE_DIR}/computFlux.f90
+    ${ENGINE_DIR}/computJacob.f90
+    ${ENGINE_DIR}/computResid.f90
+    ${ENGINE_DIR}/coupled_em.f90
+    ${ENGINE_DIR}/diagn_evar.f90
+    ${ENGINE_DIR}/eval8summa.f90
+    ${ENGINE_DIR}/groundwatr.f90
+    ${ENGINE_DIR}/opSplittin.f90
+    ${ENGINE_DIR}/snowLiqFlx.f90
+    ${ENGINE_DIR}/soilLiqFlx.f90
+    ${ENGINE_DIR}/ssdNrgFlux.f90
+    ${ENGINE_DIR}/stomResist.f90
+    ${ENGINE_DIR}/summaSolve.f90
+    ${ENGINE_DIR}/systemSolv.f90
+    ${ENGINE_DIR}/varSubstep.f90
+    ${ENGINE_DIR}/vegLiqFlux.f90
+    ${ENGINE_DIR}/vegNrgFlux.f90
+    ${ENGINE_DIR}/vegPhenlgy.f90
+    ${ENGINE_DIR}/vegSWavRad.f90)
+set(DRIVER
+    ${DRIVER_DIR}/summa_alarms.f90
+    ${DRIVER_DIR}/summa_globalData.f90)
+# Actors interface modules
+set(INTERFACE
+    ${ACTORS_DIR}/global/cppwrap_auxiliary.f90
+    ${ACTORS_DIR}/global/cppwrap_datatypes.f90
+    ${ACTORS_DIR}/global/cppwrap_metadata.f90)
+set(FILE_ACCESS_INTERFACE
+    ${FILE_ACCESS_DIR}/cppwrap_fileAccess.f90
+    ${FILE_ACCESS_DIR}/output_structure.f90
+    ${FILE_ACCESS_DIR}/read_force.f90
+    ${FILE_ACCESS_DIR}/fileAccess_writeOutput.f90)
+set(JOB_INTERFACE
+    ${JOB_ACTOR_DIR}/job_actor.f90)
+set(HRU_INTERFACE
+    ${HRU_ACTOR_DIR}/hru_init.f90
+    ${HRU_ACTOR_DIR}/hru_read.f90
+    ${HRU_ACTOR_DIR}/hru_modelRun.f90
+    ${HRU_ACTOR_DIR}/hru_writeOutput.f90)
+# Actors actual actor modules
+set(ACTORS_GLOBAL
+    ${ACTORS_DIR}/global/auxiliary.cpp
+    ${ACTORS_DIR}/global/global.cpp
+    ${ACTORS_DIR}/global/message_atoms.cpp
+    ${ACTORS_DIR}/global/settings_functions.cpp
+    ${ACTORS_DIR}/global/timing_info.cpp)
+set(SUMMA_ACTOR
+    ${ACTORS_DIR}/summa_actor/batch.cpp
+    ${ACTORS_DIR}/summa_actor/batch_container.cpp
+    ${ACTORS_DIR}/summa_actor/client.cpp
+    ${ACTORS_DIR}/summa_actor/client_container.cpp
+    ${ACTORS_DIR}/summa_actor/summa_actor.cpp
+    ${ACTORS_DIR}/summa_actor/summa_backup_server.cpp
+    ${ACTORS_DIR}/summa_actor/summa_client.cpp
+    ${ACTORS_DIR}/summa_actor/summa_server.cpp)
+set(FILE_ACCESS_ACTOR
+    ${ACTORS_DIR}/file_access_actor/file_access_actor.cpp
+    ${ACTORS_DIR}/file_access_actor/forcing_file_info.cpp
+    ${ACTORS_DIR}/file_access_actor/output_container.cpp)
+set(JOB_ACTOR
+    ${ACTORS_DIR}/job_actor/GRU.cpp
+    ${ACTORS_DIR}/job_actor/job_actor.cpp)
+set(HRU_ACTOR
+    ${ACTORS_DIR}/hru_actor/hru_actor.cpp)
+#=========================================================================================
+# COMPILE PART 3: Collect the subroutines into build groups depending on build type
+#=========================================================================================
+set(COMM_ALL
+    ${NRPROC}
+    ${HOOKUP}
+    ${DATAMS}
+    ${UTILMS}
+    ${INTERFACE})
+set(SUMMA_ALL
+    ${NETCDF}
+    ${PRELIM}
+    ${MODRUN}
+    ${SOLVER}
+    ${DRIVER})
+set(SUMMA_ALL 
+    ${SUMMA_ALL}
+    ${FILE_ACCESS_INTERFACE}
+    ${JOB_INTERFACE}
+    ${HRU_INTERFACE})
+set(MAIN_ACTOR ${ACTORS_DIR}/main.cpp)
+# Define version number, not working correctly
+set(VERSIONFILE     ${DRIVER_DIR}/summaversion.inc)
+execute_process(COMMAND "    ${GIT_EXECUTABLE} tag | tail -n 1" OUTPUT_VARIABLE VERSION)
+execute_process(COMMAND "date" OUTPUT_VARIABLE BULTTIM)
+execute_process(COMMAND "    ${GIT_EXECUTABLE} describe --long --all --always | sed -e's/heads\///'" OUTPUT_VARIABLE GITBRCH)
+execute_process(COMMAND "    ${GIT_EXECUTABLE} rev-parse HEAD" OUTPUT_VARIABLE GITHASH)
+#=========================================================================================
+# COMPILE PART 4: Do the compilation
+#=========================================================================================
+# update version information, not working correctly
+file(WRITE  ${VERSIONFILE} "character(len=64), parameter     :: summaVersion = '${VERSION}'\n")
+file(APPEND ${VERSIONFILE} "character(len=64), parameter     :: buildTime = ''\n")
+file(APPEND ${VERSIONFILE} "character(len=64), parameter     :: gitBranch = '${GITBRCH}'\n")
+file(APPEND ${VERSIONFILE} "character(len=64), parameter     :: gitHash = '${GITHASH}'")
+# Build SUMMA_NOAHMP Object
+add_library(SUMMA_NOAHMP OBJECT ${NOAHMP} ${NRUTIL})
+target_compile_options(SUMMA_NOAHMP PRIVATE ${FLAGS_NOAH})
+# Build SUMMA_COMM Object
+add_library(SUMMA_COMM OBJECT ${COMM_ALL})
+target_compile_options(SUMMA_COMM PRIVATE ${FLAGS_ALL})
+target_include_directories(SUMMA_COMM PRIVATE ${INCLUDES})
+target_link_libraries(SUMMA_COMM PUBLIC SUMMA_NOAHMP ${FLAGS_ALL}) # added flags to the link step
+add_library(summaactors SHARED ${SUMMA_ALL})
+target_compile_options(summaactors PRIVATE ${FLAGS_ALL})
+target_include_directories(summaactors PUBLIC ${INCLUDES})
+target_link_libraries(summaactors PUBLIC ${LIBRARIES} SUMMA_NOAHMP SUMMA_COMM)
+add_executable(${EXEC_NAME} 
+               ${MAIN_ACTOR}
+               ${ACTORS_GLOBAL}
+               ${FILE_ACCESS_ACTOR}
+               ${JOB_ACTOR}
+               ${HRU_ACTOR}
+               ${SUMMA_ACTOR}
+               ${SUMMA_CLIENT}
+               ${SUMMA_SERVER})
+set_property(TARGET ${EXEC_NAME} PROPERTY LINKER_LANGUAGE Fortran)
+target_compile_options(${EXEC_NAME} PUBLIC ${FLAGS_CXX})
+target_include_directories(${EXEC_NAME} PUBLIC ${INC_ACTORS})
+target_link_libraries( ${EXEC_NAME} ${LIB_ACTORS} summaactors)
\ No newline at end of file
--- a/build/example_compile.sh
+++ b/build/example_compile.sh
-#!/bin/bash
-#### load modules if using Compute Canada or Copernicus ####
-# module load gcc/9.3.0
-# module load netcdf-fortran
-# module load openblas
-# module load caf
-#### Specifiy Master Directory, parent of build directory
-# export F_MASTER=/path/to/summaActors
-#### Specifiy Compilers ####
-# export FC=gfortran
-# export CC=g++
-#### Includes and Libraries ####
-# export INCLUDES = NETCDF and OPENBLAS
-# export LIBRARIES = NETCDF and OPENBLAS
-# export ACTORS_INCLUDES = C++ Actor Framework 
-# export ACTORS_LIBRARIES = C++ Actor Framework and
-#   The directory in which libsumma.so resides
-#### Compile with the Makefile ####
-# make -f ${F_MASTER}/build/makefile lib # libsumma.so part
-# mv libsumma.so ${F_MASTER}/bin # optional move of libsumma (just ensure that summaMain knows where to find it)
-# make -f ${F_MASTER}/build/makefile main # summaMain part
-# mv summaMain ${F_MASTER}/bin # optional move, cleans things up
-# export LD_LIBRARY_PATH=Path/to/libsumma.so
\ No newline at end of file
--- a/build/includes/file_access_actor/file_access_actor.hpp
+++ b/build/includes/file_access_actor/file_access_actor.hpp
+#pragma once
+#include "caf/actor.hpp"
+#include "forcing_file_info.hpp"
+#include "timing_info.hpp"
+#include "output_container.hpp"
+#include "settings_functions.hpp"
+#include "fortran_data_types.hpp"
+#include "auxilary.hpp"
+#include "global.hpp"
+/*********************************************
+ * File Access Actor Fortran Functions
+ *********************************************/
+extern "C" {
+  void fileAccessActor_init_fortran(void* handle_forcing_file_info, int* num_forcing_files, int* num_timesteps,
+                                    int* num_timesteps_output_buffer, void* handle_output_ncid, int* startGRU,
+                                    int* numGRU, int* numHRU, int* err);
+  void writeOutput_fortran(void* handle_ncid, int* num_steps, int* start_gru, int* max_gru, 
+                           bool* writeParamFlag, int* err);
+  void read_forcingFile(void* forcFileInfo, int* currentFile, int* stepsInFile,
+                        int* startGRU, int* numGRU, int* err);
+  void FileAccessActor_DeallocateStructures(void* handle_forcFileInfo, void* handle_ncid);
+}
+/*********************************************
+ * File Access Actor state variables
+ *********************************************/
+namespace caf {
+struct file_access_state {
+    // Variables set on Spawn
+    caf::actor parent; 
+    int start_gru;
+    int num_gru;
+    void *handle_forcing_file_info = new_handle_file_info(); // Handle for the forcing file information
+    void *handle_ncid = new_handle_var_i();                  // output file ids
+    int num_vectors_in_output_manager;
+    int num_steps;
+    int stepsInCurrentFile;
+    int numFiles;
+    int filesLoaded;
+    int err;
+    int num_output_steps;
+    Output_Container* output_container;
+    File_Access_Actor_Settings file_access_actor_settings;
+    std::vector<Forcing_File_Info> forcing_file_list; // list of steps in file
+     // Timing Variables
+    TimingInfo file_access_timing;
+};
+// called to spawn a file_access_actor
+behavior file_access_actor(stateful_actor<file_access_state>* self, int startGRU, int numGRU, 
+   File_Access_Actor_Settings file_access_actor_settings, actor parent);
+/*********************************************
+ * Functions for the file access actor
+ *********************************************/
+/* Setup and call the fortran routine that writes the output */
+void writeOutput(stateful_actor<file_access_state>* self, Output_Partition* partition);
+} // end namespace
\ No newline at end of file
--- a/build/includes/file_access_actor/forcing_file_info.hpp
+++ b/build/includes/file_access_actor/forcing_file_info.hpp
+#pragma once
+#include <vector>
+class Forcing_File_Info {
+    private:
+        int file_ID;
+        int num_steps;
+        bool is_loaded;
+    public:
+        Forcing_File_Info(int file_ID);
+        int getNumSteps();
+        bool isFileLoaded();
+        void updateIsLoaded();
+        void updateNumSteps(int num_steps);
+};
+struct Forcing_Info {
+    int num_vars;
+    int num_timesteps;
+    std::vector<int> index_forc_var;
+    std::vector<int> ncid_var;
+};
\ No newline at end of file
--- a/build/includes/file_access_actor/output_container.hpp
+++ b/build/includes/file_access_actor/output_container.hpp
+#pragma once
+#include "caf/actor.hpp"
+#include <optional>
+#include <cmath>
+#include "fortran_data_types.hpp"
+#include <vector>
+#include <iostream>
+/*
+ * This class manages a portion of the HRUs in the model.
+ * All HRUs are grouped into partitions/objects of this class.
+ */
+class Output_Partition {
+  private:
+    int start_local_gru_index;    // The index of the first GRU in the partition
+    int end_local_gru_index;      // The index of the last GRU in the partition
+    int num_local_grus;           // The number of GRUs in the partition
+    int num_active_grus;          // The number of GRUs that have not failed
+    int num_timesteps_simulation; // The number of timesteps in the simulation
+    int num_stored_timesteps;     // The number of timesteps held within the partition
+    bool write_params = true;     // Flag to write the parameters to the output file (only performed once)
+    std::vector<caf::actor> ready_to_write_list;
+    std::vector<int> failed_gru_index_list; // The list of GRUs that have failed
+  public:
+    Output_Partition(int start_local_gru_index, int num_local_grus, int num_timesteps_simulation, int num_timesteps_buffer);
+    ~Output_Partition();
+    // Set the GRU to ready to write
+    void setGRUReadyToWrite(caf::actor gru_actor);
+    // Check if all GRUs are ready to write
+    bool isReadyToWrite();
+    // Get the max index of the GRUs in the partition
+    int getMaxGRUIndex();
+    // Get the number of timesteps stored in the partition
+    int getNumStoredTimesteps();
+    // Get the start gru index
+    int getStartGRUIndex();
+    // Update the number of timesteps remaining in the simulation
+    void updateTimeSteps();
+    // Get the list of GRUs that have written so we can send them the next set of timesteps
+    std::vector<caf::actor> getReadyToWriteList();
+    // Reset the list of GRUs that are ready to write
+    void resetReadyToWriteList();
+    // Add a GRU index to the list of failed GRUs
+    void addFailedGRUIndex(int local_gru_index);
+    std::vector<int> getFailedGRUIndexList();
+    int getNumActiveGRUs();
+    int getNumLocalGRUs();
+    int getRemainingTimesteps();
+    bool isWriteParams();
+};
+/*
+ * This class is used to store informaiton about when 
+ * HRUs are ready to write. This class does not store
+ * the data of the HRUs only the information about if 
+ * HRUs are ready to write and which HRUs should be 
+ * written to the output file.
+*/
+class Output_Container {
+  private:
+    int num_partitions; // The number of partitions in the model
+    int num_grus_per_partition; // The average number of GRUs per partition
+    int num_grus; // The number of GRUs in the model
+    int num_timesteps_simulation; // The number of timesteps in the simulation
+    int num_stored_timesteps; // The number of timesteps a partion can hold before needing to write
+    bool rerunning_failed_hrus = false;
+    std::vector<Output_Partition*> output_partitions; // This is the main output partition
+    std::vector<int> failed_gru_index_list; // The list of GRUs that have failed
+    // Private Method
+  public:
+    Output_Container(int num_partitions, int num_grus, int num_timesteps_simulation, int num_timesteps_buffer);
+    ~Output_Container();
+    int findPartition(int local_gru_index);
+    int getNumPartitions();
+    // The output container needs to be restructured when rerunning the failed GRUs.
+    void reconstruct();
+    Output_Partition* getOutputPartition(int local_gru_index);
+    std::vector<int> getFailedGRUIndexList();
+};
--- a/build/includes/global/auxilary.hpp
+++ b/build/includes/global/auxilary.hpp
+#pragma once
+// Setters
+void set_flagVec(std::vector<int>& arr_i, void* handle);
+void set_var_i(std::vector<int>& arr_i, void* handle);
+void set_var_d(std::vector<double> &arr_d, void* handle);
+void set_var_i8(std::vector<long int>& arr_i, void* handle);
+void set_i8length(std::vector<long int> &arr_i8length, void* handle);
+void set_ilength(std::vector<int> &arr_ilength, void* handle);
+void set_dlength(std::vector<double> &arr_dlength, void* handle);
+void set_var_flagVec(std::vector<std::vector<int> > &mat, void* handle);
+void set_var_ilength(std::vector<std::vector<int> > &mat, void* handle);
+void set_var_i8length(std::vector<std::vector<long int> > &mat, void* handle);
+void set_var_dlength(std::vector<std::vector<double> > &mat, void *handle);
+// Getters
+std::vector<int> get_flagVec(void* handle);
+std::vector<int> get_var_i(void* handle);
+std::vector<double> get_var_d(void* handle);
+std::vector<long int> get_var_i8(void* handle);
+std::vector<long int> get_i8length(void* handle);
+std::vector<int> get_ilength(void* handle);
+std::vector<double> get_dlength(void* handle);
+std::vector<std::vector<int> > get_var_flagVec(void* handle);
+std::vector<std::vector<int> > get_var_ilength(void* handle);
+std::vector<std::vector<long int> > get_var_i8length(void* handle);
+std::vector<std::vector<double> > get_var_dlength(void* handle);
+std::vector<double> get_attr_struct(void* handle);
+std::vector<int> get_type_struct(void* handle);
+std::vector<std::vector<double>> get_mpar_struct_array(void* handle);
+std::vector<double> get_bpar_struct(void* handle);
\ No newline at end of file
--- a/build/source/interface/fortran_dataTypes.h
+++ b/build/source/interface/fortran_dataTypes.h
-#ifndef FORTRAN_DATATYPES_H_
+#pragma once
-#define FORTRAN_DATATYPES_H_
 extern "C" {
    // flagVec 
@@ -15,6 +14,8 @@ extern "C" {
    void  set_data_var_i(void* handle, const int* array, int size);
    void  get_size_data_var_i(void* handle, int* size);
    void  get_data_var_i(void* handle, int* array);
+    void  get_size_data_typeStruct(void* handle, int* size);
+    void  get_data_typeStruct(void* handle, int* array);
    // var_i8 
    void* new_handle_var_i8();
@@ -29,6 +30,10 @@ extern "C" {
    void  set_data_var_d(void* handle, const double* array, int size);
    void  get_size_data_var_d(void* handle, int* size);
    void  get_data_var_d(void* handle, double* array);
+    void  get_data_attrStruct(void* handle, double* array);
+    void  get_size_data_attrStruct(void* handle, int* size);
+    void  get_data_bparStruct(void* handle, double* array);
+    void  get_size_data_bparStruct(void* handle, int* size);
    // ilength
    void* new_handle_ilength();
@@ -51,6 +56,7 @@ extern "C" {
    void  get_size_data_dlength(void* handle, int* size);
    void  get_data_dlength(void* handle, double* array);
    // var_flagVec
    void* new_handle_var_flagVec();
    void  delete_handle_var_flagVec(void* handle);
@@ -82,6 +88,9 @@ extern "C" {
    void  get_size_var_dlength(void* handle, int* num_var);
    void  get_size_data_var_dlength(void* handle, int* num_var, int* num_dat);
    void  get_data_var_dlength(void* handle, double* array);
+    void  get_size_var_mparStruct(void* handle, int* num_var);
+    void  get_size_data_mparStruct(void* handle, int* num_var, int* num_dat);
+    void  get_data_mparStruct(void* handle, double* array);
    // var_dlength_array
    void* new_handle_dlength_array();
@@ -97,6 +106,12 @@ extern "C" {
    void* new_handle_file_info();
    void delete_handle_file_info(void* handle);
-}
+    // zLookup
+    void* new_handle_z_lookup();
+    void* delete_handle_z_lookup(void* handle);
+    // hru_type
+    void* new_handle_hru_type();
+    void delete_handle_hru_type(void* handle);
-#endif
+}
\ No newline at end of file
--- a/build/includes/global/global.hpp
+++ b/build/includes/global/global.hpp
+#pragma once
+#include <chrono>
+/**
+ * Return the time between to time points
+ */
+double calculateTime(std::chrono::time_point<std::chrono::system_clock> start, 
+    std::chrono::time_point<std::chrono::system_clock> end);
+struct serializable_netcdf_gru_actor_info {
+    double run_time;
+    double init_duration;
+    double forcing_duration;
+    double run_physics_duration;
+    double write_output_duration;
+    int successful; // 0 = false, 1 = true
+    int num_attempts;
+    double rel_tol;
+    double abs_tol;
+};
+template<class Inspector>
+bool inspect(Inspector& f, serializable_netcdf_gru_actor_info& x) {
+    return f.object(x).fields(f.field("run_time", x.run_time),
+                              f.field("init_duration", x.init_duration),
+                              f.field("forcing_duration", x.forcing_duration),
+                              f.field("run_physics_duration", x.run_physics_duration),
+                              f.field("write_output_duration", x.write_output_duration),
+                              f.field("successful", x.successful),
+                              f.field("num_attempts", x.num_attempts),
+                              f.field("rel_tol", x.rel_tol),
+                              f.field("abs_tol", x.abs_tol));
+}
--- a/build/source/actors/json.hpp
+++ b/build/source/actors/json.hpp
--- a/build/includes/global/message_atoms.hpp
+++ b/build/includes/global/message_atoms.hpp
+#pragma once
+#include "batch.hpp"
+#include "batch_container.hpp"
+#include "client.hpp"
+#include "client_container.hpp"
+#include <vector>
+#include "settings_functions.hpp"
+#include "global.hpp"
+#include "caf/all.hpp"
+enum class hru_error : uint8_t {
+    run_physics_unhandleable = 1,
+    run_physics_infeasible_state = 2,
+};
+enum class file_access_error : uint8_t {
+    writing_error = 1,
+    unhandleable_error = 2,
+    mDecisions_error = 100,
+};
+// HRU Errors
+std::string to_string(hru_error err);
+bool from_string(caf::string_view in, hru_error& out);
+bool from_integer(uint8_t in, hru_error& out);
+template<class Inspector>
+bool inspect(Inspector& f, hru_error& x) {
+    return caf::default_enum_inspect(f, x);
+}
+// File Access Actor
+std::string to_string(file_access_error err);
+bool from_string(caf::string_view in, file_access_error& out);
+bool from_integer(uint8_t in, file_access_error& out);
+template<class Inspector>
+bool inspect(Inspector& f, file_access_error& x) {
+    return caf::default_enum_inspect(f, x);
+}
+CAF_BEGIN_TYPE_ID_BLOCK(summa, first_custom_type_id)
+    // Sender: job_actor 
+    // Reciever: summa_actor
+    // Summary: job_actor finished job
+    CAF_ADD_ATOM(summa, done_job)
+    // Sender: 
+    // Reciever: 
+    // Summary:
+    CAF_ADD_ATOM(summa, err)
+    CAF_ADD_ATOM(summa, err_atom)
+    // Sender:
+    // Reciever: 
+    // Summary:
+    CAF_ADD_ATOM(summa, init_gru)
+    // Sender:
+    // Reciever: 
+    // Summary:
+    CAF_ADD_ATOM(summa, done_init_gru)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, init_hru)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, done_init_hru)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, done_write)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, done_hru)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, run_failure)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, access_forcing)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, access_forcing_internal)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, write_output)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, deallocate_structures)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, write_param)
+        // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, restart_failures)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, get_attributes_params)
+    // Sender:
+    // Reciever:
+    // Summary:    
+    CAF_ADD_ATOM(summa, run_hru)
+    // Sender:
+    // Reciever:
+    // Summary:    
+    CAF_ADD_ATOM(summa, start_hru)
+    // Sender:
+    // Reciever:
+    // Summary:    
+    CAF_ADD_ATOM(summa, dt_init_factor)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, connect_to_server)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, connect_as_backup)
+    // Sender:
+    // Reciever:
+    // Summary:    
+    CAF_ADD_ATOM(summa, compute_batch)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, done_batch)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, time_to_exit)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, update_with_current_state)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, new_assigned_batch)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, no_more_batches)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, update_backup_server_list)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, client_removed)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, new_client)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, is_lead_server)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, yes)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, no)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, new_forcing_file)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, num_steps_before_write)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, get_num_output_steps)
+    // Sender:
+    // Reciever:
+    // Summary:
+    CAF_ADD_ATOM(summa, finalize)
+    // Struct Types
+    CAF_ADD_TYPE_ID(summa, (Distributed_Settings))
+    CAF_ADD_TYPE_ID(summa, (Summa_Actor_Settings))
+    CAF_ADD_TYPE_ID(summa, (File_Access_Actor_Settings))
+    CAF_ADD_TYPE_ID(summa, (Job_Actor_Settings))
+    CAF_ADD_TYPE_ID(summa, (HRU_Actor_Settings))
+    CAF_ADD_TYPE_ID(summa, (serializable_netcdf_gru_actor_info))
+    // Class Types
+    CAF_ADD_TYPE_ID(summa, (Client))
+    CAF_ADD_TYPE_ID(summa, (Client_Container))
+    CAF_ADD_TYPE_ID(summa, (Batch))
+    CAF_ADD_TYPE_ID(summa, (Batch_Container))
+    CAF_ADD_TYPE_ID(summa, (std::vector<std::vector<double>>))
+    CAF_ADD_TYPE_ID(summa, (std::vector<std::vector<int>>))
+    CAF_ADD_TYPE_ID(summa, (std::vector<int>))
+    CAF_ADD_TYPE_ID(summa, (std::vector<double>))
+    CAF_ADD_TYPE_ID(summa, (std::vector<long int>))
+    CAF_ADD_TYPE_ID(summa, (std::vector<std::tuple<caf::actor, std::string>>))
+    CAF_ADD_TYPE_ID(summa, (std::vector<serializable_netcdf_gru_actor_info>))
+    // GRU Parameter/Attribute Vectors
+    CAF_ADD_TYPE_ID(summa, (std::tuple<std::vector<double>, 
+                                       std::vector<int>, 
+                                       std::vector<long int>, 
+                                       std::vector<double>, 
+                                       std::vector<double>, 
+                                       std::vector<std::vector<double>>>))
+    // File_Access_Actor Read/Write times
+    CAF_ADD_TYPE_ID(summa, (std::tuple<double, double>))
+    CAF_ADD_TYPE_ID(summa, (std::optional<caf::strong_actor_ptr>))
+    // error types
+    CAF_ADD_TYPE_ID(summa, (hru_error))
+    CAF_ADD_TYPE_ID(summa, (file_access_error))
+CAF_END_TYPE_ID_BLOCK(summa)
+CAF_ERROR_CODE_ENUM(hru_error)
+CAF_ERROR_CODE_ENUM(file_access_error)
\ No newline at end of file
--- a/build/includes/global/settings_functions.hpp
+++ b/build/includes/global/settings_functions.hpp
+#pragma once
+#include <string>
+#include <vector>
+#include <thread>
+#include <optional>
+#include "json.hpp"
+#include <bits/stdc++.h>
+#include <sys/stat.h>
+using json = nlohmann::json;
+struct File_Access_Actor_Settings;
+struct Job_Actor_Settings;
+struct HRU_Actor_Settings;
+// ####################################################################
+//                    Distributed Settings
+// ####################################################################
+struct Distributed_Settings;
+struct Distributed_Settings {
+    bool distributed_mode;  // flag for starting summa in distributed mode
+    std::vector<std::string> servers_list;   // the hostname of the server actor
+    int port;               // the port number of the server actor
+    int total_hru_count;
+    int num_hru_per_batch;
+};
+template<class Inspector>
+bool inspect(Inspector& inspector, Distributed_Settings& distributed_settings) {
+    return inspector.object(distributed_settings).fields(
+                inspector.field("distributed_mode", distributed_settings.distributed_mode),
+                inspector.field("servers_list",     distributed_settings.servers_list),
+                inspector.field("port",             distributed_settings.port),
+                inspector.field("total_hru_count",  distributed_settings.total_hru_count),
+                inspector.field("num_hru_per_batch",distributed_settings.num_hru_per_batch));
+}
+Distributed_Settings readDistributedSettings(std::string json_settings_file);
+// ####################################################################
+//                     SUMMA Actor Settings
+// ####################################################################
+struct Summa_Actor_Settings;
+struct Summa_Actor_Settings {
+    int max_gru_per_job;
+};
+template<class Inspector>
+bool inspect(Inspector& inspector, Summa_Actor_Settings& summa_actor_settings) {
+    return inspector.object(summa_actor_settings).fields(
+                inspector.field("max_gru_per_job",  summa_actor_settings.max_gru_per_job));  
+}
+Summa_Actor_Settings readSummaActorSettings(std::string json_settings_file);
+// ####################################################################
+//                      File Access Actor Settings
+// ####################################################################
+struct File_Access_Actor_Settings {
+    int num_partitions_in_output_buffer;
+    int num_timesteps_in_output_buffer;
+};
+template<class Inspector>
+bool inspect(Inspector& inspector, File_Access_Actor_Settings& file_access_actor_settings) {
+    return inspector.object(file_access_actor_settings).fields(
+                inspector.field("num_partitions_in_output_buffer", 
+                    file_access_actor_settings.num_partitions_in_output_buffer),
+                     inspector.field("num_timesteps_in_output_buffer", 
+                    file_access_actor_settings.num_timesteps_in_output_buffer));
+}
+File_Access_Actor_Settings readFileAccessActorSettings(std::string json_settings_file);
+// ####################################################################
+//                          Job Actor Settings
+// ####################################################################
+struct Job_Actor_Settings {
+    std::string file_manager_path;
+    int max_run_attempts; // maximum number of times to attempt to run each HRU in a job
+};
+template<class Inspector>
+bool inspect(Inspector& inspector, Job_Actor_Settings& job_actor_settings) {
+    return inspector.object(job_actor_settings).fields(
+                inspector.field("file_manager_path", job_actor_settings.file_manager_path),
+                inspector.field("max_run_attempts",  job_actor_settings.max_run_attempts));
+}
+Job_Actor_Settings readJobActorSettings(std::string json_settings_file);
+// ####################################################################
+//                          HRU Actor Settings
+// ####################################################################
+struct HRU_Actor_Settings {
+    bool print_output;
+    int output_frequency;
+    int dt_init_factor; // factor to multiply the initial timestep by
+    double rel_tol;
+    double abs_tol;
+    double relTolTempCas;
+    double absTolTempCas;
+    double relTolTempVeg;
+    double absTolTempVeg;
+    double relTolWatVeg;
+    double absTolWatVeg;
+    double relTolTempSoilSnow;
+    double absTolTempSoilSnow;
+    double relTolWatSnow;
+    double absTolWatSnow;
+    double relTolMatric;
+    double absTolMatric;
+    double relTolAquifr;
+    double absTolAquifr;
+};
+template<class Inspector>
+bool inspect(Inspector& inspector, HRU_Actor_Settings& hru_actor_settings) {
+    return inspector.object(hru_actor_settings).fields(
+                inspector.field("print_output",     hru_actor_settings.print_output),
+                inspector.field("output_frequency", hru_actor_settings.output_frequency),
+                inspector.field("dt_init_factor",   hru_actor_settings.dt_init_factor),
+                inspector.field("rel_tol",          hru_actor_settings.rel_tol),
+                inspector.field("abs_tol",          hru_actor_settings.abs_tol));
+}
+HRU_Actor_Settings readHRUActorSettings(std::string json_settings_file);
+// ####################################################################
+//                      Non Actor Specific Settings
+// ####################################################################
+int checkFileExists(std::string file_path);
+/**
+ * @brief Get the Settings from json
+ * Template function that can be used with retrieving any singular type from the settings file
+ */
+template <typename T>
+std::optional<T> getSettings(std::string json_settings_file, std::string key_1, std::string key_2, 
+    T return_value) {
+    json settings;
+    std::ifstream settings_file(json_settings_file);
+    if (!settings_file.good()) return {};
+    settings_file >> settings;
+    settings_file.close();
+    // find first key
+    try {
+        if (settings.find(key_1) != settings.end()) {
+            json key_1_settings = settings[key_1];
+            // find value behind second key
+            if (key_1_settings.find(key_2) != key_1_settings.end()) {
+                return key_1_settings[key_2];
+            } else 
+                return {};
+        } else {
+            return {}; // return none in the optional (error value)
+        }
+    } catch (json::exception& e) {
+        std::cout << e.what() << "\n";
+        std::cout << key_1 << "\n";
+        std::cout << key_2 << "\n";
+        return {};
+    }
+}
+// Get settings from settings file in array format
+std::optional<std::vector<std::string>> getSettingsArray(std::string json_settings_file, std::string key_1, std::string key_2);
+// Check the settings - Print them out to stdout
+void check_settings_from_json(Distributed_Settings &distributed_settings, 
+    Summa_Actor_Settings &summa_actor_settings, File_Access_Actor_Settings &file_access_actor_settings, 
+    Job_Actor_Settings &job_actor_settings, HRU_Actor_Settings &hru_actor_settings);
+// Output a default configuration file
+void generate_config_file();
\ No newline at end of file
--- a/build/includes/global/timing_info.hpp
+++ b/build/includes/global/timing_info.hpp
+#pragma once
+#include <chrono>
+#include <optional>
+#include <vector>
+#include <string>
+using chrono_time = std::chrono::time_point<std::chrono::system_clock>;
+/**
+ * Class to manage timing information. This allows the user to add an arbitrary amount of timing variables.
+ * The timing variables are accessed through their named string and will keep a running duration of the amount 
+ * of time spent through multiple calls to updateStartPoint and updateEndPoint
+ */
+class TimingInfo {
+    private:
+        std::vector<std::optional<chrono_time>> start;
+        std::vector<std::optional<chrono_time>> end;
+        std::vector<double> duration;
+        std::vector<std::string> name_of_time_point; // the name you want for the time point (ie. reading, writing, duration)
+        int num_time_points;
+        std::optional<double> calculateDuration(int index);
+        std::optional<int>  getIndex(std::string time_point_name);
+    public:
+        TimingInfo();
+        ~TimingInfo();
+        void addTimePoint(std::string time_point_name);
+        void updateStartPoint(std::string time_point_name);
+        void updateEndPoint(std::string time_point_name);
+        std::optional<double> getDuration(std::string time_point_name); // returns duration in seconds
+};
\ No newline at end of file
--- a/build/includes/gru_actor/gru_actor.hpp
+++ b/build/includes/gru_actor/gru_actor.hpp
+#pragma once
+#include "caf/all.hpp"
+#include "fortran_data_types.hpp"
+#include "timing_info.hpp"
+#include "global.hpp"
+#include <vector>
+#include <array>
+#include <chrono>
+#include <string>
+#include "var_lookup.hpp"
+namespace caf{
+struct gru_state {
+    // GRU Initalization
+    int ref_gru;
+    int indx_gru;
+    std::string config_path;
+    caf::actor file_access_actor;
+    int output_struc_size;
+    caf::actor parent;
+    // HRU information
+    std::vector<caf::actor> hru_list;
+    int num_hrus;
+    int hrus_complete = 0;
+    // Global Data
+    int nTimeDelay = 2000; // number of hours in the time delay histogram (default: ~1 season = 24*365/4)
+    struct iLookVarType var_type_lookup;
+    // Data Structure local to the GRU
+    int num_bpar_vars;                               // number of variables in the fortran structure for bpar_struct
+    std::vector<double> bpar_struct;   
+    std::vector<int> bpar_struct_var_type_list;  // The types to the variable at each element in bpar_struct
+    int num_bvar_vars;                               // number of variables in the fortran structure for bvar_struct
+    std::vector<std::vector<double>> bvar_struct;
+    std::vector<int> bvar_struct_var_type_list;
+    int num_var_types;
+    std::vector<int> i_look_var_type_list; // The types to the variable at each element in bvar_struct
+};
+behavior gru_actor(stateful_actor<gru_state>* self, 
+    int ref_gru, 
+    int indx_gru, 
+    std::string config_path, 
+    caf::actor file_access_actor, 
+    caf::actor parent);
+}
\ No newline at end of file
--- a/build/includes/gru_actor/gru_actor_subroutine_wrappers.hpp
+++ b/build/includes/gru_actor/gru_actor_subroutine_wrappers.hpp
+#pragma once
+extern "C" {
+    void getVarSizes(int* num_var_types, int* num_bpar_vars, int* num_bvar_vars);
+    void initVarType(void* var_type_lookup);
+    void fillVarTypeLists( int* num_bpar_vars, int* num_bvar_vars,
+        void* bpar_struct_var_type_list, void* bvar_struct_var_type_list, int* err);
+    int getNumHRU(int* indx_gru);
+}
\ No newline at end of file
--- a/build/includes/gru_actor/gru_data_structure.hpp
+++ b/build/includes/gru_actor/gru_data_structure.hpp
+#pragma once
--- a/build/includes/hru_actor/hru_actor.hpp
+++ b/build/includes/hru_actor/hru_actor.hpp
+#pragma once
+#include "caf/all.hpp"
+#include "fortran_data_types.hpp"
+#include "auxilary.hpp"
+#include "timing_info.hpp"
+#include "settings_functions.hpp"
+#include <string>
+#include "message_atoms.hpp"
+#include "global.hpp"
+/*********************************************
+ * HRU Actor Fortran Functions
+ *********************************************/
+extern "C" {
+  // Initialize HRU data_structures
+    void initHRU(int* indxGRU, int* num_steps, void* hru_data, int* err);
+    void setupHRUParam( int* indxGRU, int* indxHRU, void* hru_data, double* upArea, int* err);
+    // Setup summa_readRestart File if this option has been chosen 
+    void summa_readRestart(int* indxGRU, int* indxHRU, void* hru_data, double* dtInit, int* err);
+    // Run the model for one timestep
+    void RunPhysics(int* id, int* stepIndex, void* hru_data, double* dt, int* dt_int_factor, int* err);
+    void hru_writeOutput(int* index_hru, int* index_gru, int* timestep, int* output_step, void* hru_data, int* err);
+    void setTimeZoneOffset(int* iFile, void* hru_data, int* err);
+    void HRU_readForcing(int* index_gru, int* iStep, int* iRead, int* iFile, void* hru_data,  int* err);
+    void get_sundials_tolerances(void* hru_data, double* relTol, double* absTol);
+    void set_sundials_tolerances(void* hru_data, double* relTol, double* absTol);
+    void setIDATolerances(void* hru_data, double* relTolTempCas, double* absTolTempCas, double* relTolTempVeg,
+                          double* absTolTempVeg, double* relTolWatVeg, double* absTolWatVeg, double* relTolTempSoilSnow,
+                          double* absTolTempSoilSnow, double* relTolWatSnow, double* absTolWatSnow, double* relTolMatric,
+                          double* absTolMatric, double* relTolAquifr, double* absTolAquifr);
+}
+/*********************************************
+ * HRU Actor state variables
+ *********************************************/
+namespace caf {
+struct hru_state {
+	// Actor References
+	caf::actor file_access_actor;
+	caf::actor parent;
+    // Info about which HRU we are and our indexes
+    // into global structures in Fortran
+    int indxHRU; 	    // index for hru part of derived types in FORTRAN
+    int indxGRU; 		// index for gru part of derived types in FORTRAN
+    int refGRU;			// The actual ID of the GRU we are
+    // Variables for forcing structures
+	int stepsInCurrentFFile;        // number of time steps in current forcing file
+    int num_steps_until_write;      // number of time steps until we pause for FA_Actor to write
+    // HRU data structures (formerly summa_type)
+    void *hru_data = new_handle_hru_type();
+    // Misc Variables
+	int     timestep = 1;	    // Current Timestep of HRU simulation
+    int     forcingStep = 1;    // index of current time step in current forcing file
+    int     num_steps = 0;      // number of time steps
+    int     iFile = 1;              // index of current forcing file from forcing file list
+    int     dt_init_factor = 1; // factor of dt_init (coupled_em)
+    int     output_structure_step_index = 1; // index of current time step in output structure
+    double  dt_init;            // used to initialize the length of the sub-step for each HRU
+    double	upArea;             // area upslope of each HRU
+    int     err = 0;	
+    // Sundials variables
+    double rtol = -9999; // -9999 uses default
+    double atol = -9999; // -9999 uses default		        
+    // Settings
+    HRU_Actor_Settings hru_actor_settings;
+    ~hru_state() {
+        delete_handle_hru_type(hru_data);
+    }
+};
+behavior hru_actor(stateful_actor<hru_state>* self, int refGRU, int indxGRU,
+    HRU_Actor_Settings hru_actor_settings, caf::actor file_access_actor, 
+    caf::actor parent);
+/*********************************************
+ * Functions for the HRU Actor
+ *********************************************/
+/** Function to initalize the HRU for running */
+void Initialize_HRU(stateful_actor<hru_state>* self);
+/** Function runs all of the hru time_steps */
+int Run_HRU(stateful_actor<hru_state>* self);
+}
\ No newline at end of file
No results found