Model Parameters

Model Parameters#

Overview#

The Pywr-DRB model is requires the definition of “parameters” which are able to define values, calculate values, and establish relationships between nodes and other paramters. These are critical to the implementation of the Flexible Flow Management operations policies and every other part of the model simulation.

This notebook provides an introduction to these parameter, and how they are defined and used in the model. This notebook focuses on the pywrdrb.STARFITReservoirRelease parameter as an example.

Content:#

  • Pywr parameters

    • The role of parameters

    • Basic parameters

    • Custom parameters

  • Pywr-DRB parameters

    • A quick tangent: pywrdrb.make_model()

    • Basic parameters in Pywr-DRB

    • Custom parameters in Pywr-DRB

  • Activity: Create a custom parameter

1.0 Pywr parameters#

The term “parameter” is used to describe many different things in different modeling contexts. In the context of pywr, Parameters are one of the three major objects used to define a model:

  • Nodes

  • Edges

  • Parameters

1.1. The role of parameters#

Parameters are a critical part of a pywr model. Parameters are used to enforce the “rules” in the model.

In a linear programming problem, the goal is to find a vector \(x\) which maximizes the objective function such that the constraints are not violated:

\(\text{Maximize } c^Tx\)

\(\text{s.t. } Ax \leq b \,\, \text{and} \,\, x \geq 0\)

For a water resource model, like Pywr-DRB, the decision variable \(x\) corresponds to the reservoir release for a specific day. In this context, the parameters are used to calculate, set, and update the constraints which consist of \(A\) and \(b\) in this formulation. These constraints correspond to minimum and maximum release requirements.

As you will see, we connect specific parameters to different nodes or to other parameters. This is necessary when you have dependencies in the model.

1.2 Basic parameters#

Pywr allows for both premade and custom parameters to be used in a model. The “basic” parameters are helpful for a lot of the simple operations we want to implement, but we will discuss the importance of custom parameters below.

See the full list of pywr parameters HERE.

Here, I highlight just a few basic parameters including:

  • ConstantParameter

  • AggregatedParameter

  • DataFrameParameter

ConstantParameter#

This is a simple parameter, which is used to provide a single constant value during each step.

AggregatedParameter#

From the pywr.parameters.AggregatedParameter API: It’s value is the value of it’s child parameters aggregated using a aggregating function (e.g. sum).”

This class takes two inputs:

  • parameters: The parameters to aggregate

  • agg_func: The aggregation function. Must be one of {“sum”, “min”, “max”, “mean”, “product”}, or a callable function which accepts a list of values.

DataFrameParameter#

This is designed to let you access CSV data during the simulation using pd.DataFrame style syntax.

This example from the documentation shows how it works.

{"parameters": [
    {
        "name": "catchment_inflow",
        "type": "dataframe",
        "url": "data/catchmod_outputs_v2.csv",
        "column": "Flow",
        "index_col": "Date",
        "parse_dates": true
    }
]}

1.3 Custom parameters#

“Custom parameters can be used to model specific behaviors otherwise not possible with the existing parameters. To write a custom parameter you must (as a minimum) inherit from the base pywr.parameter.Parameter class and implement the Parameter.value method. The arguments to this method represent the current timestep and scenario allowing the value of the parameter to be dynamic.”

Each custom parameter should include the following methods:

  • Parameter.__init__()

    • This will be run once when setting up the parameter

  • Parameter.value()

    • This is necessary, and will be called once per timestep. The output of this method is the output of the parameter each timestep.

  • Parameter.load()

    • This special @classmethod is used to help create the parameter, and will establish connections between the parameter and each of the other nodes and parameters which influence the value().

Here is an example provided in the Pwyr documentation, which just returns a constant value each timestep, and contains each of the methods described above.

from pywr.parameters import Parameter

class MyParameter(Parameter):
    def __init__(self, model, value, **kwargs):
        # called once when the parameter is created
        super().__init__(model, **kwargs)
        self._value = value

    def value(self, timestep, scenario_index):
        # called once per timestep for each scenario
        return self._value

    @classmethod
    def load(cls, model, data):
        # called when the parameter is loaded from a JSON document
        value = data.pop("value")
        return cls(model, value, **data)

MyParameter.register()  # register the name so it can be loaded from JSON

The Parameter.value() function is the most important! Pywr is going to ask for the value() during each timestep, and it must get something in response.

2.0 Pywr-DRB parameters#

2.1 A quick tangent: pywrdrb.make_model()#

Recall from Pywr-DRB Tutorial 1 that the file drb_model_full_<input type>.json contains all of the structural information defining the model.

Essentially, this is a dictionary containing lists of nodes, edges, and parameters. Together, this information is used by Pywr to construct the linear program which is used to simulate operations.

The structure of this file will look something like this:

{
    "metadata": {},
    "timestepper": {},
    "solver": {},
    "nodes": {},
    "edges": {},
    "parameters": {}
}

When working with Pywr-DRB, we use a specific function called make_model() to generate this JSON.

Future tutorials will focus on the details of the make_model() process, but it is important to point out in this tutorial since the parameters are used in this process.

For now, go and take a look in the file: Pywr-DRB/pywrdrb/make_model.py.

There are two main functions to point out:

  • add_major_node()

  • make_model()

2.2 Basic parameters in Pywr-DRB#

We use the constant parameter to define the maximum reservoir volume constraint in make_model.add_major_node(), like this:

### max volume of reservoir, from GRanD database except where adjusted from other sources (eg NYC)
if node_type == 'reservoir':
	model['parameters'][f'max_volume_{name}'] = {
		'type': 'constant',
		'url': 'drb_model_istarf_conus.csv',
		'column': 'Adjusted_CAP_MG',
		'index_col': 'reservoir',
		'index': f'{name}'
	}

2.3 Custom parameters in Pywr-DRB#

Go look in the folder Pywr-DRB/pywrdrb/parameters… This folder contains the scripts which define the custom parameters used in our model.

These include:

  • starfit.py

    • This contains the STARFITReservoirRelease parameter which is used to determine the daily release volume at each of the STARFIT reservoirs. This is discussed in more detail below.

  • ffmp.py

    • This contains many different parameters which are used to determine NYC reservoir releases using the FFMP rules.

  • lower_basin_ffmp.py

    • This contains a few parameters which are used to determine when and how much the lower basin reservoirs should contribute to the Montague or Trenton flow targets.

  • general.py

    • This contains the LaggedReservoirRelease parameter. Pywr doesn’t have a parameter to return a previous (>1 timesteps) node flow or parameter value. But we can calculate release for N timesteps ago based on rolling avg parameters for N & (N-1) timesteps.

  • inflow_ensemble.py

    • When running multiple scenarios at once, we use a custom parameter called FlowEnsemble which provides access to inflow ensemble timeseries during the simulation period.

2.3.1 STARFITReservoirRelease#

This custom parameter is used to calculate the target reservoir release for each day of the simulation, for non-NYC reservoirs.

If you are unfamiliar with the STARFIT reservoir operations functions, start by reviewing the following publication:

For each reservoir, we have a set of values which define the functions used to calculate the target release. There are multiple steps in the calculation. This workflow is shown in Figure 6 from Turner et al. (2021):

Figure 6 from Turner et al. (2021)

As depicted in the figure, we need to perform several calculations to determine the target release including:

  • Normal operating range (NOR) upper and lower bounds

  • Seasonal release harmonic

  • Release adjustment

With this in mind, now look at the outline for the STARFITReservoirRelease below, and notice how this class has several internal functions design to help calculate the target release.

Again, the most important part of this is the output of the STARFITReservoirRelease.value() function. Pywr is not interested in the value of the get_NORhi() or any other supplemental functions. These are just necessary to help calculate the value().

class STARFITReservoirRelease(Parameter):
    """
    Args:
        model (dict): The PywrDRB model.
        storage_node (str): The storage node associated with the reservoir.
        flow_parameter: The PywrDRB catchment inflow parameter corresponding to the reservoir.
    """
    def __init__(self, model, storage_node, flow_parameter, **kwargs):
        super().__init__(model, **kwargs)
        # This is where we get all reservoir-specific parameters
        # from the drb_istarf_conus.csv data
        ...

    def standardize_inflow(self, inflow):
	    return (inflow - self.I_bar) / self.I_bar
	
    def calculate_percent_storage(self, storage):
        return (storage / self.S_cap)

	def get_NORhi(self, timestep):
		...
		return 

	def get_NORlo(self, timestep):
		...
		return

    def get_harmonic_release(self, timestep):
	    ...
	    return

    def calculate_release_adjustment(self, S_hat, I_hat,
                                     NORhi_t, NORlo_t):
        ...
        return

    def calculate_target_release(self, harmonic_release, epsilon,
                                 NORhi, NORlo, S_hat, I):
		...
		return target

    def value(self, timestep, scenario_index):
		# Get current storage and inflow conditions
        I_t = self.inflow.get_value(scenario_index)
        S_t = self.node.volume[scenario_index.indices]
        
        I_hat_t = self.standardize_inflow(I_t)
        S_hat_t = self.calculate_percent_storage(S_t)
        
        NORhi_t = self.get_NORhi(timestep)
        NORlo_t = self.get_NORlo(timestep)
        
        seasonal_release_t = self.get_harmonic_release(timestep)
            
        # Get adjustment from seasonal release
        epsilon_t = self.calculate_release_adjustment(S_hat_t, 
                                                      I_hat_t, 
                                                      NORhi_t, NORlo_t)
        
        # Get target release
        target_release = self.calculate_target_release(S_hat = S_hat_t,
                                                    I = I_t,
                                                    NORhi=NORhi_t,
                                                    NORlo=NORlo_t,
                                                    epsilon=epsilon_t,
                                                    harmonic_release=seasonal_release_t)
    
        # Get actual release subject to constraints
        release_t = max(min(target_release, I_t + S_t), (I_t + S_t - self.S_cap)) 
	    return  max(0, release_t)

    @classmethod
    def load(cls, model, data):
        name = data.pop("node")
        storage_node = model.nodes[f'reservoir_{name}']
        flow_parameter = load_parameter(model, f'flow_{name}')
        return cls(model, storage_node, flow_parameter, **data)

# Register the parameter for use with Pywr
STARFITReservoirRelease.register()

We assign the custom STARFIT parameter to each of the STARFIT reservoir nodes (non-NYC reservoirs) in make_model.add_major_node(), like this:

### For STARFIT reservoirs, use custom parameter
if starfit_release:
	model['parameters'][f'starfit_release_{name}'] = {
		'type': 'STARFITReservoirRelease',
		'node': name
	}

2.3.2 FFMP and lower basin FFMP parameters#

The Flexible Flow Management Program (FFMP) is implemented using a set of custom parameters. These parameters are too complex to describe here, and would require their own informational guide.

Activity: Explore the parameters in Pywr-DRB#

Go back to Pywr-DRB Tutorial 01, and run the code up through section 3.3 Parameters.

You will find this code block:

### Read model parameter names into a list
model_parameters = [p for p in model.parameters if p.name]

print(f'There are {len(model_parameters)} parameters in the model')
# [output]: There are 394 parameters in the model

Now you have a list of 394 Parameter objects which are used in the model.

For this activity, spend some time exploring this list of parameters.

Some questions to answer:

  1. How many different types of parameters are in the model? What types?

  2. How frequently is each parameter type used?

As a bonus, you can consider exploring the dependencies between this parameters using NetworkX, using something like this:

import networkx as nx

param_graph = model_parameters[100].parents.graph

# plot
import matplotlib.pyplot as plt
plt.figure(figsize=(10,10))
nx.draw(param_graph, with_labels=False)
plt.show()

This code produces a graph, but it is not very informative… can you improve the graph to make it more helpful by doing things such as color coding the elements in the graph according to their type.