API Reference

DataValidationException

Exception related to data validation of the Tulipa Energy Model input data.

EnergyProblem

Structure to hold all parts of an energy problem. It is a wrapper around various other relevant structures. It hides the complexity behind the energy problem, making the usage more friendly, although more verbose.

Fields

• db_connection: A DuckDB connection to the input tables in the model.
• variables: A dictionary of TulipaVariables containing the variables of the model.
• expressions: A dictionary of TulipaExpressions containing the expressions of the model attached to tables.
• constraints: A dictionary of TulipaConstraints containing the constraints of the model.
• profiles: Holds the profiles per rep_period or over_clustered_year in dictionary format. See ProfileLookup.
• model_parameters: A ModelParameters structure to store all the parameters that are exclusive of the model.
• model: A JuMP.Model object representing the optimization model.
• solved: A boolean indicating whether the model has been solved or not.
• objective_value: The objective value of the solved problem (Float64).
• termination_status: The termination status of the optimization model.

Constructor

• EnergyProblem(connection): Constructs a new EnergyProblem object with the given connection.

The constraints_partitions field is computed from the representative_periods, and the other fields are initialized with default values.

See the basic example tutorial to see how these can be used.

ModelParameters(;key = value, ...)
ModelParameters(path; ...)
ModelParameters(connection; ...)
ModelParameters(connection, path; ...)

Structure to hold the model parameters. Some values are defined by default and some required explicit definition.

If path is passed, it is expected to be a string pointing to a TOML file with a key = value list of parameters. Explicit keyword arguments take precedence.

If connection is passed, the default discount_year is set to the minimum of all milestone years. In other words, we check for the table year_data for the column year where the column is_milestone is true. Explicit keyword arguments take precedence.

If both are passed, then path has preference. Explicit keyword arguments take precedence.

Parameters

• discount_rate::Float64 = 0.0: The model discount rate.
• discount_year::Int: The model discount year.
• power_system_base::Float64 = 100.0: The power system base in MVA.

Structure to hold the dictionaries of profiles.

Structure to hold the JuMP constraints for the TulipaEnergyModel

Structure to hold some JuMP expressions that are not attached to constraints but are attached to a table.

TulipaTabularIndex

Abstract structure for TulipaVariable, TulipaConstraint and TulipaExpression. All deriving types must satisfy:

• Have fields
  • indices::DuckDB.QueryResult
  • table_name::String

Structure to hold the JuMP variables for the TulipaEnergyModel

_append_variable_ids(
    connection,
    constraint_table_name,
    variables_to_append,
)

Create table containing all rows of the given constraint (constraint_table_name) and their matching variable ids of the variables in variables_to_append

_check_if_table_exists(connection, table_name)

Check if table table_name exists in the connection.

_create_group_table_if_not_exist!(
    connection,
    table_name,
    grouped_table_name,
    group_by_columns,
    array_agg_columns;
    rename_columns = Dict(),
    order_agg_by = "id",
)

Create a grouped table grouping the table_name into the grouped_table_name. The group_by_columns are the columns that are used in the group by (e.g., asset, year, repperiod), and the `arrayaggcolumns` are the columns that are aggregated into arrays (e.g., id, timeblockstart, timeblock_end).

It is expected that the original table has an id column, which is used in the ordering of the array_agg_columns. Otherwise, please pass the argument order_agg_by with the column that should be used for this ordering.

If one of the columns has to be renamed, use the rename_columns dictionary.

_create_variables_from_indices!(
model,
variables,
name,
keys_from_row;
lower_bound_from_row = row -> -Inf,
upper_bound_from_row = row -> Inf,
integer_from_row = row -> false,

)

This function creates variables by iterating over the variable indices, where each variable can have different properties determined by the index/row data.

_create_variables_from_specifications!(model, variables, specifications)

Creates variables based on a dictionary of specifications. Each specification should contain keysfromrow, lowerboundfromrow, upperboundfromrow, and integerfromrow functions.

_get_decommission_variable_specifications()

Returns a dictionary containing specifications for all decommission variables. Each specification includes the keys extraction function, bounds functions, and integer constraint function.

_get_investment_variable_specifications()

Returns a dictionary containing specifications for all investment variables. Each specification includes the keys extraction function, bounds functions, and integer constraint function.

_profile_aggregate(profiles, tuple_key, time_block, agg_functions, default_value)

Aggregates the profiles[tuple_key] over the time_block using the agg_function function. If the profile does not exist, uses default_value instead of each profile value.

profiles should be a dictionary of profiles, and tuple_key should be either (profile_name, year, rep_period) for the profiles of representative periods or (profile_name, year) for the profiles over clustered years.

If profiles[tuple_key] exists, then this function computes the aggregation of V = profiles[tuple_key] over the range time_block using the aggregator agg_function, i.e., agg_function(V[time_block]). If it does not exist, then V[time_block] is substituted by a vector of the corresponding size and default_value.

create_str, select_str = _sql_arguments_for_defaults(connection, table_name, table_schema)

Returns the strings to complement the table creation informing the name, type and possible default, and the select with coalescing and type casting.

The table_schema should be the complete schema, not only the type. For instance, TulipaEnergyModel.schema[table_name].

For each column of the table_name table, the creation string is like

COLUMN_NAME COLUMN_TYPE
-- For a column with no default type

or

COLUMN_NAME COLUMN_TYPE DEFAULT COLUMN_DEFAULT
-- For a column with default type

and the selection string is like

COLUMN_NAME::COLUMN_TYPE
-- For an existing column with no default type

or

COALESCE(COLUMN_NAME::COLUMN_TYPE, COLUMN_DEFAULT) AS COLUMN_NAME
-- For an existing column with default type

or

COLUMN_DEFAULT::COLUMN_TYPE AS COLUMN_NAME
-- For a non-existing column, just use the default type

The name, type, and defaults are based on the existing table, but overridden by the table_schema.

add_capacity_constraints!(connection, model, expressions, constraints, profiles)

Adds the capacity constraints for all asset types to the model

add_capacity_outgoing_semi_compact_method_constraints!(connection, model, expressions, constraints, profiles)

Adds the capacity constraints for the semi-compact investment method.

add_consumer_constraints!(connection, model, constraints, profiles)

Adds the consumer asset constraints to the model.

add_conversion_constraints!(connection, model, constraints)

Adds the conversion asset constraints to the model.

add_dc_power_flow_constraints!(connection, model, variables, constraints, model_parameters)

Adds the dc power flow constraints to the model.

add_decommission_variables!(model, variables)

Adds decommission variables to the optimization model, and sets bounds on selected variables based on the input data.

add_energy_constraints!(connection, model, constraints, profiles)

Adds the energy constraints for assets within the period blocks of the timeframe (overclusteredyear) to the model.

add_expression_terms_over_clustered_year_constraints!(
    connection,
    cons,
    flow,
    profiles;
    is_storage_level = false,
)

Computes the incoming and outgoing expressions per row of df_inter for the constraints that are between (inter) the representative periods.

This function is only used internally in the model.

add_expression_terms_rep_period_constraints!(
    connection,
    cons::TulipaConstraint,
    flow::TulipaVariable,
    workspace;
    use_highest_resolution = true,
    multiply_by_duration = true,
    add_min_outgoing_flow_duration = false,
    multiply_by_capacity_coefficient = false,
    include_commission_year = false,
)

Computes the incoming and outgoing expressions per row of cons for the constraints that are within the representative periods.

Includecommissionyear is only used for the constraints regarding the flows for the semi-compact investment method. If true, the expression will include the commission year of the flows. If false (the default), it will only use the year (i.e., milestone_year).

This function is only used internally in the model.

This strategy is based on the replies in this discourse thread:

• https://discourse.julialang.org/t/help-improving-the-speed-of-a-dataframes-operation/107615/23

Implementation

This expression computation uses a workspace to store all variables defined for each timestep. The idea of this algorithm is to append all variables defined at time timestep in workspace[timestep] and then aggregate then for the constraint time block.

The algorithm works like this:

1. Loop over each group of (asset, year, rep_period)
1.1. Loop over each variable in the group: (var_id, var_time_block_start, var_time_block_end)
1.1.1. Loop over each timestep in var_time_block_start:var_time_block_end
1.1.1.1. Compute the coefficient of the variable based on the rep_period resolution and the variable efficiency
1.1.1.2. Store (var_id, coefficient) in workspace[timestep]
1.2. Loop over each constraint in the group: (cons_id, cons_time_block_start, cons_time_block_end)
1.2.1. Aggregate all variables in workspace[timestep] for timestep in the time block to create a list of variable ids and their coefficients [(var_id1, coef1), ...]
1.2.2. Compute the expression using the variable container, the ids and coefficients

Note:

• On step 1.2.1, the aggregation can be either by uniqueness or not, i.e., if the variable happens in more that one workspace[timestep], should we add up the coefficients or not. This is defined by the keyword multiply_by_duration

add_flow_variables!(connection, model, variables)

Adds flow variables to the optimization model based on data from the variables. The flow variables are created using the @variable macro for each row in the :flow table.

add_flows_relationships_constraints!(connection, model, variables, constraints)

Adds the flows relationships constraints to the model.

add_group_constraints!(connection, model, variables, constraints)

Adds group constraints for assets that share a common limits or bounds.

add_hub_constraints!(model, constraints)

Adds the hub asset constraints to the model.

add_investment_variables!(model, variables)

Adds investment variables to the optimization model, and sets bounds on selected variables based on the input data.

add_limit_decommission_compact_method_constraints!(connection, model, expressions, constraints)

Adds the lower bound for the available capacity of decommissionable assets for the compact investment method. This is used to give a upper bound for the decommission variable.

add_power_flow_variables!(model, variables)

Adds power flow variables to the optimization model based on the :electricity_angle indices.

add_ramping_and_unit_commitment_constraints!(
    connection,
    model,
    variables,
    expressions,
    constraints,
    profiles
)

Adds the ramping constraints for producer and conversion assets where ramping = true in assets_data

add_shut_down_upper_bound_constraints!(model, constraints)

Adds the shut down constraints to the model.

add_start_up_and_shut_down_variables!(model, variables)

Adds 3var UC variables to the optimization model based on the :start_up and :shut_down indices. Additionally, variables are constrained to be integers based on the unit_commitment_integer property.

add_start_up_upper_bound_constraints!(model, constraints)

Adds the start up upper bound constraints to the model.

add_storage_constraints!(connection, model, variables, expressions, constraints, profiles)

Adds the storage asset constraints to the model.

add_storage_variables!(connection, model, variables)

Adds storage-related variables to the optimization model, including storage levels for both within rep-period and over-clustered-year, as well as charging state variables. The function also optionally sets binary constraints for certain charging variables based on storage methods.

add_transport_constraints!(connection, model, variables, expressions, constraints, profiles)

Adds the transport flow constraints to the model.

add_uc_logic_constraints!(model, constraints)

Adds the unit commitment logic constraint (i.e., start up - shut down = units_on difference) to the model.

add_unit_commitment_variables!(model, variables)

Adds unit commitment variables to the optimization model based on the :units_on indices. Additionally, variables are constrained to be integers based on the unit_commitment_integer property.

add_vintage_flow_sum_constraints!(connection, model, variables, constraints)

Adds the vintage flow sum constraints to the model.

attach_coefficient!(cons, name, container)

Attach a coefficient named name stored in container. This checks that the container length matches the stored indices number of rows.

attach_constraint!(model, cons, name, container)

Attach a constraint named name stored in container, and set model[name] = container. This checks that the container length matches the stored indices number of rows.

attach_expression!(cons_or_expr, name, container)
attach_expression!(model, cons_or_expr, name, container)

Attach a expression named name stored in container, and optionally set model[name] = container. This checks that the container length matches the stored indices number of rows.

attach_expression_on_constraints_grouping_variables!(
    connection,
    constraint,
    variable,
    expr_name;
    agg_strategy
)

Computes the intersection of the constraint and variable grouping by (:asset, :year, :rep_period).

The intersection is made on time blocks, so both the constraint table and the variable table must have columns id, timeblockstart and timeblockend.

The variable expr_name will be the name of the attached expression.

The agg_strategy must be either :sum, :mean, and :unique_sum, indicating how to aggregate the variables for a given constraint time block.

Implementation

The expression uses a workspace to store all variables defined for each timestep. The idea of this algorithm is to append all variables defined at time timestep in workspace[timestep] and then aggregate then for the constraint time block.

The algorithm works like this:

1. Loop over each group of (asset, year, rep_period)

1.1. Loop over each variable in the group: (varid, vartimeblockstart, vartimeblockend) 1.1.1. Loop over each timestep in vartimeblockstart:vartimeblockend 1.1.1.1. Compute the coefficient of the variable based on the repperiod resolution and the variable efficiency 1.1.1.2. Store (varid, coefficient) in workspace[timestep] 1.2. Loop over each constraint in the group: (consid, constimeblockstart, constimeblockend) 1.2.1. Aggregate all variables in workspace[timestep] for timestep in the time block to create a list of variable ids and their coefficients [(var_id1, coef1), ...] 1.2.2. Compute the expression using the variable container, the ids and coefficients

Notes:

• On step 1.2.1, the aggregation can be by either
  • :sum - add the coefficients
  • :mean - add the coefficients and divide by number of times the variable appears
  • :unique_sum - use 1.0 for the coefficient (this is not robust)

compute_dual_variables!(model)

Compute the dual variables for the given model.

If the model does not have dual variables, this function fixes the discrete variables, optimizes the model, and then computes the dual variables.

Arguments

• model: The model for which to compute the dual variables.

Returns

A named tuple containing the dual variables of selected constraints.

create_internal_tables!(connection)

Creates internal tables.

create_merged_tables!(connection)

Create the internal tables of merged flows and assets time partitions to be used in the computation of the lowest and highest resolution tables. The inputs tables are the flows table flow_time_resolution_rep_period, the assets table asset_time_resolution_rep_period and the flows_relationships. All merged tables have the same columns: asset, year, rep_period, time_block_start, and time_block_end. Given a "group" (asset, year, rep_period), the table will have the list of all partitions that should be used to compute the resolution tables. These are the output tables:

• merged_in_flows: Set asset = from_asset and drop to_asset from flow_time_resolution_rep_period.
• merged_out_flows: Set asset = to_asset and drop from_asset from flow_time_resolution_rep_period.
• merged_assets_and_out_flows: Union of merged_out_flows and asset_time_resolution_rep_period.
• merged_all_flows: Union (i.e., vertically concatenation) of the tables above.
• merged_all: Union of merged_all_flows and asset_time_resolution_rep_period.
• merged_flows_relationship: Set asset from flow_time_resolution_rep_period depending on flows_relationships

This function is intended for internal use.

create_model!(energy_problem; kwargs...)

Create the internal model of a TulipaEnergyModel.EnergyProblem. Any keyword argument will be passed to the underlying create_model.

model, expressions = create_model(
    connection,
    variables,
    constraints,
    profiles,
    model_parameters;
    optimizer = HiGHS.Optimizer,
    optimizer_parameters = default_parameters(optimizer),
    model_file_name = "",
    enable_names = true
    direct_model = false,
)

Create the energy model manually. We recommend using create_model! instead.

The optimizer argument should be an MILP solver from the JuMP list of supported solvers. By default we use HiGHS.

The keyword argument optimizer_parameters should be passed as a dictionary of key => value pairs. These can be created manually, obtained using default_parameters, or read from a file using read_parameters_from_file.

parameters = Dict{String,Any}("presolve" => "on", "time_limit" => 60.0, "output_flag" => true)
solve_model(model; optimizer = HiGHS.Optimizer, optimizer_parameters = parameters)

Set enable_names = false to avoid assigning names to variables and constraints, which improves speed but reduces the readability of log messages. For more information, see JuMP.set_string_names_on_creation.

Set direct_model = true to create a JuMP direct_model using optimizer_with_attributes, which has memory improvements. For more information, see JuMP.direct_model.

create_unrolled_partition_table!(connection)

Create unrolled partitions tables from existing DuckDB tables, i.e., adds the time block start and end information.

Input

The following tables are expected to exist in the connection, containing the partition of some, or all, assets or flows, with their respective time information.

• assets_rep_periods_partitions
• flows_rep_periods_partitions
• assets_timeframe_partitions

Output

The generated tables are the unrolled version of the tables above. It transforms each row in (possibly) multiple rows. The columns specification and partition are used to determine the time blocks, and are replaced by columns time_block_start and time_block_end.

• asset_time_resolution_rep_period
• flow_time_resolution_rep_period
• asset_time_resolution_over_clustered_year

default_parameters(Val(optimizer_name_symbol))
default_parameters(optimizer)
default_parameters(optimizer_name_symbol)
default_parameters(optimizer_name_string)

Returns the default parameters for a given JuMP optimizer. Falls back to Dict() for undefined solvers.

Arguments

There are four ways to use this function:

• Val(optimizer_name_symbol): This uses type dispatch with the special Val type. Pass the solver name as a Symbol (e.g., Val(:HiGHS)).
• optimizer: The JuMP optimizer type (e.g., HiGHS.Optimizer).
• optimizer_name_symbol or optimizer_name_string: Pass the name in Symbol or String format and it will be converted to Val.

Using Val is necessary for the dispatch. All other cases will convert the argument and call the Val version, which might lead to type instability.

Examples

using HiGHS
default_parameters(HiGHS.Optimizer)

# output

Dict{String, Any} with 1 entry:
  "output_flag" => false

Another case

default_parameters(Val(:Cbc))

# output

Dict{String, Any} with 1 entry:
  "logLevel" => 0

default_parameters(:Cbc) == default_parameters("Cbc") == default_parameters(Val(:Cbc))

# output

true

export_solution_to_csv_files(output_file, connection)

Saves the solution in CSV files inside output_folder. Notice that this assumes that the solution has already been computed (e.g., by save_solution!, or using rolling horizon).

export_solution_to_csv_files(output_folder, energy_problem)

Saves the solution from energy_problem in CSV files inside output_file. Notice that this assumes that the solution has been computed by save_solution!.

get_single_element_from_query_and_ensure_its_only_one(query_result :: QueryResult)

Given a DuckDB query_result (output of DuckDB.query(...) or DuckDB.execute), return the single element returned by it.

In other words, this assumes that query_result has a single row with a single column.

We use only twice to obtain this single element.

populate_with_defaults!(connection)

Overwrites all tables expected by TulipaEnergyModel appending columns that have defaults. The expected columns and their defaults are obtained from TulipaEnergyModel.schema.

This should be called when you have enough data for a TulipaEnergyModel, but doesn't want to fill out all default columns by hand.

read_parameters_from_file(filepath)

Parse the parameters from a file into a dictionary. The keys and values are NOT checked to be valid parameters for any specific solvers.

The file should contain a list of lines of the following type:

key = value

The file is parsed as TOML, which is intuitive. See the example below.

Example

# Creating file
filepath, io = mktemp()
println(io,
  """
    true_or_false = true
    integer_number = 5
    real_number1 = 3.14
    big_number = 6.66E06
    small_number = 1e-8
    string = "something"
  """
)
close(io)
# Reading
read_parameters_from_file(filepath)

# output

Dict{String, Any} with 6 entries:
  "string"         => "something"
  "integer_number" => 5
  "small_number"   => 1.0e-8
  "true_or_false"  => true
  "real_number1"   => 3.14
  "big_number"     => 6.66e6

energy_problem = run_scenario(
    connection;
    output_folder,
    optimizer,
    optimizer_parameters,
    model_parameters_file,
    model_file_name,
    enable_names,
    log_file,
    show_log)

Run the scenario in the given connection and return the energy problem.

The optimizer and optimizer_parameters keyword arguments can be used to change the optimizer (the default is HiGHS) and its parameters. The arguments are passed to the create_model function.

Set model_file_name = "some-name.lp" to export the problem that is sent to the solver to a file for viewing (.lp or .mps). Set enable_names = false to turn off variable and constraint names (faster model creation). Set direct_model = true to create a JuMP direct model (faster & less memory). Set show_log = false to silence printing the log while running.

Specify a output_folder name to export the solution to CSV files. Specify a model_parameters_file name to load the model parameters from a TOML file. Specify a log_file name to export the log to a file.

save_solution!(connection, model, variables, constraints; compute_duals = true)

Saves the primal and dual variables values, in the following way:

• For each variable in variables, get the solution value and save it in a

column named solution in the corresponding dataset.

• For each constraint in constraints, get the dual of each attached

constraint listed in constraint.constraint_names and save it to the dictionary constraint.duals with the key given by the name.

Notice that the duals are only computed if compute_duals is true.

save_solution!(energy_problem::EnergyProblem; compute_duals = true)

solve_model!(energy_problem::EnergyProblem)

Solve the internal model of an energy_problem.

solve_model(model::JuMP.Model)

Solve the JuMP model.

validate_data!(connection)

Raises an error if the data is not valid.