Data pipeline/workflow

In this section we will take a look into the data in more details, focusing on what you need to go from your raw data all the way to the results.

Workflow overview
Minimum data and using defaults
- Example of using populate_with_defaults!
Groups of tables
Table Schemas

Workflow overview

Here is a brief look at how we imagine a normal usage of the Tulipa model:

Tulipa Workflow. Textual explanation below.

Workflow explanation:

External source: The first thing that you need, and hopefully have, is data. Currently, Tulipa does not provide any public data sources, so we expected that you will load all required data.
Create connection: Tulipa uses a DuckDB database to store the input data, the representation of variables, constraints, and other internal tables, as well as the output. This database is informed through the connection argument in various parts of the API. Most notably, run_scenario and EnergyProblem receive the connection as main argument to create the model (and various internal tables).
- DuckDB connection: This visual representation of the DuckDB shows which tables are created throughout the steps.
Load data: Whether you have CSV/Excel/Parquet files or a separate Database containing your data, you have to load it into the DuckDB connection, i.e., create tables (or table views) with the data that you will process for Tulipa.
- DuckDB connection: We denote this by the Sources tables. We have no expectations or control over what this data contains, but you will need to prepare it for Tulipa in some specific way later.
Data processing instance data with DuckDB/TulipaIO: Now we need to prepare this data for clustering. Even if you don't need to cluster your data, you still need to run TulipaClustering.dummy_cluster!. Since your data in now inside the connection, we assume that you'll use DuckDB's SQL and/or TulipaIO.jl's convenience functions to manipulate it. However, you can create the data that you need externally and just load it again. You are also free to read the data from the connection in whatever other way you find useful (e.g., from Julia or Python, processing with Data Frames, and loading the result into the connection). The important thing is:
- You need to satisfy the TulipaClustering format
- DuckDB connection: This data is represented by the Instance data tables, but it could have been loaded with the Sources.
Cluster into representative periods using TulipaClustering: Run TulipaClustering to compute the representative periods and create tables with this information.
- DuckDB connection: We call these tables the Time data tables. The tables created in this step are part of the cluster group (see Groups of tables below)
Prepare data for TulipaEnergyModel's format: Once more, you have to process your data and create tables for the next step. Since the TulipaEnergyModel is reasonably extensive, you might want to use the populate_with_defaults! function. See Minimum data and using defaults for more details.
- DuckDB connection: TulipaEnergyModel expects the Time data tables from the previous step and new tables that we denote Tulipa format. In rare instances, your data will be complete, but most likely you will need to, at the very least, enhance your tables with default values for columns that are not important for your problem. The tables prepared in this step are part of the input group of tables.
Create internal tables for the model indices: Finally, we start using Tulipa. Your exact experience will depend on what functions you use to create and solve the model, but the underlying idea is the same. The first interaction with Tulipa will create the tables that Tulipa uses to store the variables, constraints, and expressions indices, as well as other internal tables required to get there. We also validate the data to try to make sure that the tables follow the expected requirements.
- DuckDB connection: There are various new tables at this stage that we denote simply as Internal data. Notably, we create the variables and constraints indices tables, part of the groups variables and constraints, respectively. See the Groups of tables section for more details.
Create model: This is where most of the heavy lifting is done in the workflow, apart from solving the model. This step creates all the Julia/JuMP structures using the tables from the previous step. For instance, for each row of each variable table, there is an associated JuMP variable created in this step.
- DuckDB connection: Very little data is created in DuckDB actually. For the most part, we create expressions part of expressions group, that could not have been created before. A lot of Julia/JuMP specific things are created, though, but they cannot be stored in a DuckDB connection.
Solve model: Finally, we give the model to the solver and wait for a result.
Store primal and dual solutions: In this step we compute the dual variables and then load the values of the primal and dual variables in the variable and constraints tables.
- DuckDB connection: We technically don't create any new tables in this step. Instead, we attach new columns to the variables and constraints tables with the corresponding values.
Data processing for plots and dashboard: This is the time to prepare the output that you need, once again using DuckDB/TulipaIO. You can also move all the data out of DuckDB and continue your analysis elsewhere.
- DuckDB connection: You might, optionally, create new tables. We denote these Analysis tables. They can be used for the next steps or for a possible dashboard connecting directly to the DuckDB connection.
Create plots: Optionally create plots.
Export solution: Optionally export all the tables from the DuckDB connection to files, and/or create and save plots.
Output files: External. The outputs of the workflow, which can be whatever you can produce. For instance, CSV/Parquet files with the full variables and constraints tables can be created by exporting the corresponding DuckDB tables.
Dashboard: Representation of a possible dashboard connecting directly to the DuckDB connection.

Minimum data and using defaults

Since TulipaEnergyModel is at a late stage in the workflow, its input data requirements are stricter. Therefore, the input data required by the Tulipa model must follow the schema in the follow section.

Dealing with defaults is hard. A missing value might represent two different things to different people. That is why we require the tables to be complete. However, we also understand that it is not reasonable to expect people to fill a lot of things that they don't need for their models. Therefore, we have created the function populate_with_defaults! to fill the remaining columns of your tables with default values.

To know the defaults, check the Table Schemas below.

Beware implicit assumptions

When data is missing and you automatically fill it with defaults, beware of your assumptions on what that means. For instance, maybe you expect the default capacity to be "unlimited", or maybe you expect it to be 0. Check what are the default values and decide if you want to use them or not. If you think a default does not make sense, open an issue, or a discussion thread.

Example of using `populate_with_defaults!`

Below we have the minimum amount of data (essentially, nothing), that is necessary to start Tulipa.

using TulipaEnergyModel, TulipaIO, DuckDB, DataFrames

data = Dict(
    # Basic asset data
    "asset" => DataFrame(
        :asset => ["some_producer", "some_consumer"],
        :type => ["producer", "consumer"],
    ),
    "asset_both" => DataFrame(
        :asset => ["some_producer", "some_consumer"],
        :commission_year => [2030, 2030],
        :milestone_year => [2030, 2030],
    ),
    "asset_commission" => DataFrame(
        :asset => ["some_producer", "some_consumer"],
        :commission_year => [2030, 2030],
    ),
    "asset_milestone" => DataFrame(
        :asset => ["some_producer", "some_consumer"],
        :milestone_year => [2030, 2030],
    ),

    # Basic flow data
    "flow" => DataFrame(:from_asset => ["some_producer"], :to_asset => ["some_consumer"]),
    "flow_both" => DataFrame(
        :from_asset => ["some_producer"],
        :to_asset => ["some_consumer"],
        :commission_year => [2030],
        :milestone_year => [2030],
    ),
    "flow_commission" => DataFrame(
        :from_asset => ["some_producer"],
        :to_asset => ["some_consumer"],
        :commission_year => [2030],
    ),
    "flow_milestone" => DataFrame(
        :from_asset => ["some_producer"],
        :to_asset => ["some_consumer"],
        :milestone_year => [2030],
    ),

    # Basic time information
    "year_data" => DataFrame(:year => [2030]),
    "rep_periods_data" => DataFrame(:year => [2030, 2030], :rep_period => [1, 2]),
    "timeframe_data" => DataFrame(:year => 2030, :period => 1:365),
    "rep_periods_mapping" =>
        DataFrame(:year => 2030, :period => 1:365, :rep_period => mod1.(1:365, 2)),
)

Dict{String, DataFrames.DataFrame} with 12 entries:
  "asset_both"          => 2×3 DataFrame…
  "flow_milestone"      => 1×3 DataFrame…
  "year_data"           => 1×1 DataFrame…
  "asset_milestone"     => 2×2 DataFrame…
  "flow_both"           => 1×4 DataFrame…
  "asset_commission"    => 2×2 DataFrame…
  "rep_periods_mapping" => 365×3 DataFrame…
  "rep_periods_data"    => 2×2 DataFrame…
  "asset"               => 2×2 DataFrame…
  "flow"                => 1×2 DataFrame…
  "flow_commission"     => 1×3 DataFrame…
  "timeframe_data"      => 365×2 DataFrame…

And here we load this data into a DuckDB connection.

connection = DBInterface.connect(DuckDB.DB)

# Loading the minimum data in the connection
for (table_name, table) in data
    DuckDB.register_data_frame(connection, table, table_name)
end

# Table `asset`:
DuckDB.query(connection, "FROM asset") |> DataFrame

2×2 DataFrame

Row	asset	type
	String	String
1	some_producer	producer
2	some_consumer	consumer

Now we run populate_with_defaults! to fill the remaining columns with default values:

TulipaEnergyModel.populate_with_defaults!(connection)

DuckDB.query(connection, "FROM asset") |> DataFrame

2×23 DataFrame

Row	asset	type	capacity	capacity_storage_energy	consumer_balance_sense	discount_rate	economic_lifetime	energy_to_power_ratio	group	investment_integer	investment_integer_storage_energy	investment_method	is_seasonal	max_ramp_down	max_ramp_up	min_operating_point	ramping	storage_method_energy	technical_lifetime	unit_commitment	unit_commitment_integer	unit_commitment_method	use_binary_storage_method
	String	String	Float64	Float64	String	Float64	Int32	Float64	String?	Bool	Bool	String	Bool	Float64	Float64	Float64	Bool	Bool	Int32	Bool	Bool	String?	String?
1	some_producer	producer	0.0	0.0	==	0.0	1	0.0	missing	false	false	none	false	0.0	0.0	0.0	false	false	1	false	false	missing	missing
2	some_consumer	consumer	0.0	0.0	==	0.0	1	0.0	missing	false	false	none	false	0.0	0.0	0.0	false	false	1	false	false	missing	missing

You can see that the table above has been modified to include many more columns.

Finally, the problem can be solved:

energy_problem = TulipaEnergyModel.run_scenario(
    connection;
    output_folder = mktempdir(),
    show_log = false,
)

DuckDB.query(connection, "FROM var_flow LIMIT 5") |> DataFrame

5×10 DataFrame

Row	id	from_asset	to_asset	year	rep_period	efficiency	flow_coefficient_in_capacity_constraint	time_block_start	time_block_end	solution
	Int64	String	String	Int32	Int32	Float64	Float64	Int32	Int32	Float64
1	1	some_producer	some_consumer	2030	1	1.0	1.0	1	1	0.0
2	2	some_producer	some_consumer	2030	1	1.0	1.0	2	2	0.0
3	3	some_producer	some_consumer	2030	1	1.0	1.0	3	3	0.0
4	4	some_producer	some_consumer	2030	1	1.0	1.0	4	4	0.0
5	5	some_producer	some_consumer	2030	1	1.0	1.0	5	5	0.0

Groups of tables

After creating a connection and loading data in a way that follows the schema (see the previous section on minimum data), then Tulipa will create tables to handle the model data and various internal tables. There are different groups of tables, which we explain below. See the Workflow overview to see where they pop up.

Required by TulipaEnergyModel. These are described in the Table Schemas.
- cluster: Tables created by TulipaClustering.
- input: Tables expected by TulipaEnergyModel.
Created by TulipaEnergyModel.
- variables: Variable indices. These tables are prefixed by var_ in the DuckDB connection.
- constraints: Constraints indices. These tables are prefixed by cons_ in the DuckDB connection.
- expressions: Expressions indices. These tables are prefixed by expr_ in the DuckDB connection.
- resolution: Unrolled partition blocks of assets and flows. These tables are prefixed by either asset_time_resolution_ or flow_time_resolution_.

Additionally, we create various temporary tables, which are prefixed by t_.

Names and prefixes might change

The tables created by TulipaEnergyModel might change in name or prefix unless explicitly noted. Once Tulipa 1.0 is released, we might change this policy.

You probably don't have to navigate these tables yourself, but if you want more information, you can list them all from DuckDB using the duckdb_tables() table. Here are 10 random tables:

DuckDB.query(
    connection,
    "SELECT table_name
    FROM duckdb_tables()
    WHERE table_name NOT LIKE 't_%'
    ORDER BY random() LIMIT 10"
) |> DataFrame

10×1 DataFrame

Row	table_name
	String
1	cons_max_ramp_without_unit_commitment
2	var_storage_level_rep_period
3	var_storage_level_over_clustered_year
4	var_flows_decommission
5	asset_milestone
6	assets_timeframe_profiles
7	var_assets_investment_energy
8	cons_limit_units_on_compact_method
9	merged_all_flows
10	rep_periods_data

Table Schemas

The optimization model parameters with the input data must follow the schema below for each table.

The schemas below are in input-schemas.json. You can also view the schemas after loading the package by typing TulipaEnergyModel.schema in the Julia console. Here is the complete list of model parameters in the schemas per table (or CSV file):

Optional tables/files and their defaults

The following tables/files are allowed to be missing: "assets_rep_periods_partitions", "assets_timeframe_partitions", "assets_timeframe_profiles", "flows_rep_periods_partitions", "group_asset", "profiles_timeframe".

For the partitions tables/files, the default value are specification = uniform and partition = 1 for each asset/flow and year
For the profiles tables/files, the default value is a flat profile of value 1.0 p.u.
If no group table/file is available there will be no group constraints in the model

Table 1 : `asset`

asset

Description: Unique identifier with the name of the asset.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

capacity

Description: Capacity for one unit of the asset (is therefore multiplied by number of existing plus number of invested assets).
Type: DOUBLE
Unit of measure: MW
Default: 0

Constraints	Value
minimum	`0`

capacity_storage_energy

Description: Capacity of one storage unit.
Type: DOUBLE
Unit of measure: MWh
Default: 0

Constraints	Value
minimum	`0`

consumer_balance_sense

Description: Is the sense of the consumer balance constraint, equal to, greater than or less than.
Type: VARCHAR
Unit of measure: No unit
Default: ==

Constraints	Value
oneOf	`Any["==", ">=", "<="]`

discount_rate

Description: e.g. 0.05 is 5 %. discount rate for the annuity calculation.
Type: DOUBLE
Unit of measure: ratio
Default: 0

Constraints	Value
minimum	`0`

economic_lifetime

Description: Economic lifetime of the unit for annuity calculation.
Type: INTEGER
Unit of measure: years
Default: 1

Constraints	Value
minimum	`0`

energy_to_power_ratio

Description: Fixed ratio between the energy storage capacity [MWh] and the discharge/charge capacity [MW] for energy storage investments where storage_method_energy = false.
Type: DOUBLE
Unit of measure: h
Default: 0

Constraints	Value
minimum	`0`

group

Description: Group to which the asset belongs to (null/empty/missing -> no group).
Type: VARCHAR
Unit of measure: No unit
Default: nothing
Constraints: No constraints

investment_integer

Description: Whether investment decisions are using integer variables.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

investment_integer_storage_energy

Description: Whether investment for storage energy is integer or continuous. It only applies for energy storage investments where storage_method_energy = true.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

investment_method

Description: How investments are treated.
Type: VARCHAR
Unit of measure: No unit
Default: none

Constraints	Value
oneOf	`Any["none", "simple", "compact"]`

is_seasonal

Description: Whether seasonal storage (e.g., hydro) or not (e.g., battery)
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

max_ramp_down

Description: Maximum ramping down rate as a portion of the capacity of asset.
Type: DOUBLE
Unit of measure: p.u./h
Default: 0

Constraints	Value
minimum	`0`

max_ramp_up

Description: Maximum ramping up rate as a portion of the capacity of asset.
Type: DOUBLE
Unit of measure: p.u./h
Default: 0

Constraints	Value
minimum	`0`

min_operating_point

Description: Minimum operating point or minimum stable generation level defined as a portion of the capacity of asset.
Type: DOUBLE
Unit of measure: p.u.
Default: 0

Constraints	Value
maximum	`1`
minimum	`0`

ramping

Description: Whether asset has ramping constraints or not.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

storage_method_energy

Description: Whether there is independent investment on storage capacity or not. If false, the investment on storage capacity uses the energytopower_ratio as a constant fixed value.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

technical_lifetime

Description: Technical lifetime of the unit to determine for how long the capacity is considered from the commission year.
Type: INTEGER
Unit of measure: years
Default: 1

Constraints	Value
minimum	`0`

type

Description: Type of energy asset.
Type: VARCHAR
Unit of measure: No unit
Default: No default

Constraints	Value
oneOf	`Any["producer", "consumer", "storage", "conversion", "hub"]`

unit_commitment

Description: Whether asset has unit commitment constraints or not
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

unit_commitment_integer

Description: Whether the unit commitment variables are integer or not.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

unit_commitment_method

Description: Which unit commitment method to use (null/empty/missing -> no unit commitment method).
Type: VARCHAR
Unit of measure: No unit
Default: nothing

Constraints	Value
oneOf	`Any[nothing, "basic"]`

use_binary_storage_method

Description: Whether to use an extra binary variable for the storage assets to avoid charging and discharging simultaneously (null/empty/missing -> no binary).
Type: VARCHAR
Unit of measure: No unit
Default: nothing

Constraints	Value
oneOf	`Any[nothing, "binary", "relaxed_binary"]`

Table 2 : `asset_both`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

commission_year

Description: Year of commissioning
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

decommissionable

Description: Whether or not the asset can be decommissioned.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

initial_storage_units

Description: Number of existing storage units
Type: DOUBLE
Unit of measure: number
Default: 0
Constraints: No constraints

initial_units

Description: Number of existing units
Type: DOUBLE
Unit of measure: number
Default: 0
Constraints: No constraints

milestone_year

Description: Year of investment and operation decisions in the optimization.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 3 : `asset_commission`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

commission_year

Description: Year of commissioning.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

fixed_cost

Description: Fixed annual cost for the asset capacity.
Type: DOUBLE
Unit of measure: CUR/MW/year
Default: 0.0
Constraints: No constraints

fixed_cost_storage_energy

Description: Fixed annual cost for the asset storage capacity
Type: DOUBLE
Unit of measure: CUR/MWh/year
Default: 0.0
Constraints: No constraints

investment_cost

Description: Investment cost for the asset capacity.
Type: DOUBLE
Unit of measure: CUR/MW
Default: 0.0
Constraints: No constraints

investment_cost_storage_energy

Description: Investment cost for the asset energy storage capacity.
Type: DOUBLE
Unit of measure: CUR/MWh
Default: 0.0
Constraints: No constraints

investment_limit

Description: Maximum capacity for the asset investment. If the initial value is null, empty, or missing, it will be no limit.
Type: DOUBLE
Unit of measure: MWh
Default: nothing
Constraints: No constraints

investment_limit_storage_energy

Description: Maximum capacity for the asset storage investment. If the initial value is null, empty, or missing, it will be no limit.
Type: DOUBLE
Unit of measure: MWh
Default: nothing
Constraints: No constraints

storage_loss_from_stored_energy

Description: [e.g. 0.01 means 1% every hour] Loss of stored energy over time.
Type: DOUBLE
Unit of measure: p.u./h
Default: 0
Constraints: No constraints

Table 4 : `asset_milestone`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

initial_storage_level

Description: The initial storage level at the beginning of the optimization. The final storage level needs to be above this initial value. If the initial value is null, empty, or missing, it will be optimized using a cycling constraint that links the last period to the initial period.
Type: DOUBLE
Unit of measure: MWh
Default: nothing

Constraints	Value
minimum	`0`

investable

Description: Whether there is an investment variable created for the asset or not.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

max_energy_timeframe_partition

Description: The maximum amount of energy across the timeframe (e.g., a year) that the asset must produce. If the initial value is null, empty, or missing, it will be no limit.
Type: DOUBLE
Unit of measure: MWh
Default: nothing

Constraints	Value
minimum	`0`

milestone_year

Description: Year of investment and operation decisions in the optimization.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

min_energy_timeframe_partition

Description: The minimum amount of energy across the timeframe (e.g., a year) that the asset must produce. If the initial value is null, empty, or missing, it will be no limit.
Type: DOUBLE
Unit of measure: MWh
Default: nothing

Constraints	Value
minimum	`0`

peak_demand

Description: Value that multiplies the demand profile time series.
Type: DOUBLE
Unit of measure: MW
Default: 0

Constraints	Value
minimum	`0`

storage_inflows

Description: Value that multiplies the inflow profile time series.
Type: DOUBLE
Unit of measure: MWh/year
Default: 0

Constraints	Value
minimum	`0`

units_on_cost

Description: Cost of keeping unit online for one hour or the objective function coefficient on units_on variable. e.g., no_load cost or idling cost
Type: DOUBLE
Unit of measure: CUR/p.u./h
Default: nothing

Constraints	Value
minimum	`0`

Table 5 : `assets_profiles`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

commission_year

Description: Year of commissioning
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_name

Description: Name of profile, used to determine data inside the DuckDB table
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_type

Description: Type of profile, used to determine DuckDB table with source profile
Type: VARCHAR
Unit of measure: No unit
Default: No default

Constraints	Value
oneOf	`Any["availability", "demand", "inflows", "max_storage_level", "min_storage_level"]`

Table 6 : `assets_rep_periods_partitions`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

partition

Description: Partition or temporal resolution in the representative periods. For example, for a uniform specification 1 is hourly, 2 is every two hours.
Type: VARCHAR
Unit of measure: No unit
Default: 1
Constraints: No constraints

rep_period

Description: Number of the representative period
Type: INTEGER
Unit of measure: number
Default: No default
Constraints: No constraints

specification

Description: Partition (or temporal resolution) specification in the representative periods.
Type: VARCHAR
Unit of measure: No unit
Default: uniform

Constraints	Value
oneOf	`Any["uniform", "explicit", "math"]`

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 7 : `assets_timeframe_partitions`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

partition

Description: Partition or temporal resolution in the timeframe. For example, if a period is equivalent to a day then for a uniform specification 1 is per day, 2 is every two days.
Type: VARCHAR
Unit of measure: No unit
Default: 1
Constraints: No constraints

specification

Description: Partition (or temporal resolution) specification in the timeframe.
Type: VARCHAR
Unit of measure: No unit
Default: uniform

Constraints	Value
oneOf	`Any["uniform", "explicit", "math"]`

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 8 : `assets_timeframe_profiles`

asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

commission_year

Description: Year of commissioning
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_name

Description: Name of profile, used to determine data inside the DuckDB table
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_type

Description: Type of profile, used to determine DuckDB table with source profile
Type: VARCHAR
Unit of measure: No unit
Default: No default

Constraints	Value
oneOf	`Any["max_storage_level", "min_storage_level", "max_energy", "min_energy"]`

Table 9 : `flow`

capacity

Description: Capacity for one unit of the transport flow (is therefore multiplied by number of existing plus invested number of transport assets).
Type: DOUBLE
Unit of measure: MW
Default: 0

Constraints	Value
minimum	`0`

carrier

Description: Energy carrier
Type: VARCHAR
Unit of measure: No unit
Default: nothing
Constraints: No constraints

discount_rate

Description: e.g. 0.05 is 5 %. discount rate for the annuity calculation.
Type: DOUBLE
Unit of measure: ratio
Default: 0

Constraints	Value
minimum	`0`

economic_lifetime

Description: Economic lifetime of the transport asset for annuity calculation.
Type: INTEGER
Unit of measure: years
Default: 1

Constraints	Value
minimum	`0`

from_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

investment_integer

Description: Whether investment decisions are using integer variables.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

is_transport

Description: Whether a transport flow or not. Transport assets can have flows in both directions and can be invested in.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

technical_lifetime

Description: Technical lifetime of the transport asset to determine for how long the capacity is considered from the commission year.
Type: INTEGER
Unit of measure: years
Default: 1

Constraints	Value
minimum	`0`

to_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 10 : `flow_both`

commission_year

Description: Year of commissioning
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

decommissionable

Description: Whether the transport asset can be decomission or not.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

from_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

initial_export_units

Description: Number of existing units in from_asset -> to_asset direction
Type: DOUBLE
Unit of measure: number
Default: 0
Constraints: No constraints

initial_import_units

Description: Number of existing units in to_asset -> from_asset direction
Type: DOUBLE
Unit of measure: number
Default: 0
Constraints: No constraints

milestone_year

Description: Year of investment and operation decisions in the optimization.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

to_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 11 : `flow_commission`

commission_year

Description: Year of commissioning
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

efficiency

Description: Efficiency of transfer in from_asset -> to_asset direction
Type: DOUBLE
Unit of measure: ratio
Default: 1.0

Constraints	Value
minimum	`0`

fixed_cost

Description: Fixed annual cost for the transport asset capacity.
Type: DOUBLE
Unit of measure: CUR/MW/year
Default: 0.0
Constraints: No constraints

flow_coefficient_in_capacity_constraint

Description: Coefficient for the flow in the maximum capacity constraints.
Type: DOUBLE
Unit of measure: p.u.
Default: 1

Constraints	Value
minimum	`0`

from_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

investment_cost

Description: Investment cost for the transport asset capacity.
Type: DOUBLE
Unit of measure: CUR/MW
Default: 0.0
Constraints: No constraints

investment_limit

Description: Maximum capacity for the transport asset investment. If the initial value is null, empty, or missing, it will be no limit.
Type: DOUBLE
Unit of measure: MWh
Default: nothing
Constraints: No constraints

to_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 12 : `flow_milestone`

dc_opf

Description: Whether a flow uses direct current optimal power flow (dc-opf) constraints or not. This method only applies to transport flows.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

from_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

investable

Description: Whether there is an investment variable created for the asset or not.
Type: BOOLEAN
Unit of measure: No unit
Default: false
Constraints: No constraints

milestone_year

Description: Year of investment and operation decisions in the optimization.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

reactance

Description: Reactance for the transport flow.
Type: DOUBLE
Unit of measure: p.u.
Default: 0.3
Constraints: No constraints

to_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

variable_cost

Description: Variable cost for the flow.
Type: DOUBLE
Unit of measure: CUR/MWh
Default: 0
Constraints: No constraints

Table 13 : `flows_profiles`

from_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_name

Description: Name of profile, used to determine data inside the DuckDB table
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_type

Description: Type of profile, used to determine DuckDB table with source profile
Type: VARCHAR
Unit of measure: No unit
Default: No default

Constraints	Value
oneOf	`Any["availability"]`

to_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 14 : `flows_relationships`

constant

Description: Constant C in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint
Type: DOUBLE
Unit of measure: MWh
Default: 0
Constraints: No constraints

flow_1_from_asset

Description: Name of the from_asset of the flow_1 in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

flow_1_to_asset

Description: Name of the to_asset of the flow_1 in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

flow_2_from_asset

Description: Name of the from_asset of the flow_2 in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

flow_2_to_asset

Description: Name of the to_asset of the flow_2 in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

milestone_year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

ratio

Description: Ratio A in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint
Type: DOUBLE
Unit of measure: p.u.
Default: 1
Constraints: No constraints

sense

Description: Is the sense in flow_1 {==;>=;<=} C + A x flow_2 relationship constraint, equal to, greater than or less than.
Type: VARCHAR
Unit of measure: No unit
Default: ==

Constraints	Value
oneOf	`Any["==", ">=", "<="]`

Table 15 : `flows_rep_periods_partitions`

from_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

partition

Description: Partition or temporal resolution in the representative periods. For example, for a uniform specification 1 is hourly, 2 is every two hours.
Type: VARCHAR
Unit of measure: No unit
Default: 1
Constraints: No constraints

rep_period

Description: Number of the representative period
Type: INTEGER
Unit of measure: number
Default: No default
Constraints: No constraints

specification

Description: Partition (or temporal resolution) specification in the representative periods.
Type: VARCHAR
Unit of measure: No unit
Default: uniform

Constraints	Value
oneOf	`Any["uniform", "explicit", "math"]`

to_asset

Description: Name of the asset. Same as the one in the asset table.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 16 : `group_asset`

invest_method

Description: true -> activate group constraints; false -> no group investment constraints
Type: BOOLEAN
Unit of measure: No unit
Default: No default
Constraints: No constraints

max_investment_limit

Description: MW (Missing -> no limit)
Type: DOUBLE
Unit of measure: No unit
Default: No default
Constraints: No constraints

milestone_year

Description: Year of investment and operation decisions in the optimization.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

min_investment_limit

Description: MW (Missing -> no limit)
Type: DOUBLE
Unit of measure: No unit
Default: No default
Constraints: No constraints

name

Description: Name of the Group
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 17 : `profiles_rep_periods`

profile_name

Description: Profile name.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

rep_period

Description: Representative period number.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

timestep

Description: Timestep number.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

value

Description: Value of the profile.
Type: DOUBLE
Unit of measure: p.u.
Default: No default
Constraints: No constraints

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 18 : `profiles_timeframe`

period

Description: Period.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

profile_name

Description: Profile name.
Type: VARCHAR
Unit of measure: No unit
Default: No default
Constraints: No constraints

value

Description: value of the profile.
Type: DOUBLE
Unit of measure: p.u.
Default: No default
Constraints: No constraints

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 19 : `rep_periods_data`

num_timesteps

Description: Number of timesteps
Type: INTEGER
Unit of measure: No unit
Default: 8760
Constraints: No constraints

rep_period

Description: Representative period number.
Type: INTEGER
Unit of measure: number
Default: No default
Constraints: No constraints

resolution

Description: Duration of each timestep
Type: DOUBLE
Unit of measure: hours
Default: 1
Constraints: No constraints

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 20 : `rep_periods_mapping`

period

Description: Period number.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

rep_period

Description: Representative period number.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

weight

Description: Hours
Type: DOUBLE
Unit of measure: No unit
Default: 1.0
Constraints: No constraints

year

Description: Milestone year.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 21 : `timeframe_data`

num_timesteps

Description: Number of timesteps of the representative period.
Type: INTEGER
Unit of measure: No unit
Default: 8760
Constraints: No constraints

period

Description: Period.
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

year

Description: Unique identifier (currently, the year itself)
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Table 22 : `year_data`

is_milestone

Description: Whether the year is a milestone year or a vintage year
Type: BOOLEAN
Unit of measure: No unit
Default: true
Constraints: No constraints

length

Description: How many hours in a year, e.g., 8760
Type: INTEGER
Unit of measure: No unit
Default: 8760
Constraints: No constraints

year

Description: Unique identifier (currently, the year itself)
Type: INTEGER
Unit of measure: No unit
Default: No default
Constraints: No constraints

Data pipeline/workflow

Workflow overview

Minimum data and using defaults

Example of using populate_with_defaults!

Groups of tables

Table Schemas

Table 1 : asset

Table 2 : asset_both

Table 3 : asset_commission

Table 4 : asset_milestone

Table 5 : assets_profiles

Table 6 : assets_rep_periods_partitions

Table 7 : assets_timeframe_partitions

Table 8 : assets_timeframe_profiles

Table 9 : flow

Table 10 : flow_both

Table 11 : flow_commission

Table 12 : flow_milestone

Table 13 : flows_profiles

Table 14 : flows_relationships

Table 15 : flows_rep_periods_partitions

Table 16 : group_asset

Table 17 : profiles_rep_periods

Table 18 : profiles_timeframe

Table 19 : rep_periods_data

Table 20 : rep_periods_mapping

Table 21 : timeframe_data

Table 22 : year_data

Example of using `populate_with_defaults!`

Table 1 : `asset`

Table 2 : `asset_both`

Table 3 : `asset_commission`

Table 4 : `asset_milestone`

Table 5 : `assets_profiles`

Table 6 : `assets_rep_periods_partitions`

Table 7 : `assets_timeframe_partitions`

Table 8 : `assets_timeframe_profiles`

Table 9 : `flow`

Table 10 : `flow_both`

Table 11 : `flow_commission`

Table 12 : `flow_milestone`

Table 13 : `flows_profiles`

Table 14 : `flows_relationships`

Table 15 : `flows_rep_periods_partitions`

Table 16 : `group_asset`

Table 17 : `profiles_rep_periods`

Table 18 : `profiles_timeframe`

Table 19 : `rep_periods_data`

Table 20 : `rep_periods_mapping`

Table 21 : `timeframe_data`

Table 22 : `year_data`