Tutorial

Explanation

To simplify, let's consider a single profile, for a single year. Let's denote it as $p_i$, where $i = 1,\dots,N$. The clustering process consists of:

  1. Split N into (let's assume equal) periods of size m = period_duration. We can rename $p_i$ as

    \[p_{j,k}, \qquad \text{where} \qquad j = 1,\dots,m, \quad k = 1,\dots,N/m.\]

  2. Compute num_rps representative periods

    \[r_{j,\ell}, \qquad \text{where} \qquad j = 1,\dots,m, \qquad \ell = 1,\dots,\text{num\_rps}.\]

  3. During computation of the representative periods, we obtained weight $w_{k,\ell}$ between the period $k$ and the representative period $\ell$, such that

    \[p_{j,k} = \sum_{\ell = 1}^{\text{num\_rps}} r_{j,\ell} \ w_{k,\ell}, \qquad \forall j = 1,\dots,m, \quad k = 1,\dots,N/m\]

High level API/DuckDB API

High level API

This tutorial focuses on the highest level of the API, which requires the use of a DuckDB connection.

The high-level API of TulipaClustering focuses on using TulipaClustering as part of the Tulipa workflow. This API consists of three main functions: transform_wide_to_long!, cluster!, and dummy_cluster!. In this tutorial we'll use all three.

Normally, you will have the DuckDB connection from the larger Tulipa workflow, so here we will create a temporary connection with fake data to show an example of the workflow. You can look into the source code of this documentation to see how to create this fake data.

Here is the content of that connection:

using DataFrames, DuckDB

nice_query(str) = DataFrame(DuckDB.query(connection, str))
nice_query("show tables")
1×1 DataFrame
Rowname
String
1profiles_wide

And here is the first rows of profiles_wide:

nice_query("from profiles_wide limit 10")
10×5 DataFrame
Rowyeartimestepavailsolardemand
Int32Int64Float64Float64Float64
1203014.31460.03.67711
2203024.192940.03.91308
3203034.025240.04.23725
4203044.033630.04.72897
5203053.850060.05.07988
6203063.726680.05.92864
7203073.8980.5083155.87297
8203083.854431.563697.07249
9203094.142682.159347.5567
102030104.159652.837137.43377

And finally, this is the plot of the data:

using Plots

table = DuckDB.query(connection, "from profiles_wide")
plot(size=(800, 400))
timestep = [row.timestep for row in table]
for profile_name in (:avail, :solar, :demand)
    value = [row[profile_name] for row in table]
    plot!(timestep, value, lab=string(profile_name))
end
plot!()
Example block output

Transform a wide profiles table into a long table

Required

The long table format is a requirement of TulipaClustering, even for the dummy clustering example.

In this context, a wide table is a table where each new profile occupies a new column. A long table is a table where the profile names are stacked in a column with the corresponding values in a separate column. Given the name of the source table (in this case, profiles_wide), we can create a long table with the following call:

using TulipaClustering

transform_wide_to_long!(connection, "profiles_wide", "profiles")

nice_query("FROM profiles LIMIT 10")
10×4 DataFrame
Rowyeartimestepprofile_namevalue
Int32Int64StringFloat64
120301avail4.3146
220302avail4.19294
320303avail4.02524
420304avail4.03363
520305avail3.85006
620306avail3.72668
720307avail3.898
820308avail3.85443
920309avail4.14268
10203010avail4.15965

Here, we decided to save the long profiles table with the name profiles to use in the clustering below.

Dummy Clustering

A dummy cluster will essentially ignore the clustering and create the necessary tables for the next steps in the Tulipa workflow.

for table_name in (
    "rep_periods_data",
    "rep_periods_mapping",
    "profiles_rep_periods",
    "timeframe_data",
)
    DuckDB.query(connection, "DROP TABLE IF EXISTS $table_name")
end

clusters = dummy_cluster!(connection)

nice_query("FROM rep_periods_data LIMIT 5")
1×4 DataFrame
Rowyearrep_periodnum_timestepsresolution
Int32Int64Int64Float64
1203016721.0
nice_query("FROM rep_periods_mapping LIMIT 5")
1×4 DataFrame
Rowyearperiodrep_periodweight
Int32Int64Int64Float64
12030111.0
nice_query("FROM profiles_rep_periods LIMIT 5")
5×5 DataFrame
Rowrep_periodtimestepyearprofile_namevalue
Int64Int64Int32StringFloat64
1112030avail4.3146
2122030avail4.19294
3132030avail4.02524
4142030avail4.03363
5152030avail3.85006
nice_query("FROM timeframe_data LIMIT 5")
1×3 DataFrame
Rowyearperiodnum_timesteps
Int32Int64Int64
120301672

Clustering

We can perform a real clustering by using the cluster! function with two extra arguments (see Explanation for their deeped meaning):

  • period_duration: How long are the split periods;
  • num_rps: How many representative periods.
period_duration = 24
num_rps = 3

for table_name in (
    "rep_periods_data",
    "rep_periods_mapping",
    "profiles_rep_periods",
    "timeframe_data",
)
    DuckDB.query(connection, "DROP TABLE IF EXISTS $table_name")
end

clusters = cluster!(connection, period_duration, num_rps)

nice_query("FROM rep_periods_data LIMIT 5")
3×4 DataFrame
Rowyearrep_periodnum_timestepsresolution
Int32Int64Int64Float64
120301241.0
220302241.0
320303241.0
nice_query("FROM rep_periods_mapping LIMIT 5")
5×4 DataFrame
Rowyearperiodrep_periodweight
Int32Int64Int64Float64
12030110.922327
22030120.0776726
32030221.0
42030311.0
52030410.187746
nice_query("FROM profiles_rep_periods LIMIT 5")
5×5 DataFrame
Rowrep_periodtimestepyearprofile_namevalue
Int64Int64Int32StringFloat64
1112030avail4.35491
2122030avail4.25846
3132030avail4.11431
4142030avail3.96932
5152030avail3.85393
nice_query("FROM timeframe_data LIMIT 5")
5×3 DataFrame
Rowyearperiodnum_timesteps
Int32Int64Int64
12030124
22030224
32030324
42030424
52030524