Reference

Structure to hold the time series used in clustering together with some summary statistics on the data.

Structure to hold the clustering result.

append_period_from_source_df_as_rp!(df; source_df, period, rp, key_columns)

Extracts a period with index period from source_df and appends it as a representative period with index rp to df, using key_columns as keys.

Examples

julia> source_df = DataFrame([:period => [1, 1, 2, 2], :timestep => [1, 2, 1, 2], :a .=> "b", :value => 5:8])
4×4 DataFrame
 Row │ period  timestep  a       value
     │ Int64   Int64      String  Int64
─────┼──────────────────────────────────
   1 │      1          1  b           5
   2 │      1          2  b           6
   3 │      2          1  b           7
   4 │      2          2  b           8

julia> df = DataFrame([:rep_period => [1, 1, 2, 2], :timestep => [1, 2, 1, 2], :a .=> "a", :value => 1:4])
4×4 DataFrame
 Row │ rep_period  timestep  a       value
     │ Int64       Int64      String  Int64
─────┼──────────────────────────────────────
   1 │          1          1  a           1
   2 │          1          2  a           2
   3 │          2          1  a           3
   4 │          2          2  a           4

julia> TulipaClustering.append_period_from_source_df_as_rp!(df; source_df, period = 2, rp = 3, key_columns = [:timestep, :a])
6×4 DataFrame
 Row │ rep_period  timestep  a       value
     │ Int64       Int64      String  Int64
─────┼──────────────────────────────────────
   1 │          1          1  a           1
   2 │          1          2  a           2
   3 │          2          1  a           3
   4 │          2          2  a           4
   5 │          3          1  b           7
   6 │          3          2  b           8

combine_periods!(df)

Modifies a dataframe df by combining the columns timestep and period into a single column timestep of global time steps. The period duration is inferred automatically from the maximum time step value, assuming that periods start with time step 1.

Examples

julia> df = DataFrame([:period => [1, 1, 2], :timestep => [1, 2, 1], :value => 1:3])
3×3 DataFrame
 Row │ period  timestep  value
     │ Int64   Int64      Int64
─────┼──────────────────────────
   1 │      1          1      1
   2 │      1          2      2
   3 │      2          1      3

julia> TulipaClustering.combine_periods!(df)
3×2 DataFrame
 Row │ timestep  value
     │ Int64      Int64
─────┼──────────────────
   1 │         1      1
   2 │         2      2
   3 │         3      3

df_to_matrix_and_keys(df, key_columns)

Converts a dataframe df (in a long format) to a matrix, ignoring the columns specified as key_columns. The key columns are converted from long to wide format and returned alongside the matrix.

Examples

julia> df = DataFrame([:period => [1, 1, 2, 2], :timestep => [1, 2, 1, 2], :a .=> "a", :value => 1:4])
4×4 DataFrame
 Row │ period  timestep  a       value
     │ Int64   Int64      String  Int64
─────┼──────────────────────────────────
   1 │      1          1  a           1
   2 │      1          2  a           2
   3 │      2          1  a           3
   4 │      2          2  a           4

julia> m, k = TulipaClustering.df_to_matrix_and_keys(df, [:timestep, :a]); m
2×2 Matrix{Float64}:
 1.0  3.0
 2.0  4.0

julia> k
2×2 DataFrame
 Row │ timestep  a
     │ Int64      String
─────┼───────────────────
   1 │         1  a
   2 │         2  a

find_auxiliary_data(clustering_data)

Calculates auxiliary data associated with the clustering_data. These include:

• key_columns_demand: key columns in the demand dataframe
• key_columns_generation_availability: key columns in the generation availability dataframe
• period_duration: duration of time periods (in time steps)
• last_period_duration: duration of the last period
• n_periods: total number of periods

find_period_weights(period_duration, last_period_duration, n_periods, drop_incomplete_periods)

Finds weights of two different types of periods in the clustering data:

• complete periods: these are all of the periods with length equal to period_duration.
• incomplete last period: if last period duration is less than period_duration, it is incomplete.

findrepresentativeperiods( clusteringdata; nrp = 10, rescaledemanddata = true, dropincompletelastperiod = false, method = :kmeans, distance = SqEuclidean(), args..., )

Finds representative periods via data clustering.

• clustering_data: the data to perform clustering on.
• n_rp: number of representative periods to find.
• rescale_demand_data: if true, demands are first divided by the maximum demand value, so that they are between zero and one like the generation availability data
• drop_incomplete_last_period: controls how the last period is treated if it is not complete: if this parameter is set to true, the incomplete period is dropped and the weights are rescaled accordingly; otherwise, clustering is done for n_rp - 1 periods, and the last period is added as a special shorter representative period
• method: clustering method to use, either :k_means and :k_medoids
• distance: semimetric used to measure distance between data points.
• other named arguments can be provided; they are passed to the clustering method.

fitrepperiodweights!(weightmatrix, clusteringmatrix, rpmatrix; weight_type, tol, args...)

Given the initial weight guesses, finds better weights for convex or conical combinations of representative periods. For conical weights, it is possible to bound the total weight by one.

The arguments:

• clustering_result: the result of running TulipaClustering.find_representative_periods
• weight_type: the type of weights to find; possible values are:
  • :convex: each period is represented as a convex sum of the representative periods (a sum with nonnegative weights adding into one)
  • :conical: each period is represented as a conical sum of the representative periods (a sum with nonnegative weights)
  • :conical_bounded: each period is represented as a conical sum of the representative periods (a sum with nonnegative weights) with the total weight bounded from above by one.
• tol: algorithm's tolerance; when the weights are adjusted by a value less then or equal to tol, they stop being fitted further.
• other arguments control the projected subgradient method; they are passed through to TulipaClustering.projected_subgradient_descent!.

fitrepperiodweights!(weightmatrix, clusteringmatrix, rpmatrix; weight_type, tol, args...)

Given the initial weight guesses, finds better weights for convex or conical combinations of representative periods. For conical weights, it is possible to bound the total weight by one.

The arguments:

• weight_matrix: the initial guess for weights; the weights are adjusted using a projected subgradient descent method
• clustering_matrix: the matrix of raw clustering data
• rp_matrix: the matrix of raw representative period data
• weight_type: the type of weights to find; possible values are:
  • :convex: each period is represented as a convex sum of the representative periods (a sum with nonnegative weights adding into one)
  • :conical: each period is represented as a conical sum of the representative periods (a sum with nonnegative weights)
  • :conical_bounded: each period is represented as a conical sum of the representative periods (a sum with nonnegative weights) with the total weight bounded from above by one.
• tol: algorithm's tolerance; when the weights are adjusted by a value less then or equal to tol, they stop being fitted further.
• show_progress: if true, a progress bar will be displayed.
• other arguments control the projected subgradient method; they are passed through to TulipaClustering.projected_subgradient_descent!.

greedy_convex_hull(matrix; n_points, distance, initial_indices, mean_vector)

Greedy method for finding n_points points in a hull of the dataset. The points are added iteratively, at each step the point that is the furthest away from the hull of the current set of points is found and added to the hull.

• matrix: the clustering matrix
• n_points: number of hull points to find
• distance: distance semimetric
• initial_indices: initial points which must be added to the hull, can be nothing
• mean_vector: when adding the first point (if initial_indices is not given), it will be chosen as the point furthest away from the mean_vector; this can be nothing, in which case the first step will add a point furtherst away from the centroid (the mean) of the dataset

matrix_and_keys_to_df(matrix, keys)

Converts a a matrix matrix to a dataframe, appending the key columns given by keys.

Examples

julia> m = [1.0 3.0; 2.0 4.0]
2×2 Matrix{Float64}:
 1.0  3.0
 2.0  4.0

julia> k = DataFrame([:timestep => 1:2, :a .=> "a"])
2×2 DataFrame
 Row │ timestep  a
     │ Int64      String
─────┼───────────────────
   1 │         1  a
   2 │         2  a

julia> TulipaClustering.matrix_and_keys_to_df(m, k)
4×4 DataFrame
 Row │ rep_period  timestep  a       value
     │ Int64       Int64      String  Float64
─────┼────────────────────────────────────────
   1 │          1          1  a           1.0
   2 │          1          2  a           2.0
   3 │          2          1  a           3.0
   4 │          2          2  a           4.0

projectontononnegative_orthant(vector)

Projects vector onto the nonnegative orthant. This projection is trivial: replace negative components of the vector with zeros.

projectontosimplex(vector)

Projects vector onto a unit simplex using Michelot's algorithm in Condat's accelerated implementation (2017). See Figure 2 of Condat, L. Fast projection onto the simplex and the ball. Math. Program. 158, 575–585 (2016).. For the details on the meanings of v, ṽ, ρ and other variables, see the original paper.

projectontostandard_basis(vector)

Projects vector onto the standard basis. This projection is trivial: replace all components of the vector with zeros, except for the largest one, which is replaced with one.

projectedsubgradientdescent!(x; gradient, projection, niters, rtol, learningrate, adaptivegrad)

Fits x using the projected gradient descent scheme.

The arguments:

• x: the value to fit
• subgradient: the subgradient operator, that is, a function that takes vectors of the same shape as x as inputs and returns a subgradient of the loss at that point; the fitting is done to minimize the corresponding implicit loss
• projection: the projection operator, that is, a function that, given a vector x, finds a point within some subspace that is closest to x
• niters: maximum number of projected gradient descent iterations
• tol: tolerance; when no components of x improve by more than tol, the algorithm stops
• learning_rate: learning rate of the algorithm
• adaptive_grad: if true, the learning rate is adjusted using the adaptive gradient method, see John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. J. Mach. Learn. Res. 12, null (2/1/2011), 2121–2159.

split_into_periods!(df; period_duration=nothing)

Modifies a dataframe df by separating the column timestep into periods of length period_duration. The new data is written into two columns:

• period: the period ID;
• timestep: the time step within the current period.

If period_duration is nothing, then all of the time steps are within the same period with index 1.

Examples

julia> df = DataFrame([:timestep => 1:4, :value => 5:8])
4×2 DataFrame
 Row │ timestep  value
     │ Int64      Int64
─────┼──────────────────
   1 │         1      5
   2 │         2      6
   3 │         3      7
   4 │         4      8

julia> TulipaClustering.split_into_periods!(df; period_duration=2)
4×3 DataFrame
 Row │ period  timestep  value
     │ Int64   Int64      Int64
─────┼──────────────────────────
   1 │      1          1      5
   2 │      1          2      6
   3 │      2          1      7
   4 │      2          2      8

julia> df = DataFrame([:period => [1, 1, 2], :timestep => [1, 2, 1], :value => 1:3])
3×3 DataFrame
 Row │ period  timestep  value
     │ Int64   Int64      Int64
─────┼──────────────────────────
   1 │      1          1      1
   2 │      1          2      2
   3 │      2          1      3

julia> TulipaClustering.split_into_periods!(df; period_duration=1)
3×3 DataFrame
 Row │ period  timestep  value
     │ Int64   Int64      Int64
─────┼──────────────────────────
   1 │      1          1      1
   2 │      2          1      2
   3 │      3          1      3

julia> TulipaClustering.split_into_periods!(df)
3×3 DataFrame
 Row │ period  timestep  value
     │ Int64   Int64      Int64
─────┼──────────────────────────
   1 │      1          1      1
   2 │      1          2      2
   3 │      1          3      3

validate_df_and_find_key_columns(df)

Checks that dataframe df contains the necessary columns and returns a list of columns that act as keys (i.e., unique data identifiers within different periods).

Examples

julia> df = DataFrame([:period => [1, 1, 2], :timestep => [1, 2, 1], :a .=> "a", :value => 1:3])
3×4 DataFrame
 Row │ period  timestep  a       value
     │ Int64   Int64      String  Int64
─────┼──────────────────────────────────
   1 │      1          1  a           1
   2 │      1          2  a           2
   3 │      2          1  a           3

julia> TulipaClustering.validate_df_and_find_key_columns(df)
2-element Vector{Symbol}:
 :timestep
 :a

julia> df = DataFrame([:value => 1])
1×1 DataFrame
 Row │ value
     │ Int64
─────┼───────
   1 │     1

julia> TulipaClustering.validate_df_and_find_key_columns(df)
ERROR: DomainError with 1×1 DataFrame
 Row │ value
     │ Int64
─────┼───────
   1 │     1:
DataFrame must contain columns `timestep` and `value`

weight_matrix_to_df(weights)

Converts a weight matrix from a (sparse) matrix, which is more convenient for internal computations, to a dataframe, which is better for saving into a file. Zero weights are dropped to avoid cluttering the dataframe.

write_clustering_result_to_table(connection, clustering_result)

Writes a TulipaClustering.ClusteringResult to CSV files in the output_folder.