Hyperparameter Tuning

    MPSRandomSearch(sampling::Symbol=:LatinHypercube)

Value type used to specify a random search algorithm for hyperparameter tuning an MPS.

Sampling Specifies the method used to determine the search space. The supported sampling methods are

• :LatinHypercube: An implementation of LatinHypercubeSampling.jl's random (pseudo-) Latin Hypercubesearch space generator. Supports both discrete and continuous hyperparameters.
• :UniformRandom: Generate a search space by randomly sampling from the interval [lower bound, upper bound] for each hyperparameter. Supports both discrete andcontinuous hyperparameters.
• Exhaustive: Perform an exhaustive gridsearch of all hyperparameters within the lower and upper bounds. Only supports discrete hyperparameters. Not actually a random search.

function tune(
    Xs::AbstractMatrix, 
    [ys::AbstractVector], 
    nfolds::Integer,
    parameters::NamedTuple,
    optimiser=MPSRandomSearch(:LatinHypercubeSampling);
    <Keyword Arguments>) -> best_parameters::NamedTuple, cache::Dictionary

Perform nfolds-fold hyperparameter tuning of an MPS on the timeseries data Xs, optionally specifying the data classes ys. Returns a NamedTuple containing the optimal hyperparameters, and a cache Dictionary that saves the loss of every tested hyperparameter combination.

parameters specifies the hyperparameters to tune and (optionally) a search space. Currently, every numeric field of MPSOptions is supported. parameters are specified as named tuples, with the key being the name of the hyperparameter (See Example below). There are a couple of options for specifying the bounds:

• The preferred option is Tuple of lower/upper bounds: "params=(eta=(upper_bound,lower_bound), ...)", which allows the optimiser to choose any value in the interval [upperbound, lowerbound]. You can also pass an empty tuple: "params=(eta=(), ...)" which allows the parameter to take any non-negative value.
• As a Vector of possible values, e.g. params = (d=[1,3,6,7,8], ...). For convenience, you can also use the Tuple syntax "params=(d=(start,step,stop), ...)", which is equivalent to "params = (d=start:step:stop, ...)"

Note that if you use the first method, the upper and lower bounds are passed to the hyperparameter tuning algorithm, but may not be strictly enforced depending on your choice of algorithm. Alternatively, use the enforce_bounds=false keyword argument to disable bounds checking completely (for compatible optimisers).

Example:

To tune a classification problem by searching the hyperparameter space η ∈ [0.001, 0.1], d ∈ {5,6,7}, and χmax ∈ {20,21,...,25}:

julia> params = (
    eta=(1e-3, 1e-1), 
    d=(5,7), 
    chi_max=(20,25)
); # configure the search space

julia> nfolds = 5;

julia> best_params, cache = tune(
    X_train, # Training data as a matrix, rows are time series
    y_train, # Vector of time series class labels
    nfolds, # Number of cross validation folds
    params,
    MPSRandomSearch(); # Tuning algorithm
    objective=MisclassificationRate(), # Type of loss to use
    maxiters=20, # Maximum number of tuning iterations
    logspace_eta=true # When true, the eta search space [0.001, 0.1] is sampled logarithmically
)
[...]

julia> best_opts = MPSOptions(best_params); # convert to an MPSOptions object
[...]

Other problem classes are available, see the MPSTime documentation.

Hyperparameter Tuning Methods

The hyperparameter tuning algorithm can be specified with the optimiser argument. This supports the default builtin MPSRandomSearch methods, as well as (in theory) any solver that is supported by the Optimization.jl interface. Note that many of these solvers struggle with discrete search spaces, such as tuning the integer valued chi_max and d, or tuning an eta specified with a vector. Some of them require initial conditions (set provide_x0=true), and some require no initial conditions (set provide_x0=false), so your mileage may vary. By default, tune() handles optimisers attempting to evaluate discrete hyperparameters at a non-integer value by rounding, and using its own cache to avoid rounding based cache misses.

There are a lot of keyword arguments... Extended help is avaliable with ??tune

See also: evaluate

Extended Help

Keyword Arguments

Hyperparameter Options

• opts0::AbstractMPSOptions=MPSOptions(; verbosity=-5, log_level=-1, sigmoid_transform=objective isa ClassificationLoss): Default hyperparamaters to pass to fitMPS. Hyperparameter candidates are generated by modifying opts0 with the values in the search space specified by parameters.
• enforce_bounds::Bool=true: Whether to pass the constraints given in params to the optimisation algorithm.
• logspace_eta::Bool=false: Whether to treat the eta parameterspace as logarithmic. E.g. setting parameters=(eta=(10^-3,10^-1) ) and logspace_eta=true will sample each eta_candidate from the log10 search space [-3.,-1.], and then pass eta = 10^(eta_candidate) to MPSOptions.
• input_supertype::Type=Float64: A numeric type that can represent the types of each hyperparameter being tuned as well as their upper and lower bounds. Typically, Float64 is sufficient, but it can be set to Int for purely discrete optimisation problems etc. This is necessary for mixed integer / Float hyperparameter tuning because certain solvers in Optimization.jl require variables in the search space to all be the same type.

Loss and Windowing

• objective::TuningLoss=ImputationLoss(): The objective of the hyperparameter optimisation. This comes in two categories:

Imputation Loss

obvjective=ImputationLoss() Uses the mean of the mean absolute error to measure the performance of an MPS on imputing unseen data. First, it generates 'corrupted' time series data by applying missing data windows to the validation set, using one of the following options:

• pms::Union{Nothing, AbstractVector}=nothing: Stands for 'percentage missings'. Will remove a randomly selected contiguous blocks from each time series in the validation set, according to the percentages missing listed in the pms vector. For example, pms=[0.05, 0.05, 0.6, 0.95] will generate four windows, two with 5% missing, and one each with 60% and 95% missing
• windows::Union{Nothing, AbstractVector, Dict}=nothing: Manually input missing windows. Expects a vector of missing windows, or a dictionary where values(windows) is a vector of missing windows.

The tuning loss is the average of computing the mean absolute error of imputing every window on every element of the validation set.

Classification Loss

Classification type problems can be hyperparameter tuned to minimise either MisclassificationRate() (1 - classification accuracy), or BalancedMisclassificationRate() (1 - balanced accuracy).

Custom Loss Functions

Custom losses can be used by implementing a custom loss value type (CustomLoss <: TuningLoss) and extending the definition of MPSTime.eval_loss with the signature

eval_loss(
    CustomLoss(), 
    mps::TrainedMPS, 
    X_validation::AbstractMatrix, 
    y_validation::AbstractVector, 
    windows; 
    p_fold=nothing, 
    distribute::Bool=false
) -> Union{Float64, Vector{Float64}

if eval_loss returns a vector, then tune() will optimise mean(eval_loss(...)). For concrete examples, see the documentation

Tuning algorithm

• abstol::Float64=1e-3: Passed directly to Optimization.jl: Absolute tolerance in changes to the objective (loss) function
• maxiters::Integer=250: Maximum number of iterations allowed when solving
• provide_x0::Bool=true: Whether to provide initial conditions to the solve, ignored by MPSRandomSearch. The initial condition will be opts0, unless it contains a hyperparameter outside the range specified by parameters, in which case the lower bound of that hyperparameter will be used.
• rng::Union{Integer, AbstractRNG}=1: An Integer or RNG object used to seed any randomness in imputation window or search space generation.
• kwargs...: Any further keyword arguments to passed through to Optimization.jl through the Optimization.solve function

Folds and Cross validation

• foldmethod::Union{Function, AbstractVector}=make_stratified_cvfolds: The method used to generate the train/validation folds from Xs. Can either be an nfolds-long Vector of [train_indices::Vector, validation_indices::Vector] pairs, or a function that produces them, with the signature foldmethod(Xs,ys, nfolds; rng::AbstractRNG) To clarify, the tune function determines the train/validation splits for the ith fold in the following way:

julia> folds::Vector = foldmethod isa Function ? foldmethod(Xs,ys, nfolds; rng=rng) : foldmethod;

julia> train_inds, validation_inds = folds[i];

julia> X_train, y_train = Xs[train_inds, :], ys[train_inds];

julia> X_validation, y_validation = Xs[validation_inds, :], ys[validation_inds];

Logging

• verbosity::Integer=1: Controls how explicit the logging is. 0 for none, 5 for maximimum. This is separate to the verbosity in MPSOptions.
• pre_string::String="": Prints this string on the same line before logging messages are printed. Useful for logging when calling tune() multiple times in parallel.

Distributed Computing

Parallel processing is available using processors added via Distributed.jl's addprocs function.

• distribute_iters::Bool=false: When using an MPSRandomSearch, distribute the search grid across all available processors. For thread safety, using distribute_iters disables caching.
• distribute_folds::Bool=false: Distribute each fold to its own processor. Scales up to at most nfolds processors. Not very compatible with distribute_iters.
• workers::AbstractVector{Int}=distribute_folds ? workers() : Int[]: Workers that may be used to distribute folds, does not affect distribute_iters. This can be used to run multiple instances of tune() on different sets of workers.
• disable_nondistributed_threading::Bool=false: Attempts to disable threading using BLAS.set_num_threads(1) and Strided.disable_threads() (May not work if using the MKL.jl linear algebra backend).

function evaluate(
    Xs::AbstractMatrix, 
    [ys::AbstractVector], 
    nfolds::Integer,
    tuning_parameters::NamedTuple,
    tuning_optimiser=MPSRandomSearch();
    <Keyword Arguments>) -> results::Vector{Dictionary}

Evaluate the performance of MPSTime by hyperparameter tuning on nfolds resampled folds of the timeseries dataset Xs with classes ys.

tuning_parameters controls the hyperparamters to tune over, and tuning_optimiser specifies the hyperparameter tuning algorithm. These are passed directly to tune, refer to its documentation for details.

Example

To evaluate a classification problem by searching the hyperparameter space η ∈ [0.001, 0.1], d ∈ {5,6,7}, and χmax ∈ {20,21,...,25},

julia> params = (
    eta=(1e-3, 1e-1), 
    d=(5,7), 
    chi_max=(20,25)
);
julia> nfolds = 30;

julia> results = evaluate(
    X_train, # training data as a matrix, rows are time series
    y_train, # Vector of time series class labels
    nfolds, # number of resample folds
    params, # search space
    MPSRandomSearch(); # Hyperparameter search method, see Extended help
    objective=MisclassificationRate(), # Type of lsso to use
    maxiters=20, # Maximum number of tuning iterations
    logspace_eta=true # the eta search space [0.001, 0.1] is sampled logarithmically
)
[...]

Return value

A length nfolds vector of dictionaries that contain detailed informatation about each fold. Each dictionary has the following keys:

• "fold"=>Integer: The fold index.
• "objective"=>String: The objective (loss function) this fold was trained on.
• "train_inds"=>Vector: The indices (rows of Xs) this fold was tuned/trained on.
• "test_inds"=>Vector: The indices (rows of Xs) this fold was tested on.
• "optimiser"=>String: Name of the optimiser used to hyperparameter tune each fold.
• "tuning_windows"=>Vector: The windows used to hyperparameter tune this fold.
• "tuning_pms"=>Vector: The 'Percentages Missing' used to hyperparameter tune this fold (possibly used to generate tuning_windows).
• "eval_windows"=>Vector: The windows used to evaluate the test loss.
• "eval_pms"=>eval_pms: The 'Percentages Missing' used to evaluate the test loss (possibly used to generate eval_windows).
• "time"=>Vector: Total time to tune and test this fold in seconds.
• "opts"=>MPSOptions: Optimal options for this fold as determined by tune(). Used to compute the test loss.
• "cache"=>Dict: Cache of the validation losses of every set of hyperparameters evaluated on this fold. Disabled if distribute_iters is true.
• "loss"=>Union{Vector{Float64}, Float64}. The test loss of this fold. If objective is an ImputationLoss(), this is a vector with each entry corresponding to a window in results[fold]["eval_windows"].

There are a lot of keyword arguments... Extended help is avaliable with ??evaluate

Extended Help

Keyword Arguments

Loss and Windowing

• objective::TuningLoss=ImputationLoss(): The loss used to evaluate and tune the MPS. If its an ImputationLoss, then either pms or windows must be specified for each of evaluation and tuning. See the tune extended documentation for more details.

• eval_pms::Union{Nothing, AbstractVector}=nothing: 'Percentage MissingS' used to evaluate the test loss.

• eval_windows::Union{Nothing, AbstractVector, Dict}=nothing: Windows used to evaluate the test loss.

• tuning_pms::Union{Nothing, AbstractVector}=eval_pms: 'Percentage MissingS' passed to tune, and used to compute validation loss.

• tuning_windows::Union{Nothing, AbstractVector, Dict}=eval_windows: Windows passed to tune, and used to compute validation loss.

• rng::Union{Integer, AbstractRNG}=1: An integer or RNG object used to seed any randomness in imputation window or search space generation. Random.seed!(fold)` is called prior to tuning each fold, so that any optimization algorithms that are random but don't take rng objects should still be deterministic.

• tuning_rng::AbstractVector{<:Union{Integer, AbstractRNG}=collect(1:nfolds): Passed through to tune. An integer or RNG object used to seed any randomness in tuning imputation window generation or hyperparameter searching.

• opts0::AbstractMPSOptions=MPSOptions(; verbosity=-5, log_level=-1, sigmoid_transform=(objective isa ClassificationLoss)): Options that are modified by the best options returned by tune. Used to train the MPS which evalutates the test loss.

• tuning_opts0::AbstractMPSOptions=opts0: Initial guess passed as opts0 to tune that sets the values of the non-tuned hyperparameters. Should generally always be the same as opts0, but can be specified separately in case you wish to make the final mps train with more verbosity etc.

Resampling and Cross Validation

• foldmethod::Union{Function, Vector}=make_stratified_cvfolds: Can either be an nfolds-long Vector of [train_indices::Vector, test_indices::Vector] pairs, or a function that produces them, with the signature foldmethod(Xs,ys, nfolds; rng::AbstractRNG) To clarify, the tune function determines the train/test splits for the ith fold in the following way:

julia> folds::Vector = foldmethod isa Function ? foldmethod(Xs,ys, nfolds; rng=rng) : foldmethod;

julia> train_inds, test_inds = folds[i];

julia> X_train, y_train = Xs[train_inds, :], ys[train_inds];

julia> X_test, y_test = Xs[test_inds, :], ys[test_inds];

foldmethod defaults to nfold-fold cross validation.

• tuning_foldmethod::Union{Function, Vector}=make_stratified_cvfolds: Same as above, although it is passed to tune and used to split the training set into hyperparameter train/validation sets. The fold number specified by the n_cvfolds keyword.
• fold_inds::Vector{<:Integer}=collect(1:nfolds): A vector of the fold indices to evaluate. This can be used to split large training runs into batches, or to resume a halted benchmark.

Distributed Computing

Several parallel processing paradigms are availble for different use cases, implented using processors added via Distributed.jl's addprocs` function.

• distribute_iters::Bool=false: When using an MPSRandomSearch, for each fold, distribute the search grid across all available processors. For thread safety, using distributed_iters disables caching.
• distribute_folds::Bool=false: Allocate one processor to each fold.
• distribute_cvfolds::Bool=false: Equivalent to passing distribute_folds to tune. Allocates a processor to each hyperparameter train/val split.
• distribute_final_eval::Bool=false: Allocate a processor to each test timeseries when computing the test loss. Useful when the test set is very large. The only option compatible with the others.

Saving and Resuming

If write is enabled, evaluate will automatically resume if it finds a partially complete run.

• write::Bool=false: Whether to write output to files. If true, it will save temporary files, saving each completed fold inside "$writedir/$(simname)_temp/", and the final result to "$writedir/$(simname).jld2".
• writedir::String="evals": The directory to save data to.
• simname::String="$(objective)_$(tuning_optimiser)_f=$(nfolds)_cv$(n_cvfolds)_iters=$(tuning_maxiters)": The simulation name. Used to determine save location.
• delete_tmps::Bool=length(fold_inds)==nfolds: Whether to delete the temp directory at the end.

Logging

• verbosity::Integer=1: Controls how explicit the logging is. 0 for none, 5 for maximimum. This is separate to the verbosity in MPSOptions.

Hyperparameter Tuning Options

These options are passed directly to their corresponding keywords in tune

• n_cvfolds::Integer=5: Corresponds to nfolds in tune, number of train/val splits.
• logspace_eta::Bool=false: Whether to treat the eta parameterspace as logarithmic. E.g. setting parameters=(eta=(10^-3,10^-1) ) and logspace_eta=true will sample each eta_candidate from the log10 search space [-3.,-1.], and then pass eta = 10^(eta_candidate) to MPSOptions.
• input_supertype::Type=Float64: A numeric type that can represent the types of each hyperparameter being tuned as well as their upper and lower bounds. Typically, Float64 is sufficient, but it can be set to Int for purely discrete optimisation problems etc. This is necessary for mixed Integer / Float hyperparameter tuning because certain solvers in Optimization.jl require variables in the search space to all be the same type.
• tuning_abstol::Float64=1e-3: Passed directly to Optimization.jl: Absolute tolerance in changes to the objective (loss) function.
• tuning_maxiters::Integer=250: Maximum number of iterations allowed when solving.
• provide_x0::Bool=true: Whether to provide initial conditions to the solve, ignored by MPSRandomSearch. The initial condition will be opts0, unless it contains a hyperparameter outside the range specified by parameters, in which case the lower bound of that hyperparameter will be used.

Further keyword arguments to evaluate are passed through to tune, and then Optimization.jl through the Optimization.solve function.

make_stratified_cvfolds(
    Xs::AbstractMatrix, 
    ys::AbstractVector, 
    nfolds::Integer; 
    rng=Union{Integer, AbstractRNG}, 
    shuffle::Bool=true
) -> folds::Vector{Vector{Vector{Int}}}

Creates nfold-fold stratified cross validation train/validation splits for hyperparameter tuning, with the form:

julia> train_indices_fold_i, validation_indices_fold_i = folds[i];

Uses MLJ's StratifiedCV method (https://juliaai.github.io/MLJBase.jl/stable/resampling/#MLJBase.StratifiedCV).

eval_loss(
    ::TuningLoss, 
    mps::TrainedMPS, 
    X_val::AbstractMatrix, 
    y_val::AbstractVector, 
    windows::Union{Nothing, AbstractVector}=nothing;
    p_fold::Union{Nothing, Tuple}=nothing,
    distribute::Bool=false,
    )

Evaluate the TuningLoss of mps on the validation time-series dataset specified by X_val, y_val.

p_fold is to allow verbose logging during runs of tune and evaluate. When computing an imputation loss, windows are used to compute imputation losses, as specified in tune, and distribute will distribute the loss calculation across each time-series instance.

    MPSRandomSearch(sampling::Symbol=:LatinHypercube)

Value type used to specify a random search algorithm for hyperparameter tuning an MPS.

Sampling Specifies the method used to determine the search space. The supported sampling methods are

• :LatinHypercube: An implementation of LatinHypercubeSampling.jl's random (pseudo-) Latin Hypercubesearch space generator. Supports both discrete and continuous hyperparameters.
• :UniformRandom: Generate a search space by randomly sampling from the interval [lower bound, upper bound] for each hyperparameter. Supports both discrete andcontinuous hyperparameters.
• Exhaustive: Perform an exhaustive gridsearch of all hyperparameters within the lower and upper bounds. Only supports discrete hyperparameters. Not actually a random search.