A simulation experiment with any number of DGP, Method, Evaluator, and Visualizer objects.
Generally speaking, users won't directly interact with the Experiment R6
class, but instead indirectly through create_experiment() and the tidy
Experiment helpers listed in below in the See also section.
Details
When run, an Experiment seamlessly combines DGPs and
Methods, computing results in parallel. Those results can then be
evaluated using Evaluators and visualized using Visualizers.
See also
The following tidy helpers take an Experiment object as their
first argument: create_experiment(), generate_data(),
fit_experiment(), evaluate_experiment(), visualize_experiment(),
run_experiment(), clear_cache(), get_cached_results(),
get_save_dir(), set_save_dir(), save_experiment(),
export_visualizers(), add_*(),
update_*(), remove_*(),
get_*(), and *_vary_across().
Methods
Method new()
Initialize a new Experiment object.
Usage
Experiment$new(
name = "experiment",
dgp_list = list(),
method_list = list(),
evaluator_list = list(),
visualizer_list = list(),
future.globals = TRUE,
future.packages = NULL,
clone_from = NULL,
save_dir = NULL,
...
)Arguments
nameThe name of the
Experiment.dgp_listAn optional list of DGP objects.
method_listAn optional list of Method objects.
evaluator_listAn optional list of Evaluator objects.
visualizer_listAn optional list of Visualizer objects.
future.globalsCharacter vector of names in the global environment to pass to parallel workers. Passed as the argument of the same name to
future.apply::future_lapply()and related functions. To set for a specific run of the experiment, use the same argument inrun_experiment().future.packagesCharacter vector of packages required by parallel workers. Passed as the argument of the same name to
future.apply::future_lapply()and related functions. To set for a specific run of the experiment, use the same argument inrun_experiment().clone_fromAn optional
Experimentobject to use as a base for this one.save_dirAn optional directory in which to save the experiment's results. If
NULL, results are saved in the current working directory in a directory called "results" with a sub-directory named afterExperiment$namewhen usingrun_experiment()orfit_experiment()withsave=TRUE....Not used.
Method run()
Run the full Experiment pipeline (fitting, evaluating,
and visualizing).
Usage
Experiment$run(
n_reps = 1,
parallel_strategy = "reps",
future.globals = NULL,
future.packages = NULL,
future.seed = TRUE,
use_cached = FALSE,
return_all_cached_reps = FALSE,
save = FALSE,
checkpoint_n_reps = 0,
verbose = 1,
...
)Arguments
n_repsThe number of replicates of the
Experimentfor this run.parallel_strategyA vector with some combination of "reps", "dgps", or "methods". Determines how computation will be distributed across available resources. Currently only the default, "reps", is supported.
future.globalsCharacter vector of names in the global environment to pass to parallel workers. Passed as the argument of the same name to
future.apply::future_lapplyand related functions. To set for all runs of the experiment, use the same argument during initialization.future.packagesCharacter vector of packages required by parallel workers. Passed as the argument of the same name to
future.apply::future_lapplyand related functions. To set for all runs of the experiment, use the same argument during initialization.future.seedPassed as the argument of the same name in
future.apply::future_apply.use_cachedLogical. If
TRUE, find and return previously saved results. If cached results cannot be found, continue as ifuse_cachedwasFALSE.return_all_cached_repsLogical. If
FALSE(default), returns only the fit results for the requestedn_reps. IfTRUE, returns fit results for the requestedn_repsplus any additional cached replicates from the (DGP,Method) combinations in theExperiment. Note that even ifreturn_all_cached_reps = TRUE, only then_repsreplicates are used when evaluating and visualizing theExperiment.saveIf
TRUE, save outputs to disk.checkpoint_n_repsThe number of experiment replicates to compute before saving results to disk. If 0 (the default), no checkpoints are saved.
verboseLevel of verbosity. Default is 1, which prints out messages after major checkpoints in the experiment. If 2, prints additional debugging information for warnings and messages from user-defined functions (in addition to error debugging information). If 0, no messages are printed other than user-defined function error debugging information.
...Not used.
Returns
A named list of results from the simulation experiment with the following entries:
- fit_results
A tibble containing results from the
fitmethod. In addition to results columns, has columns named '.rep', '.dgp_name', '.method_name', and thevary_acrossparameter names if applicable.- eval_results
A list of tibbles containing results from the
evaluatemethod, which evaluates eachEvaluatorin theExperiment. Length of list is equivalent to the number ofEvaluators.- viz_results
A list of tibbles containing results from the
visualizemethod, which visualizes eachVisualizerin theExperiment. Length of list is equivalent to the number ofVisualizers.
Method generate_data()
Generate sample data from all DGP objects that were added
to the Experiment, including their varied params. Primarily useful
for debugging. Note that results are not generated in parallel.
Returns
A list of length equal to the number of DGPs in the
Experiment. If the Experiment does not have a
vary_across component, then each element in the list is a list
of n_reps datasets generated by the given DGP. If the
Experiment does have a vary_across component, then each
element in the outermost list is a list of lists. The second layer of
lists corresponds to a specific parameter setting within the
vary_across scheme, and the innermost layer of lists is of
length n_reps with the dataset replicates, generated by the
DGP.
Method fit()
Fit Methods in the Experiment across all
DGPs for n_reps repetitions and return results from fits.
Usage
Experiment$fit(
n_reps = 1,
parallel_strategy = "reps",
future.globals = NULL,
future.packages = NULL,
future.seed = TRUE,
use_cached = FALSE,
return_all_cached_reps = FALSE,
save = FALSE,
checkpoint_n_reps = 0,
verbose = 1,
...
)Arguments
n_repsThe number of replicates of the
Experimentfor this run.parallel_strategyA vector with some combination of "reps", "dgps", or "methods". Determines how computation will be distributed across available resources. Currently only the default, "reps", is supported.
future.globalsCharacter vector of names in the global environment to pass to parallel workers. Passed as the argument of the same name to
future.apply::future_lapplyand related functions. To set for all runs of the experiment, use the same argument during initialization.future.packagesCharacter vector of packages required by parallel workers. Passed as the argument of the same name to
future.apply::future_lapplyand related functions. To set for all runs of the experiment, use the same argument during initialization.future.seedPassed as the argument of the same name in
future.apply::future_apply.use_cachedLogical. If
TRUE, find and return previously saved results. If cached results cannot be found, continue as ifuse_cachedwasFALSE.return_all_cached_repsLogical. If
FALSE(default), returns only the fit results for the requestedn_reps. IfTRUE, returns fit results for the requestedn_repsplus any additional cached replicates from the (DGP,Method) combinations in theExperiment.saveIf
TRUE, save outputs to disk.checkpoint_n_repsThe number of experiment replicates to compute before saving results to disk. If 0 (the default), no checkpoints are saved.
verboseLevel of verbosity. Default is 1, which prints out messages after major checkpoints in the experiment. If 2, prints additional debugging information for warnings and messages from user-defined functions (in addition to error debugging information). If 0, no messages are printed other than user-defined function error debugging information.
...Additional
future.*arguments to pass to future.apply functions. Seefuture.apply::future_lapply()andfuture.apply::future_mapply().
Method evaluate()
Evaluate the performance of method(s) across all
Evaluator objects in the Experiment and return results.
Arguments
fit_resultsA tibble, as returned by
fit_experiment().use_cachedLogical. If
TRUE, find and return previously saved results. If cached results cannot be found, continue as ifuse_cachedwasFALSE.saveIf
TRUE, save outputs to disk.verboseLevel of verbosity. Default is 1, which prints out messages after major checkpoints in the experiment. If 2, prints additional debugging information for warnings and messages from user-defined functions (in addition to error debugging information). If 0, no messages are printed other than user-defined function error debugging information.
...Not used.
Method visualize()
Visualize the performance of methods and/or its evaluation metrics
using all Visualizer objects in the Experiment and return
visualization results.
Usage
Experiment$visualize(
fit_results,
eval_results = NULL,
use_cached = FALSE,
save = FALSE,
verbose = 1,
...
)Arguments
fit_resultsA tibble, as returned by
fit_experiment().eval_resultsA list of result tibbles, as returned by
evaluate_experiment().use_cachedLogical. If
TRUE, find and return previously saved results. If cached results cannot be found, continue as ifuse_cachedwasFALSE.saveIf
TRUE, save outputs to disk.verboseLevel of verbosity. Default is 1, which prints out messages after major checkpoints in the experiment. If 2, prints additional debugging information for warnings and messages from user-defined functions (in addition to error debugging information). If 0, no messages are printed other than user-defined function error debugging information.
...Not used.
Method update_dgp()
Update a DGP object in the Experiment.
Method remove_dgp()
Remove a DGP object from the Experiment.
Method get_dgps()
Retrieve the DGP objects associated with the Experiment.
Method add_method()
Add a Method object to the Experiment.
Method update_method()
Update a Method object in the Experiment.
Method remove_method()
Remove a Method object from the Experiment.
Method get_methods()
Retrieve the Method objects associated with the Experiment.
Method add_evaluator()
Add an Evaluator object to the Experiment.
Method update_evaluator()
Update an Evaluator object in the Experiment.
Method remove_evaluator()
Remove an Evaluator object from the Experiment.
Method get_evaluators()
Retrieve the Evaluator objects associated with the Experiment.
Method add_visualizer()
Add a Visualizer object to the Experiment.
Method update_visualizer()
Update a Visualizer object in the Experiment.
Method remove_visualizer()
Remove a Visualizer object from the Experiment.
Method get_visualizers()
Retrieve the Visualizer objects associated with the Experiment.
Method add_vary_across()
Arguments
.dgpName of
DGPto vary in theExperiment. Can also be aDGPobject that matches one in theExperimentor even a vector/list ofDGPnames/objects, assuming they all support the target arguments provided via.....methodName of
Methodto vary in theExperiment. Can also be aMethodobject that matches one in theExperimentor even a vector/list ofMethodnames/objects, assuming they all support the target arguments provided via.......Any number of named arguments where names match an argument in the user-specified
DGPorMethodfunction and values are vectors (for scalar parameters) or lists (for arbitrary parameters).
Method update_vary_across()
Update a vary_across component in the Experiment.
Arguments
.dgpName of
DGPto vary in theExperiment. Can also be aDGPobject that matches one in theExperimentor even a vector/list ofDGPnames/objects, assuming they all support the target arguments provided via.....methodName of
Methodto vary in theExperiment. Can also be aMethodobject that matches one in theExperimentor even a vector/list ofMethodnames/objects, assuming they all support the target arguments provided via.......Any number of named arguments where names match an argument in the user-specified
DGPorMethodfunction and values are vectors (for scalar parameters) or lists (for arbitrary parameters).
Method remove_vary_across()
Remove a vary_across component in the Experiment.
Arguments
dgpName of
DGPto vary in theExperiment. Can also be aDGPobject that matches one in theExperimentor even a vector/list ofDGPnames/objects.methodName of
Methodto vary in theExperiment. Can also be aMethodobject that matches one in theExperimentor even a vector/list ofMethodnames/objects.param_namesA character vector of parameter names to remove. If not provided, the entire set of
vary_acrossparameters will be removed for the specifiedDGP/Method.
Method get_vary_across()
Retrieve the parameters to vary across for each DGP and
Method in the Experiment.
Method clear_cache()
Clear (or delete) cached results from an Experiment to
start the experiment fresh/from scratch.
Method get_cached_results()
Read in cached results from disk from a previously saved
Experiment run.
Arguments
results_typeCharacter string indicating the type of results to read in. Must be one of "experiment", "experiment_cached_params", "fit", "eval", or "viz".
verboseLevel of verbosity. Default is 1, which prints out messages after major checkpoints in the experiment. If 2, prints additional debugging information for warnings and messages from user-defined functions (in addition to error debugging information). If 0, no messages are printed other than user-defined function error debugging information.
Returns
The cached results, specifically the cached Experiment object
if results_type = "experiment", the cached fit results if
results_type = "fit", the cached evaluation results if
results_type = "eval", the cached visualization results if
results_type = "viz", and the experiment parameters used in
the cache if results_type = "experiment_cached_params".
Method set_doc_options()
Set R Markdown options for Evaluator or Visualizer
outputs in the summary report. Some options include the height/width of
plots and number of digits to show in tables.
Arguments
field_nameOne of "evaluator" or "visualizer".
nameName of
EvaluatororVisualizerto set R Markdown options.showIf
TRUE, show output; ifFALSE, hide output in R Markdown report. DefaultNULLdoes not change the "doc_show" field inEvaluator/Visualizer.nrowsMaximum number of rows to show in the
Evaluator's results table in the R Markdown report. IfNULL, shows all rows. Default does not change the "doc_nrows" field in theEvaluator. Argument is ignored iffield_name = "visualizer"....Named R Markdown options to set. If
field_name = "visualizer", options are "height" and "width". Iffield_name = "evaluator", see options forvthemes::pretty_DT().
Method get_save_dir()
Get the directory in which the Experiment's results and
visualizations are saved.
Method set_save_dir()
Set the directory in which the Experiment's results and
visualizations are saved.
Method save()
Save the Experiment object to a .rds file under the
Experiment's results directory (see get_save_dir()).
Method export_visualizers()
Export all cached Visualizer results from an
Experiment to images in the viz_results/ directory under the
Experiment's results directory (see get_save_dir()).
Arguments
deviceSee
deviceargument ofggplot2::ggsave().widthSee
widthargument ofggplot2::ggsave().heightSee
heightargument ofggplot2::ggsave()....Additional arguments to pass to
ggplot2::ggsave().
Method print()
Print the Experiment in a nice format, showing the
DGP, Method, Evaluator, Visualizers, and varied parameters
involved in the Experiment.