Developer function to construct basic Evaluator.

Helper function for developing a new Evaluator that evaluates some function (e.g., metric) for each row in fit_results.

Usage

eval_constructor(
  fit_results,
  vary_params = NULL,
  fun,
  nested_cols = NULL,
  ...,
  group_cols = NULL,
  fun_options = NULL,
  na_rm = FALSE
)

Arguments

fit_results

A tibble, as returned by fit_experiment().

vary_params

A vector of DGP or Method parameter names that are varied across in the Experiment.

fun

Function to compute for each row in fit_results. This function can take in the following optional arguments: (1) data = fit_results[i, ]; (2) na_rm; (3) arguments specified via ... and fun_options.

nested_cols

(Optional) A character string or vector specifying the name of the column(s) in fit_results that need to be unnested before evaluating results. Default is NULL, meaning no columns in fit_results need to be unnested prior to computation.

...

Named arguments, containing names of columns to pass to fun.

group_cols

(Optional) A character string or vector specifying the column(s) to group rows by before evaluating metrics. This is useful for assessing within-group metrics.

fun_options

Named list of additional arguments to pass to fun.

na_rm

A logical value indicating whether NA values should be stripped before the computation proceeds.

Examples

# generate example fit_results data for a regression problem
fit_results <- tibble::tibble(
  .rep = rep(1:2, times = 2),
  .dgp_name = c("DGP1", "DGP1", "DGP2", "DGP2"),
  .method_name = c("Method"),
  # true response
  y = lapply(1:4, FUN = function(x) rnorm(100)),
  # predicted response
  predictions = lapply(1:4, FUN = function(x) rnorm(100))
)

# evaluate root mean squared error for each row in fit_results
rmse_fun <- function(data, truth_col, estimate_col, na_rm = TRUE) {
  yardstick::rmse_vec(
    data[[truth_col]], data[[estimate_col]], na_rm = na_rm
  )
}
eval_results <- eval_constructor(
  fit_results = fit_results,
  fun = rmse_fun,
  truth_col = "y",
  estimate_col = "predictions"
) %>%
  tidyr::unnest(.eval_result)