Create a new Evaluator.

Create a new Evaluator.

Usage

create_evaluator(
  .eval_fun,
  .name = NULL,
  .rmd_options = list(),
  .rmd_show = TRUE,
  ...
)

Arguments

.eval_fun

The evaluation function.

.name

(Optional) The name of the Evaluator. The argument must be specified by position or typed out in whole; no partial matching is allowed for this argument.

.rmd_options

(Optional) List of options to control the aesthetics of the displayed Evaluator's results table in the knitted R Markdown report. See vthemes::pretty_DT() for possible options. The argument must be specified by position or typed out in whole; no partial matching is allowed for this argument.

.rmd_show

If TRUE (default), show Evaluator's results as a table in the R Markdown report; if FALSE, hide output in the R Markdown report.

...

Arguments to pass into .eval_fun().

Value

A new Evaluator object.

Details

When evaluating or running the Experiment (see evaluate_experiment() or run_experiment()), the named arguments fit_results and vary_params are automatically passed into the Evaluator function .eval_fun() and serve as placeholders for the fit_experiment() results (i.e., the results from the method fits) and the name of the varying parameter, respectively. To evaluate the performance of a method(s) fit then, the Evaluator function .eval_fun() should almost always take in the named argument fit_results. See Experiment$fit() or fit_experiment() for details on the format of fit_results. If the Evaluator is used for Experiments with varying parameters, vary_params should be used as a stand in for the name of this varying parameter.

Examples

# create an example Evaluator function
reject_prob_fun <- function(fit_results, vary_params = NULL, alpha = 0.05) {
  group_vars <- c(".dgp_name", ".method_name", vary_params)
  eval_out <- fit_results %>%
    dplyr::group_by(across({{group_vars}})) %>%
    dplyr::summarise(
      `X1 Reject Prob.` = mean(`X1 p-value` < alpha),
      `X2 Reject Prob.` = mean(`X2 p-value` < alpha)
    )
  return(eval_out)
}

# create Evaluator using the default arguments (i.e., alpha = 0.05)
reject_prob_eval <- create_evaluator(.eval_fun = reject_prob_fun, 
                                     .name = "Rejection Prob (alpha = 0.05)")
# create Evaluator using non-default arguments (here, alpha = 0.1)
reject_prob_eval2 <- create_evaluator(.eval_fun = reject_prob_fun,
                                      .name = "Rejection Prob (alpha = 0.1)",
                                      # additional named parameters to pass to reject_prob_fun(),
                                      alpha = 0.1)

# create Evaluator from a function in the built-in Evaluator library
pred_err_eval <- create_evaluator(.eval_fun = summarize_pred_err,
                                  .name = "Prediction Error", 
                                  # additional named parameters to pass to summarize_pred_err()
                                  truth_col = "y", estimate_col = "predictions")
                                  
# set rmd options for displaying Evaluator in Rmd report to show 3 decimal points
pred_err_eval <- create_evaluator(.eval_fun = summarize_pred_err,
                                  .name = "Prediction Error", 
                                  .rmd_options = list(digits = 3),
                                  # additional named parameters to pass to summarize_pred_err()
                                  truth_col = "y", estimate_col = "predictions")