R6 class representing an evaluator

Evaluator which can evaluate() the performance of methods in an Experiment.

Generally speaking, users won't directly interact with the Evaluator R6 class, but instead indirectly through create_evaluator() and the following Experiment helpers:

• add_evaluator()

• update_evaluator()

• remove_evaluator()

• get_evaluators()

• evaluate_experiment()

Details

When evaluating or running the Experiment (see evaluate_experiment() or run_experiment()), the named arguments fit_results and vary_params are automatically passed into the Evaluator function .eval_fun() and serve as placeholders for the fit_experiment() results (i.e., the results from the method fits) and the name of the varying parameter(s), respectively.

To evaluate the performance of a method(s) fit then, the Evaluator function .eval_fun() should almost always take in the named argument fit_results. See Experiment$fit() or fit_experiment() for details on the format of fit_results. If the Evaluator is used for Experiments with varying parameters, vary_params should be used as a stand in for the name of this varying parameter(s).

See also

create_evaluator

Public fields

name

The name of the Evaluator.

eval_fun

The user-defined evaluation function.

eval_params

A (named) list of default parameters to input into the evaluator function.

doc_options

List of options to control the aesthetics of the displayed Evaluator's results table in the knitted R Markdown report.

doc_show

Boolean indicating whether or not to show the Evaluator's results as a table in the R Markdown report.

doc_nrows

Maximum number of rows to show in the Evaluator's results table in the R Markdown report. If the number of rows in the table exceeds this value, the table will be truncated and a message will be displayed indicating the number of rows that were omitted. Default is NULL, which shows all rows.

Methods

Public methods

• Evaluator$new()

• Evaluator$evaluate()

• Evaluator$print()

• Evaluator$clone()

---

Method new()

Initialize a new Evaluator object.

Usage

Evaluator$new(
  .eval_fun,
  .name = NULL,
  .doc_options = list(),
  .doc_show = TRUE,
  .doc_nrows = NULL,
  ...
)

Arguments

.eval_fun

The user-defined evaluation function.

.name

(Optional) The name of the Evaluator, helpful for later identification. The argument must be specified by position or typed out in whole; no partial matching is allowed for this argument.

.doc_options

(Optional) List of options to control the aesthetics of the displayed Evaluator's results table in the knitted R Markdown report. See vthemes::pretty_DT() for possible options. The argument must be specified by position or typed out in whole; no partial matching is allowed for this argument.

.doc_show

If TRUE (default), show Evaluator's results as a table in the R Markdown report; if FALSE, hide output in the R Markdown report.

.doc_nrows

Maximum number of rows to show in the Evaluator's results table in the R Markdown report. If the number of rows in the table exceeds this value, the table will be truncated and a message will be displayed indicating the number of rows that were omitted. Default is NULL, which shows all rows.

...

User-defined arguments to pass into .eval_fun().

Returns

A new instance of Evaluator.

---

Method evaluate()

Evaluate the performance of method(s) in the Experiment using the Evaluator and its given parameters.

Usage

Evaluator$evaluate(fit_results, vary_params = NULL, ...)

Arguments

fit_results

A tibble, as returned by fit_experiment().

vary_params

A vector of DGP or Method parameter names that are varied across in the Experiment.

...

Not used.

Returns

Result of Evaluator$eval_fun(), coerced into a tibble.

---

Method print()

Print an Evaluator in a nice format, showing the Evaluator's name, function, parameters, and R Markdown options.

Usage

Evaluator$print()

Returns

The original Evaluator object, invisibly.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

Evaluator$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

# create DGP
dgp_fun <- function(n, beta, rho, sigma) {
  cov_mat <- matrix(c(1, rho, rho, 1), byrow = TRUE, nrow = 2, ncol = 2)
  X <- MASS::mvrnorm(n = n, mu = rep(0, 2), Sigma = cov_mat)
  y <- X %*% beta + rnorm(n, sd = sigma)
  return(list(X = X, y = y))
}
dgp <- create_dgp(.dgp_fun = dgp_fun,
                  .name = "Linear Gaussian DGP",
                  n = 50, beta = c(1, 0), rho = 0, sigma = 1)

# create Method
lm_fun <- function(X, y, cols) {
  X <- X[, cols]
  lm_fit <- lm(y ~ X)
  pvals <- summary(lm_fit)$coefficients[-1, "Pr(>|t|)"] %>%
    setNames(paste(paste0("X", cols), "p-value"))
  return(pvals)
}
lm_method <- create_method(
  .method_fun = lm_fun,
  .name = "OLS",
  cols = c(1, 2)
)

# create Experiment
experiment <- create_experiment() %>%
  add_dgp(dgp) %>%
  add_method(lm_method) %>%
  add_vary_across(.dgp = dgp, rho = seq(0.91, 0.99, 0.02))

fit_results <- fit_experiment(experiment, n_reps=10)
#> Fitting experiment...
#> 10 reps completed (totals: 10/10) | time taken: 0.336091 minutes
#> ==============================

# create an example Evaluator function
reject_prob_fun <- function(fit_results, vary_params = NULL, alpha = 0.05) {
  fit_results[is.na(fit_results)] <- 1
  group_vars <- c(".dgp_name", ".method_name", vary_params)
  eval_out <- fit_results %>%
    dplyr::group_by(across({{group_vars}})) %>%
    dplyr::summarise(
      n_reps = dplyr::n(),
      `X1 Reject Prob.` = mean(`X1 p-value` < alpha),
      `X2 Reject Prob.` = mean(`X2 p-value` < alpha)
    )
  return(eval_out)
}

reject_prob_eval <- Evaluator$new(.eval_fun = reject_prob_fun,
                                  .name = "Rejection Prob (alpha = 0.05)")

reject_prob_eval$evaluate(fit_results, vary_params = "rho")
#> `summarise()` has grouped output by '.dgp_name', '.method_name'. You can
#> override using the `.groups` argument.
#> # A tibble: 5 × 6
#>   .dgp_name        .method_name   rho n_reps `X1 Reject Prob.` `X2 Reject Prob.`
#>   <chr>            <chr>        <dbl>  <int>             <dbl>             <dbl>
#> 1 Linear Gaussian… OLS           0.91     10               0.7               0  
#> 2 Linear Gaussian… OLS           0.93     10               0.7               0.1
#> 3 Linear Gaussian… OLS           0.95     10               0.5               0  
#> 4 Linear Gaussian… OLS           0.97     10               0.3               0.1
#> 5 Linear Gaussian… OLS           0.99     10               0.2               0.1