Skip to contents

Evaluate various feature selection metrics, given the true feature support and the estimated feature support. eval_feature_selection_err() evaluates the various feature selection metrics for each experimental replicate separately.. summarize_feature_selection_err() summarizes the various feature selection metrics across experimental replicates.

Usage

eval_feature_selection_err(
  fit_results,
  vary_params = NULL,
  nested_data = NULL,
  truth_col,
  estimate_col = NULL,
  imp_col,
  metrics = NULL,
  na_rm = FALSE
)

summarize_feature_selection_err(
  fit_results,
  vary_params = NULL,
  nested_data = NULL,
  truth_col,
  estimate_col = NULL,
  imp_col,
  metrics = NULL,
  na_rm = FALSE,
  summary_funs = c("mean", "median", "min", "max", "sd", "raw"),
  custom_summary_funs = NULL,
  eval_id = "feature_selection"
)

Arguments

fit_results

A tibble, as returned by the fit method.

vary_params

A vector of parameter names that are varied across in the Experiment.

nested_data

(Optional) Character string. If specified, should be the name of the column in fit_results containing columns that must be unnested before evaluating results. Default is NULL, meaning no columns in fit_results need to be unnested prior to computation.

truth_col

A character string identifying the column in fit_results with the true feature support data. Each element in this column should be an array of length p, where p is the number of features. Elements in this array should be binary with TRUE or 1 meaning the feature (corresponding to that slot) is in the support and FALSE or 0 meaning the feature is not in the support.

estimate_col

An (optional) character string identifying the column in fit_results with the estimated feature support data. Each element in this column should be an array of length p, where p is the number of features and the feature order aligns with that of truth_col. Elements in this array should be binary with TRUE or 1 meaning the feature (corresponding to that slot) is in the estimated support and FALSE or 0 meaning the feature is not in the estimated support. If NULL (default), the non-zero elements of imp_col are used as the estimated feature support.

imp_col

A character string identifying the column in fit_results with the estimated feature importance data. Each element in this column should be an array of length p, where p is the number of features and the feature order aligns with that of truth_col. Elements in this array should be numeric where a higher magnitude indicates a more important feature.

metrics

A metric_set object indicating the metrics to evaluate. See yardstick::metric_set() for more details. Default NULL will evaluate the following: number of true positives (tp), number of false positives (fp), sensitivity (sens), specificity (spec), positive predictive value (ppv), number of features in the estimated support (pos), number of features not in the estimated support (neg), AUROC (roc_auc), and AUPRC (pr_auc). If na_rm = TRUE, the number of NA values (num_na) is also computed.

na_rm

A logical value indicating whether NA values should be stripped before the computation proceeds.

summary_funs

Character vector specifying how to summarize evaluation metrics. Must choose from a built-in library of summary functions - elements of the vector must be one of "mean", "median", "min", "max", "sd", "raw".

custom_summary_funs

Named list of custom functions to summarize results. Names in the list should correspond to the name of the summary function. Values in the list should be a function that takes in one argument, that being the values of the evaluated metrics.

eval_id

Character string. ID to be used as a suffix when naming result columns. Default NULL does not add any ID to the column names.

Value

The output of eval_feature_selection_err() is a tibble with the following columns:

.rep

Replicate ID.

.dgp_name

Name of DGP.

.method_name

Name of Method.

.metric

Name of the evaluation metric.

.estimate

Value of the evaluation metric.

as well as any columns specified by vary_params. The output of summarize_feature_selection_err() is a grouped tibble containing both identifying information and the feature selection results aggregated over experimental replicates. Specifically, the identifier columns include .dgp_name, .method_name, any columns specified by vary_params, and .metric. In addition, there are results columns corresponding to the requested statistics in summary_funs and custom_summary_funs. These columns end in the suffix "_feature_selection".

Examples

# generate example fit_results data for a feature selection problem
fit_results <- tibble::tibble(
  .rep = rep(1:2, times = 2),
  .dgp_name = c("DGP1", "DGP1", "DGP2", "DGP2"),
  .method_name = c("Method"),
  feature_info = lapply(
    1:4,
    FUN = function(i) {
      tibble::tibble(
        # feature names
        feature = c("featureA", "featureB", "featureC"),  
        # true feature support
        true_support = c(TRUE, FALSE, TRUE),  
        # estimated feature support
        est_support = c(TRUE, FALSE, FALSE),  
        # estimated feature importance scores
        est_importance = c(10, runif(2, min = -2, max = 2))  
      )
    }
  )
)

# evaluate feature selection (using all default metrics) for each replicate
eval_results <- eval_feature_selection_err(
  fit_results,
  nested_data = "feature_info",
  truth_col = "true_support",
  estimate_col = "est_support",
  imp_col = "est_importance"
)
# summarize feature selection error (using all default metric) across replicates
eval_results_summary <- summarize_feature_selection_err(
  fit_results,
  nested_data = "feature_info",
  truth_col = "true_support",
  estimate_col = "est_support",
  imp_col = "est_importance"
)

# evaluate/summarize feature selection errors using specific yardstick metrics
metrics <- yardstick::metric_set(yardstick::sens, yardstick::spec)
eval_results <- eval_feature_selection_err(
  fit_results,
  nested_data = "feature_info",
  truth_col = "true_support",
  estimate_col = "est_support",
  imp_col = "est_importance",
  metrics = metrics
)
eval_results_summary <- summarize_feature_selection_err(
  fit_results,
  nested_data = "feature_info",
  truth_col = "true_support",
  estimate_col = "est_support",
  imp_col = "est_importance",
  metrics = metrics
)

# summarize feature selection errors using specific summary metric
range_fun <- function(x) return(max(x) - min(x))
eval_results_summary <- summarize_feature_selection_err(
  fit_results,
  nested_data = "feature_info",
  truth_col = "true_support",
  estimate_col = "est_support",
  imp_col = "est_importance",
  custom_summary_funs = list(range_feature_selection = range_fun)
)