Evaluate and/or summarize ROC or PR curves for feature rankings, ranked by p-value.

Evaluate the ROC or PR curves corresponding to the feature importances as ranked by their p-values. eval_testing_curve() evaluates the ROC or PR curve for each experimental replicate separately. summarize_testing_curve() summarizes the ROC or PR curve across experimental replicates.

Usage

eval_testing_curve(
  fit_results,
  vary_params = NULL,
  nested_data = NULL,
  truth_col,
  pval_col,
  curve = c("ROC", "PR"),
  options = list(),
  na_rm = FALSE
)

summarize_testing_curve(
  fit_results,
  vary_params = NULL,
  nested_data = NULL,
  truth_col,
  pval_col,
  curve = c("ROC", "PR"),
  options = list(),
  na_rm = FALSE,
  x_grid = seq(0, 1, by = 0.01),
  summary_funs = c("mean", "median", "min", "max", "sd", "raw"),
  custom_summary_funs = NULL,
  eval_id = ifelse(curve == "PR", "precision", "TPR")
)

Arguments

fit_results: A tibble, as returned by the fit method.
vary_params: A vector of parameter names that are varied across in the Experiment.
nested_data: (Optional) Character string. If specified, should be the name of the column in fit_results containing columns that must be unnested before evaluating results. Default is NULL, meaning no columns in fit_results need to be unnested prior to computation.
truth_col: A character string identifying the column in fit_results with the true feature support data. Each element in this column should be an array of length p, where p is the number of features. Elements in this array should be binary with TRUE or 1 meaning the feature (corresponding to that slot) is in the support and FALSE or 0 meaning the feature is not in the support.
pval_col: A character string identifying the column in fit_results with the estimated p-values data. Each element in this column should be an array of length p, where p is the number of features and the feature order aligns with that of truth_col.
curve: Either "ROC" or "PR" indicating whether to evaluate the ROC or Precision-Recall curve.
options: A list of named options to pass to pROC::roc() such as smooth. These options should not include response, predictor, levels, quiet, or direction. This argument is only used when computing the ROC and is ignored otherwise.
na_rm: A logical value indicating whether NA values should be stripped before the computation proceeds.
x_grid: Vector of values between 0 and 1 at which to evaluate the ROC or PR curve. If curve = "ROC", the provided vector of values are the FPR values at which to evaluate the TPR, and if curve = "PR", the values are the recall values at which to evaluate the precision.
summary_funs: Character vector specifying how to summarize evaluation metrics. Must choose from a built-in library of summary functions - elements of the vector must be one of "mean", "median", "min", "max", "sd", "raw".
custom_summary_funs: Named list of custom functions to summarize results. Names in the list should correspond to the name of the summary function. Values in the list should be a function that takes in one argument, that being the values of the evaluated metrics.
eval_id: Character string. ID to be used as a suffix when naming result columns. Default NULL does not add any ID to the column names.

Value

The output of eval_testing_curve() is a tibble with the following columns:

.rep: Replicate ID.
.dgp_name: Name of DGP.
.method_name: Name of Method.
curve_estimate: A list of tibbles with x and y coordinate values for the ROC/PR curve for the given experimental replicate. If curve = "ROC", the tibble has the columns .threshold, FPR, and TPR for the threshold, false positive rate, and true positive rate, respectively. If curve = "PR", the tibble has the columns .threshold, recall, and precision.

as well as any columns specified by vary_params. The output of summarize_testing_curve() is a grouped tibblecontaining both identifying information and the evaluation curve results aggregated over experimental replicates. Specifically, the identifier columns include .dgp_name, .method_name, and any columns specified by vary_params. In addition, there are results columns corresponding to the requested statistics in summary_funs and custom_summary_funs. If curve = "ROC", these results columns include FPR and others that end in the suffix "_TPR". If curve = "PR", the results columns include recall and others that end in the suffix "_precision".

Examples

# generate example fit_results data for a feature selection problem
fit_results <- tibble::tibble(
  .rep = rep(1:2, times = 2),
  .dgp_name = c("DGP1", "DGP1", "DGP2", "DGP2"),
  .method_name = c("Method"),
  feature_info = lapply(
    1:4,
    FUN = function(i) {
      tibble::tibble(
        # feature names
        feature = c("featureA", "featureB", "featureC"),
        # true feature support
        true_support = c(TRUE, FALSE, TRUE),
        # estimated p-values
        pval = 10^(sample(-3:0, 3, replace = TRUE))
      )
    }
  )
)

# evaluate feature selection ROC/PR curves for each replicate
roc_results <- eval_testing_curve(
  fit_results,
  curve = "ROC",
  nested_data = "feature_info",
  truth_col = "true_support",
  pval_col = "pval"
)
pr_results <- eval_testing_curve(
  fit_results,
  curve = "PR",
  nested_data = "feature_info",
  truth_col = "true_support",
  pval_col = "pval"
)
# summarize feature selection ROC/PR curves across replicates
roc_summary <- summarize_testing_curve(
  fit_results,
  curve = "ROC",
  nested_data = "feature_info",
  truth_col = "true_support",
  pval_col = "pval"
)
pr_summary <- summarize_testing_curve(
  fit_results,
  curve = "PR",
  nested_data = "feature_info",
  truth_col = "true_support",
  pval_col = "pval"
)

Evaluate and/or summarize ROC or PR curves for feature rankings, ranked by p-value.

Usage

Arguments

Value

See also

Examples