mutator

Overview

mutator is an automated mutation testing tool for the R language. It applies mutation testing principles to help developers improve test suite quality by introducing small, systematic changes (mutations) to source code and verifying if tests can detect these changes.

For instance, imagine you have the following function f in your package:

f <- function(x) {
  if (x > 0) {
    return(x + 1)
  } else {
    return(x - 1)
  }
}

mutator will generate several mutants, including:

f <- function(x) {
  if (x < 0) { # Mutated comparison operator
    return(x + 1)
  } else {
    return(x - 1)
  }
}

In this mutant, > has been replaced with <.

If your test suite does not catch this change, it indicates that your tests may not be comprehensive enough.

mutator will compute how many mutants survives, i.e. the test suite does not fail for the mutant, and how many mutants are killed, i.e. the test suite fails for the mutant. The ratio of killed mutants to total mutants is called the mutation score and is a measure of test suite effectiveness.

For a broader introduction to the tool, see the useR! 2026 talk on mutator.

Features

• Comprehensive Mutation Testing: Applies various mutation operators to R source code
• AST-Based Mutation: Uses Abstract Syntax Tree analysis for intelligent code mutations
• Parallel Test Execution: Runs tests in parallel for improved performance
• Coverage-guided Test Selection: Runs only the tests that cover mutated lines, also for improved performance
• Configurable Test Harness: First-class testthat and tinytest support, plus a generic installed-tests fallback for other layouts
• Timeout Management: Automatically calibrates timeouts for mutant test runs to prevent hangs
• Annotations and Exclusions: Allows developers to exclude specific files or code sections from mutation testing
• Equivalent Mutant Detection: Identify mutants that are functionally equivalent to the original code (currently, using LLMs)
• Detailed Reporting: Provides mutation scores and analysis of test suite effectiveness

Installation

# Install the released version from CRAN
install.packages("mutator")

Development version

# Install the development version from GitHub; package dependencies are installed automatically
# install.packages("remotes")
remotes::install_github("PRL-PRG/mutator")

# Or install pre-built binaries of the development version from R-universe
# (https://prl-prg.r-universe.dev)
install.packages("mutator", repos = c(
  "https://prl-prg.r-universe.dev",
  "https://cloud.r-project.org"
))

# Or install from local source, in an R console
# install.packages("devtools")
setwd("path/to/mutator")
devtools::install()

Quick Start Guide

library(mutator)

# Mutate a single file
mutants <- mutate_file("path/to/your/file.R")

# Optional: cap returned mutants by random selection
mutants <- mutate_file("path/to/your/file.R", max_mutants = 20)

# Mutate an entire package and run tests
result <- mutate_package("path/to/your/package")

# Optional: cap tested mutants across the whole package
result <- mutate_package("path/to/your/package", max_mutants = 100)

# Optional: set a fixed timeout (seconds) per mutant test run
result <- mutate_package("path/to/your/package", timeout_seconds = 60)

# Optional: control where mutant files are written
result <- mutate_package("path/to/your/package", mutation_dir = tempdir())

mutator, in addition to show you the number of generated mutants, the surviving ones, and the mutation score, returns an invisible list with four components, a list of the generated mutants, a list of mutant outcomes, a list of phase durations, and a summary of the mutation testing run.

Mutant outcomes are reported as:

• SURVIVED: tests passed for the mutant
• KILLED: tests failed (or execution error)
• HANG: mutant exceeded timeout

See the pkgdown reference for the full argument and return-value documentation.

Continuous integration (GitHub Actions)

mutator ships a reusable workflow so any R-package repository can run mutation testing in CI without copying scripts. Add a caller workflow at .github/workflows/mutation-testing.yaml:

on:
  pull_request:
  push:
    branches: [main, master]

name: mutation-testing

jobs:
  mutation:
    uses: PRL-PRG/mutator/.github/workflows/mutation-testing.yaml@v0.3.0
    with:
      target-margin: "0.10" # sample to +/-10 percentage points
      fail-under: "75" # fail CI below a 75% mutation score

Pin to a released tag such as @v0.3.0; the workflow is versioned with the mutator package, so the tag matches the package version. It installs mutator from CRAN and falls back to the matching GitHub tag when CRAN cannot provide it; mutator-source: r-universe or mutator-source: github picks another source. Set deploy-badge: true (with contents: write permission) to publish a shields.io badge. See the Continuous integration vignette for every input, threshold guidance, and badge setup. From an installed copy, open it with vignette("continuous-integration", package = "mutator").

Configuration

mutate_package() exposes a number of options to control how mutants are run, which tests are selected, and how results are refined. Each is covered in depth in the Configuration vignette (from an installed copy, open it with vignette("configuration", package = "mutator")):

• Test strategy selection (strategy): how mutator auto-detects a package's test harness (testthat, tinytest, or the generic installed-tests fallback) and when to override it with tinytest-installed.
• Timeouts and the contended baseline: how the per-mutant HANG timeout is self-calibrated from a parallelism-aware baseline, and timeout_seconds to override it.
• CRAN mode (cran): run the tests CRAN would (skipping guarded slow/network tests) or the full suite.
• Fail-fast (fail_fast): stop each mutant's run at the first failing test.
• Parallel execution and isolation (isolate, cores): symlink-vs-copy of the package tree and how to handle non-hermetic tests, plus the optional pbmcapply progress bar.
• Excluding code from mutation: exclude_files, in-source # mutator:ignore-* directives, covr # nocov annotations, and .covrignore.
• Coverage-guided test selection (coverage_guided, coverage_backend): on by default, runs only the tests that cover each mutated line (testthat and tinytest strategies; warns and runs the full suite under the generic installed-tests fallback).
• Precise mutant locations: the optional imputesrcref package for narrower operator-mutant source ranges.
• Equivalent mutant detection (detectEqMutants): configuring the OpenAI-compatible API used to flag equivalent mutants.

Mutation Operators

mutation_operators() lists the operators. The operators argument of mutate_file() and mutate_package() selects them by id or family, and - removes one, e.g. operators = c("default", "seq_idiom", "-bool_flip").

On by default

Operator	Mutation
arith_swap	+ ↔ -, * ↔ /
arith_extra	^ → *, %% ↔ %/%, -x → x
rel_swap	< ↔ >, <= ↔ >=, == ↔ !=
rel_boundary	< ↔ <=, > ↔ >=
logic_swap	& ↔ |, && ↔ ||
not_remove	!x → x
and_or_operand	a && b → a and → b (same for ||)
cond_negate	if (c), while (c) → if (!(c)), while (!(c))
loop_ctrl	break ↔ next
return_null	return(x) → return(NULL)
na_replace	Constant → NA of the same type, e.g. 1 → NA_real_
const_null	Constant → NULL
bool_flip	TRUE ↔ FALSE, including argument defaults: f(x, na.rm = TRUE) → f(x, na.rm = FALSE)
super_assign	x <<- v → x <- v
named_arg_drop	Drops an argument such as na.rm, drop, sep, collapse or fixed, so that its default applies
fun_swap	any ↔ all, min ↔ max, pmin ↔ pmax, which.min ↔ which.max, head ↔ tail, nrow ↔ ncol, isTRUE ↔ isFALSE, sub ↔ gsub
error_handling	tryCatch(e, ...), try(e), withCallingHandlers(e, ...) → e; stop(), stopifnot() → invisible(NULL)
stmt_delete	Deletes a statement of a { } block
line_delete	Deletes a source line (budget: max_line_deletions, 0 by default in mutate_package())

Opt-in

Operators are opt-in when they create too many equivalent mutants (more than 20%) or too many mutants in general.

We ran each operator on seven packages (prettyunits, stringr, forcats, scales, jsonlite, lumberjack, R.methodsS3; about 10,600 source lines), with up to 100 sampled mutants per operator and package: 9,414 mutants, 3,996 of which survived. An LLM (qwen3.8-flash-next) judged whether each survivor is equivalent to the original code. We checked 56 of these verdicts by hand: 45 of the 46 "equivalent" verdicts were correct. The benchmark script is benchmarks/operator_selection.R.

Operator	Mutation	Why it is opt-in
cond_force	if (c) → if (TRUE), if (FALSE)	Overlaps with cond_negate; 142 mutants per 1,000 source lines
const_off_by_one	Number n → n + 1, n - 1	185 mutants per 1,000 source lines
value_42	Numbers, assigned values and call results → 42 (or 0)	860 mutants per 1,000 source lines, mostly killed by type errors
string_empty	Non-empty string → ""	Mostly changes error and warning messages, which tests rarely check
na_type_swap	NA ↔ NA_integer_ ↔ NA_real_ ↔ NA_character_	62% of survivors are equivalent, e.g. any NA assigned into a character vector
scalar_vector_logic	&& ↔ &, || ↔ |	67% equivalent: operands are usually scalars
scalar_vector_fun	max(a, b) ↔ pmax(a, b), min ↔ pmin	83% equivalent: arguments are usually scalars
index_ops	x[[i]] ↔ x[i]	47% equivalent: same result for one element of an atomic vector
seq_idiom	seq_len(n) → 1:n, seq_along(x) → 1:length(x)	35% equivalent (lengths that cannot be zero); the other survivors show untested empty inputs
drop_idiom	m[i, , drop = FALSE] → m[i, ]	Rare; all 3 survivors were equivalent
fun_swap_extra	paste ↔ paste0, floor ↔ ceiling, sapply → lapply, ...	30% equivalent, e.g. paste vs paste0 with a single argument
call_unwrap	rev(x), unique(x), as.numeric(x), ... → x	24% equivalent: the wrapper often does not change the value

Dependencies

mutator depends on:

• R Packages:
  • pkgload: For loading mutated package copies
  • testthat: For test execution
  • xml2: For running C++ tests through testthat::run_cpp_tests()
  • covr: For coverage-guided test selection (coverage_guided = TRUE)
  • future and furrr: For parallel execution
  • callr: For subprocess test execution and hard timeouts
  • R6: For the per_file coverage backend reporter
  • httr and jsonlite: For OpenAI API integration
  • cli: For progress bars and user feedback
  • pkgbuild: For building package copies for mutation testing
  • pbmcapply: For progress bars in parallel execution
  • imputesrcref: For precise mutant source ranges (optional)
• LinkingTo: testthat (for Catch2 C++ test headers)
• C++17: For native mutation engine implementation

mutator's test suite

mutator itself uses testthat for its own R tests and testthat + Catch2 for C++ tests.

• C++ tests are located in src/test-*.cpp
• The C++ test runner is src/test-runner.cpp
• C++ tests are executed from tests/testthat/test-cpp.R via run_cpp_tests("mutator")

Run the full test suite with:

devtools::test()

System tests

mutator includes a set of system tests that run mutation testing on 9 packages from CRAN. It compares the mutants, surviving, killed, hanged mutant detections, and the mutation score against the expected results using snapshot testing, with a fixed seed on 10 mutants or 50 mutants per package depending on the release cycle. It also compares the results across a set of configuration options that ought not to change the results. The system tests are located in tests/system and are automatically run as par of the CI.

Mutation testing

We practice dogfooding and run mutator on mutator test suite itself. This already led to numerous improvements in mutator speed, as waiting for the mutation score to be computed on a package with a long running test suite can be frustrating!