The covr 4.0.0 report of the Rust code of b64, in dark mode with the colorblind palette

covr v4.0.0

The covr 4.0.0 report of the Rust code of b64, in dark mode with the colorblind palette

covr v4.0.0

I’m very excited to announce that covr 4.0.0 is now on CRAN! covr tracks the test coverage of the R, C, C++, Fortran and now Rust code in your R packages, R scripts and Shiny apps.

You can install it from CRAN with:

install.packages("covr")

Test coverage tells you how much of your code runs when you run your tests, which helps you find the paths your tests never check. A package with 100% coverage can certainly still have bugs, but coverage is a valuable tool both while you develop a package and when you review a package to decide whether to depend on it.

Since the first CRAN release in May 2015, covr has become the standard tool for test coverage in R. 2,345 CRAN packages list it in their DESCRIPTION, and it is mentioned in more than 25,000 DESCRIPTION files on GitHub.

This is by far the largest release of covr since that first one. It closes 99 issues with 112 pull requests, which change about 12,000 lines of R and C code and add 15,000 lines of tests. This post covers the changes to how coverage is counted, how much faster covr is, and the new things it can do: branch coverage, Rust code, Shiny apps, better reports, and new tools for CI. See the change log for a full list of changes.

Breaking changes

Coverage is now counted more precisely, so the coverage reported for your package may be slightly higher or lower than with the previous release. In particular, because of the way R adds source references to expressions, covr treated an if differently depending on whether its branches had braces. Take two functions that do the same thing, tested with half(4) and half_braces(4):

half <- function(x) if (x > 0) x / 2 else 0
half_braces <- function(x) if (x > 0) { x / 2 } else { 0 }
covr 3.6.5 covr 4.0.0
half() 100% 0%
half_braces() 0% 0%

covr 3.6.5 gave the branches of half_braces() their own counters, but counted all of half() as one expression, so the else that never ran was reported as covered. covr 4.0.0 counts both the same way, so you can write your code in whichever style you prefer without it changing your coverage. For most packages the difference is small.

There are a few smaller breaking changes, e.g. covr now requires R 4.1.0, and codecov() now fails your CI when an upload fails; see the change log for the full list.

Improved performance

Running your tests

Previous versions of covr counted which expressions ran by adding calls to a small R function throughout your code, which kept the counts in an R environment. covr 4.0.0 uses a counter written in C instead, which uses less memory and takes about 0.4 microseconds per counted expression, rather than 1.3–1.4 microseconds.

I also changed when covr adds these counters. It used to add them each time an R process loaded your package, and R then had to compile the modified functions again as they ran. covr 4.0.0 adds the counters and byte-compiles the package once, when package_coverage() installs it, so each process that loads the package starts faster (0.3 seconds rather than 4.6 for Luminescence).

Together, these changes make measuring coverage faster, sometimes much faster. This plot shows the time to run the tests of a selection of packages without coverage (grey), with covr 3.6.5 (orange) and with covr 4.0.0 (blue). The labels show how much covr 4.0.0 adds to the time without coverage:

Time to run each package's tests without coverage, with covr 3.6.5 and with covr 4.0.0, three runs each

For 10 of these 13 packages, measuring coverage with covr 4.0.0 adds between 7% and 45% to the time of the tests, where covr 3.6.5 often doubled or tripled it. data.table, for example, went from 2.4 minutes with covr 3.6.5 to 48 seconds, against 28 seconds without coverage.

Packages with shorter tests gained less. Of the 10 packages I measured whose tests take less than 30 seconds with coverage, all were within 7% of covr 3.6.5, except posterior and roxygen2, which were 23% and 16% faster.

These timings exclude installing the package, and were measured with R 4.6.2 on Linux. The scripts and full results are in the covr repository.

Reports and other output

I also spent time improving the functions you run after coverage is measured, in particular report(), which creates an HTML report of the coverage. Previous versions of report() built the table of each file iteratively, by repeatedly calling into htmltools for every line. covr 4.0.0 writes the table directly as HTML, which is 6 to 230 times faster on the packages I measured. For Luminescence, report() went from 4.5 minutes to 1.2 seconds, and its peak memory from 1.2 GB to 310 MB. For igraph, covr 3.6.5’s report() had not finished after nearly an hour; covr 4.0.0 takes 10 seconds.

Other functions that summarize or export coverage are also faster. Across 23 packages, the median speedup ranges from about 6 times for tally_coverage(), percent_coverage() and codecov() to 16–20 times for sonarqube() and cobertura(), with print() in between at 9 times.

Tests that start other R processes

covr has been able to measure the coverage of R processes started by your tests for a long time, but it merged their coverage in memory, and slowly. For packages whose tests start hundreds or thousands of R processes, such as languageserver, this slowed things down considerably: its tests took 9 minutes without coverage, but 80 minutes with covr 3.6.5, and covr then ran out of memory adding up the coverage of the 1,000 processes, needing more than 7 GB. With covr 4.0.0 the tests finish in about 13 minutes, and adding up their coverage takes a second and 144 MB. I have also made writing the coverage of these processes more robust, so packages whose coverage used to fail, such as mlr3 and future, now work:

Package Without coverage covr 3.6.5 covr 4.0.0
mlr3 177 s fails after 673 s 247 s (+40%)
future 230 s fails after 1,176 s 281 s (+22%)

(With covr 4.0.0, the tests of future also needed a small fix to a test that relied on how covr 3.6.5 set up the library paths of R processes.)

Branch coverage

One of the longest-standing open issues in covr, opened in August 2017, was branch coverage. Branch coverage tracks which branches of each conditional were taken, such as the TRUE and FALSE cases of an if. Line coverage tells you whether the code on a line ran, but not which way each if went, including the FALSE case of an if without an else, which runs no code at all. covr 4.0.0 can now measure branch coverage, with the experimental options(covr.branch_coverage = TRUE). Take these two functions in a package:

clamp <- function(x, lo = 0, hi = 1) {
  if (x < lo) return(lo)
  if (x > hi) return(hi)
  x
}

half <- function(x) if (x > 0) x / 2 else 0

whose tests call clamp(0.5), clamp(2) and half(4):

options(covr.branch_coverage = TRUE)
cov <- package_coverage()

tally_coverage(cov, by = "branch")
#>   functions line column branch label value
#> 1     clamp    2      3      1  TRUE     0
#> 2     clamp    2      3      2 FALSE     2
#> 3     clamp    3      3      1  TRUE     1
#> 4     clamp    3      3      2 FALSE     1
#> 5      half    7     21      1  TRUE     1
#> 6      half    7     21      2 FALSE     0

No test makes x negative, so the first if in clamp() never took its TRUE branch, and the else in half() was never taken either. (The filename column is left out of the output above.)

print(cov, missing = TRUE) and report() also show branch coverage. In the report, a line is marked as partly covered if one of its branches was never taken. Branch coverage is also included in the output of lcov(), cobertura(), codecov() and coveralls().

Branch coverage is off by default and still experimental, so what counts as a branch may change. See ?covr.branch_coverage for the details.

Rust

covr has long measured the coverage of C, C++ and Fortran code in packages, and covr 4.0.0 extends this to Rust. It works with packages built with extendr or savvy, or with any other setup that builds Rust code with cargo.

Rust coverage uses LLVM’s source-based coverage tools. Install them for your Rust toolchain with rustup component add llvm-tools, and package_coverage() then includes your Rust code with no other changes. For example, here is the coverage of the b64 package, whose report is shown below:

package_coverage("b64")
#> b64 Coverage: 22.18%
#> R/config.R: 0.00%
#> R/engine.R: 17.65%
#> src/rust/src/lib.rs: 18.46%
#> R/encode.R: 31.58%
#> R/extendr-wrappers.R: 35.29%
#> R/alphabet.R: 57.14%
#> src/entrypoint.c: 100.00%

Rust coverage isn’t measured on Windows yet.

The number of CRAN packages with Rust code is still relatively small, about 95 today, but I expect it to keep growing, and I hope being able to measure the coverage of your Rust code makes using it in your packages more appealing.

Shiny apps

covr could technically measure the coverage of Shiny apps for a long time; I made a proof of concept as far back as 2017. But it was difficult to set up, because the code of a Shiny app runs in a separate R process. In addition, many Shiny apps are not part of a package, so they did not fit into covr’s existing functions.

For Shiny apps in a package, package_coverage() now includes the app code run by shinytest2 tests. For apps outside a package, the new shiny_app_coverage() runs your shinytest2 tests and reports the coverage of the app.

For example, take this small app in app/app.R, which describes the number you enter:

library(shiny)

ui <- fluidPage(
  numericInput("n", "Pick a number", value = 1),
  textOutput("size")
)

describe <- function(n) {
  if (is.na(n)) {
    "not a number"
  } else if (n < 0) {
    "negative"
  } else if (n %% 2 == 0) {
    "even"
  } else {
    "odd"
  }
}

server <- function(input, output, session) {
  output$size <- renderText(describe(input$n))
}

shinyApp(ui, server)

and its shinytest2 test in app/tests/testthat/test-app.R:

test_that("the app describes even numbers", {
  app <- shinytest2::AppDriver$new()
  app$set_inputs(n = 4)
  expect_equal(app$get_value(output = "size"), "even")
  app$stop()
})

shiny_app_coverage() runs the test, which starts the app in another R process, and returns the coverage of the code the app ran. shinytest2 skips its tests on CRAN, so set NOT_CRAN when running them yourself:

Sys.setenv(NOT_CRAN = "true")
app_cov <- shiny_app_coverage("app")
#> [ FAIL 0 | WARN 0 | SKIP 0 | PASS 1 ]

print(app_cov, missing = TRUE)
#> Coverage: 75.00%
#> app.R: 75.00%  missed 10, 12

The app ran describe() twice, once for the starting value of 1 and once for 4, so the “odd” and “even” lines are covered, but nothing tested a missing or negative number (lines 10 and 12). See the Shiny vignette for more details, including apps in packages.

Additional features

file_coverage() and devtools’ test_coverage_active_file() now work for C, C++, Fortran and Rust code as well as R code. They load the package with pkgload::load_all() and build an instrumented copy of it, recompiling only the files that changed since the last run, so you get a tight development cycle and can work on compiled code the same way as on R code:

file_coverage("src/foo.c", "tests/testthat/test-foo.R")

Another long-standing open issue, from 2015, was that functions stored inside lists and other objects, such as the methods of ggproto objects, could not be traced. They now can, if you add Config/covr/full_traversal: true to your DESCRIPTION.

Improved reports

Apart from being much faster, report() no longer needs DT, jQuery or Bootstrap. It lists the least covered files first, keeps the Files and Source tabs at the top as you scroll, and has a new dark mode.

The default colors are green and red, which are hard to tell apart for people with red-green color blindness. You can now choose other colors for the report and the console, including a colorblind-friendly palette based on Okabe-Ito:

options(covr.report_colors = covr::colorblind_palette)

The new report_html() returns the report as HTML, so you can embed it in R Markdown or Quarto documents. Here is the report for b64, a small package that uses Rust through extendr, measured with branch coverage and the colorblind palette. Click src/rust/src/lib.rs to see the coverage of its Rust code, or use the button in the corner to switch to the dark theme:

(Open it in a new tab.)

CI and pull requests

The new badge() writes an SVG coverage badge, so you can show your coverage in your README without relying on an external service like Codecov or Coveralls. covr includes an example GitHub Actions workflow that publishes it.

For packages that do use those services, codecov() now uses Codecov’s current upload API, and coveralls() detects the CI service, so it now works natively on GitHub Actions.

lcov() writes coverage in the popular LCOV format, and coverage_to_markdown() creates a Markdown table of your coverage, e.g. for a GitHub Actions job summary or a pull request comment.

For use locally or by coding agents, print(cov, missing = TRUE) and annotate_coverage() show the lines your tests missed as plain text. For the functions in the branch coverage example:

annotate_coverage(cov)
#> R/clamp.R
#>   1 |      | clamp <- function(x, lo = 0, hi = 1) {
#>   2 |  !!0 |   if (x < lo) return(lo)  <- not taken: TRUE
#>   3 |    1 |   if (x > hi) return(hi)
#>   4 |    1 |   x
#>   5 |      | }
#>   6 |      |
#>   7 |  !!0 | half <- function(x) if (x > 0) x / 2 else 0  <- not taken: FALSE

Finally, c() now combines coverage from separate runs, such as tests run in several CI jobs, without any additional code.

Thanks

The first commit of covr was on December 19th, 2014, when my oldest daughter was only 4 months old. She is now almost 12 and a half, and it is amazing to look back at how she has grown, and how the community and the use of covr have grown in the nearly 12 years since that first commit. Thank you for being part of the R community and for your interest in covr; I hope this release makes it an even stronger choice for your package development.

Thank you to the 70 contributors who made this release possible by opening issues or submitting pull requests:

@Ahill25, @akersting, @al-obrien, @ALanguillaume, @amaltawfik, @amar-v1, @B0ydT, @bestdan, @billdenney, @Bisaloo, @BjarkeHautop, @brodieG, @cjyetman, @daattali, @dblodgett-usgs, @dgkf, @dmurdoch, @DSGeoff, @erkurtharun, @fangly, @frederikfabriciusbjerre, @gergness, @gowerc, @hadley, @hsloot, @IndrajeetPatil, @jd4ds, @jessekps, @JohannesFriedrich, @jonthegeek, @josherrickson, @JT-39, @katrinabrock, @kbodwin, @kevinushey, @kforner, @king-of-poppk, @kongdd, @krlmlr, @liu1029, @llrs-roche, @maksymiuks, @MichaelChirico, @michaelquinn32, @mjakubczak, @mjfrigaard, @MLopez-Ibanez, @moodymudskipper, @mpadge, @Naikless, @nicholascarey, @noamross, @ocschwar, @olivroy, @pdbailey0, @roaldarbol, @schaffstein, @sclewis23, @SebKrantz, @stla, @StuartWheater, @t-kalinowski, @tdhock, @thierrymoudiki, @thomasrockhu, @vjcitn, @wkdavis, @wolski, @xec-cm, and @zhuxr11.


Starting in 2027 I’ll be doing consulting for the first time, on R package development, testing and performance. If you have projects you think I would be a good fit for, I’d love to hear about them; please write to me at james.f.hester@gmail.com.

Avatar
Jim Hester
Software Engineer

I’m a Senior Software Engineer at Netflix and R package developer.

Related