Guest blog contributed by Ning Leng, People and Product Leader, Roche-Genentech. Eric Nantz, Director, Eli Lilly and Company; Ben Straub, Principal Programmer, GSK; Sam Parmar, Statistical Data Scientist, Pfizer.
Supporting the science of drug development requires computational tools with careful implementations of core statistical functions and data structures. The R programming language, a general purpose language developed by statisticians that grows dynamically through the contributions of a worldwide community of developers is a common choice for serious statistical work. However, managing new versions of the core R language and the hundreds of specialized libraries (called packages in R) necessary to support multiple development groups in a way that ensures the consistency, reproducibility, and reliability of results poses many practical challenges
The FDA, for example, requires that the software and tools supporting a clinical trial submission are capable of producing reproducible results for an extended period of time. This means submitting code based on a version of R that is sufficiently tested and stable yet new enough to support the critical R packages over the required FDA time horizon.
So, how is the R environment release managed across different healthcare organizations? We interviewed individuals from different pharma companies to learn their internal approaches to keep their R environment up-to-date and secure.
- Roche’s Scientific Computing Environment is container based, with clinical reporting done from managed qualified images being released twice per year – roughly timed to capture the last update to an R major version (April release) and a 6 month later update (September release). For each image, R packages undergo a mostly automated risk assessment to document package quality. Automated indicators of package quality include test coverage, thoroughness of documentation, test coverage of exported objects (using covtracer), and may be supplemented with package adoption measured using download counts, author reputation or other peripheral knowledge of the package’s history. Prior to internal publication, a representative sample of reverse dependencies are re-evaluated to safeguard against breaking changes. If the package meets our quality criteria it is published to a continuously updated repository of validated packages corresponding to the image’s R minor version (e.g. x.x). This allows flexibility for teams to roll forward to newer releases of packages within a managed release by moving their renv snapshot to a later date, easing the transition between bi-annual image releases. A generalized version of Roche’s automated process has been open sourced as ‘theValidator‘, and more details on the Roche process was shared within the R Validation case studies series.
- Eli Lilly currently updates its qualified R environment only after a new major release of R is available and the corresponding release of Bioconductor (utilizing that version of R) is also available. In a new release, all packages currently installed from the CRAN and Bioconductor repositories are refreshed to their latest versions at the time of the release. Once the new R version is deployed, all packages are frozen for that particular release to ensure stability and reproducibility. Lilly maintains multiple R versions for backward compatibility. Only packages available on CRAN or Bioconductor are permitted for installation in the central package library. Lilly uses a hybrid approach of automation and risk-based assessment when a new package is requested for installation. In the event that a new version of a package is necessary for a project (such as a new Shiny application), the users are encouraged to leverage the renv package created by Posit to create a project-based environment which will not impact the central package library. As technology evolves and the R language becomes more prominent in clinical data analysis, Lilly continues to assess the current and future possibilities of a robust clinical computing environment primed for innovation while adhering to the foundational principles of reproducibility and transparency.
- GSK releases “frozen R environments” for clinical reporting work on a 6-12 month cycle. The choice of R version is the latest stable release with at least one patch release of R, e.g. 4.3.1 rather than 4.3.0. As R itself is stable with decades of use, we prefer to focus on package assessment for building of our “frozen R environments.” Packages for this environment can be from external sources (CRAN, Bioconductor) or internally built at GSK, regardless of origin we assess the same way. We pay close attention to author qualification and institutional backing, types and breadth of testing, documentation and examples, and the practice of software development life cycle practices. Once a package is approved in this process it will be included in the frozen environment. Packages change over time, if substantial changes are implemented in the packages, then we re-assess with a focus on those changes for allowing up-versioning of the package in the frozen environment. These frozen environments ensure that clinical reporting can be easily reproduced if needed as all packages versions and the version of R used during the analysis are contained in the frozen environment.
- Pfizer releases one new R version every year. We typically target R-x.y.1 releases to pick up patches – so we might consider this a “stable” release. The process of testing, documenting, and deploying R into validated containers is performed every 6 months, with a new release of R once per year, e.g. R-4.3.1, and an update to package set and package versions 6 months later (for the same R version). We take a snapshot date of CRAN to form the basis of our package set for the container build. We try to balance competing priorities of getting latest versions of packages and newest package releases, while maintaining a snapshot and version-controlled release to ensure reproducibility and stability.
Here is what we have: four companies and four somewhat complex bespoke solutions. It seems likely that if we interviewed a hundred representatives from a hundred different companies we would get at least a hundred different solutions. It is also not difficult to imagine that multiple protocols for managing R and package versions imposed a fairly complex project management solution on the FDA as it simultaneously deals with submissions from multiple sponsors.
In the R Consortium’s R Submissions Work Group meeting we have been discussing whether there might be a simple solution for at least dealing with the R versioning problem that might serve as a de facto standard for the industry. One suggestion that has gained some traction is that sponsors use the previous minor and latest patched R version for a submission. For example, if R version 4.4.0 is currently available then it is suggested that a sponsor uses the latest patch version (4.3.z). If R version 4.5.0 becomes available, then it is suggested that a sponsor uses the latest patch version (4.4.z). This ensures that the minor version is stable and most likely available to all stakeholders. Of course, if a version change eliminates a security problem, that might be preferred. (Note that R versions are organized as R x.y.z where, x is the major version, y is the minor version, and z is the patch version.)
We would love to hear what you think. Please, go to Issue number 117 on the GitHub repository of our working group and leave a comment.