A BiocParallelParam class using Futures

Henrik Bengtsson

Source: vignettes/FutureParam.md.rsp

Introduction

The future package provides a generic API for using futures in R. A future is a simple yet powerful mechanism to evaluate an R expression and retrieve its value at some point in time. Futures can be resolved in many different ways depending on which strategy is used. There are various types of synchronous and asynchronous futures to choose from in the future package. Additional futures are implemented in other packages. For instance, the future.batchtools package provides futures for any type of backend that the batchtools package supports. For an introduction to futures in R, please consult the vignettes of the future package.

The BiocParallel.FutureParam package provides FutureParam, a BiocParallelParam class, for the BiocParallel package that works with any type of future. The BiocParallel.FutureParam package is cross platform just as the future package.

Below is an example showing how to use FutureParam with multicore futures. A multicore future will be evaluated in parallel using forked workers, which is not supported on MS Windows when it will fall back to sequential processing.

library("BiocParallel.FutureParam")
register(FutureParam())
plan(multicore)

mu <- 1.0
sigma <- 2.0
x <- bplapply(1:3, mu = mu, sigma = sigma, function(i, mu, sigma) {
  rnorm(i, mean = mu, sd = sigma)
})

FutureParam replaces existing BiocParallelParam classes

Due to the generic nature of futures, the FutureParam class provides the same functionality as many of the existing BiocParallelParam classes, e.g. SerialParam, SnowParam, MulticoreParam, BatchtoolsParam and DoparParam. In addition, it provides supports for additional backends that are not yet implemented in BiocParallel, e.g. callr and batchtools.

BiocParallel usage BiocParallel.FutureParam alternative 
library("BiocParallel")
register(SerialParam())

library("BiocParallel.FutureParam")
register(FutureParam())
plan(sequential)
 
library("BiocParallel")
register(MulticoreParam())

library("BiocParallel.FutureParam")
register(FutureParam())
plan(multicore)
 
library("BiocParallel")
register(SnowParam(2, type = "SOCK"))

library("BiocParallel.FutureParam")
register(FutureParam())
plan(multisession, workers = 2)
 
library("BiocParallel")
cl <- parallel::makeCluster(2, type = "SOCK")
register(as(cl, "SnowParam"))

library("BiocParallel.FutureParam")
register(FutureParam())
cl <- parallel::makeCluster(2, type = "SOCK")
plan(cluster, workers = cl)
 
library("BiocParallel")
register(SnowParam(4, type = "MPI"))


library("BiocParallel.FutureParam")
register(FutureParam())
cl <- parallel::makeCluster(4, type = "MPI")
plan(cluster, workers = cl)
 
library("BiocParallel")
register(BatchtoolsParam(cluster="sge",
                         template="~/sge.tmpl"))
 
library("BiocParallel.FutureParam")
register(FutureParam())
plan(future.batchtools::batchtools_sge,
     template = "~/sge.tmpl")
N/A 
library("BiocParallel.FutureParam")
register(FutureParam())
plan(future.callr::callr)

Something not working?

Please note that this package, BiocParallel.FutureParam, is in an experimental stage and does not get as much real-world use as other BiocParallel backends. Thus, if you run into a problem when using this package, it could very well be a bug. However, before you report the problem, please try with the doFuture, registerDoFuture(), and the DoparParam() backend of BiocParallel, e.g.

library("BiocParallel")
library("doFuture")
register(DoparParam()) ## Tell BiocParallel to use a foreach backend
registerDoFuture()     ## Tell foreach to use a future backend
plan(multicore)        ## Tell future to use the multicore backend

mu <- 1.0
sigma <- 2.0
x <- bplapply(1:3, mu = mu, sigma = sigma, function(i, mu, sigma) {
  rnorm(i, mean = mu, sd = sigma)
})

If that works, but not with register(FutureParam()), then it’s a bug in the BiocParallel.FutureParam package. Please report this at https://github.com/HenrikBengtsson/BiocParallel.FutureParam/issues.