Building R 4+ for Windows with OpenBLAS

This post outlines the steps needed to build R 4+ for Windows with OpenBLAS. The release of R 4.0 includes significant changes to the Windows build system from prior versions—for the better! Before anything, we all owe Jeroen Ooms significant gratitude for the many hours he spent working on the build system. Thank you, Jeroen!!

The build process below creates an installation file for both 32-bit and 64-bit R but makes some compromises. Extra time spent building the installer is traded for reduced time spent adjusting files and minimizing needed file changes. The OpenBLAS library itself is built using the “dynamic” keyword so it will work for all architectures, and it is built to use up to 64 threads. It isn’t completely optimized to the actual building CPU, which is not only reasonable but necessary. A centralized library must be generic enough to serve all comers. Eventually, I hope to refine the process to build just the 64 bit version and to link to a static, locally-compiled, and locally-optimized OpenBLAS library. Lastly, I intend to address the steps needed to add specific packages from source like XML or nloptr in a future post. For now, installation as binaries should work.

Step 1: Install Rtools40

Installing Rtools40 is straightforward. Make sure to download the 64 bit version and add the PATH variable as described on the website.

Step 2: Install the necessary support software

Install MiKTeX and Inno Setup to their default locations if they are not already. Install qpdf anywhere if you wish to use it.

Step 3: Obtaining the R-windows build scripts

Clone or download the r-windows/r-base repository from Github. Personally, I download the ZIP file. I tend to monkey around a lot with the code while figuring out how to customize the build, so I do not want to clone/fork the original repository. When I am ready to suggest changes, I clone the repository and send a pull request. Regardless, there should be a directory with the files listed at the r-base repository, likely called r-base-master.

Step 4: Create blas patch

The build uses the same process which works in R3+; it hooks OpenBLAS into the build via the ATLAS variables in the Makefiles. However, the new system downloads the R source code as part of the build process, so changes to src/extra/blas/Makefile.win are made using patch file instead of manual edits. Therefore, save the following text as a file named blas.diff in the same subdirectory as the extracted/cloned files.

--- a/src/extra/blas/Makefile.win
+++ b/src/extra/blas/Makefile.win
@@ -12,7 +12,7 @@
 ../../../$(BINDIR)/Rblas.dll: blas00.o ../../gnuwin32/dllversion.o
 	@$(ECHO) -------- Building $@ --------
 	$(DLL) -shared $(DLLFLAGS) -o $@ $^ Rblas.def \
-	   -L../../../$(IMPDIR) -lR  -L"$(ATLAS_PATH)" -lf77blas -latlas
+	   -L../../../$(IMPDIR) -lR  -L"$(ATLAS_PATH)" -lopenblas
 else
 ../../../$(BINDIR)/Rblas.dll: blas.o cmplxblas.o ../../gnuwin32/dllversion.o
 	@$(ECHO) -------- Building $@ --------

--- a/src/extra/blas/Makefile.win

+++ b/src/extra/blas/Makefile.win

@@ -12,7 +12,7 @@

../../../$(BINDIR)/Rblas.dll: blas00.o ../../gnuwin32/dllversion.o

@$(ECHO) -------- Building $@ --------

$(DLL) -shared $(DLLFLAGS) -o $@ $^ Rblas.def \

- -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -lf77blas -latlas

+ -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -lopenblas

else

../../../$(BINDIR)/Rblas.dll: blas.o cmplxblas.o ../../gnuwin32/dllversion.o

@$(ECHO) -------- Building $@ --------

Step 5: Adjust existing files

Make the following changes to the build files to ensure that OpenBLAS is pulled from pacman (the package manager, not the Namco character) and that the proper libraries are accessed at the right times.

full-build.sh

Add the highlighted line to full-build.sh after the call to cairo, tk, and curl. This is where other packages like nlopt or xml will be added eventually too.

pacman -S --needed --noconfirm mingw-w64-{i686,x86_64}-{cairo,tk,curl}
pacman -S --needed --noconfirm mingw-w64-{i686,x86_64}-openblas

1 2	pacman -S --needed --noconfirm mingw-w64-{i686,x86_64}-{cairo,tk,curl} pacman -S --needed --noconfirm mingw-w64-{i686,x86_64}-openblas

MkRules.local.in

Edit EOPTS (extra optimization flags) in MkRules.local.in as in prior versions of R. Personally, I use -march=native -pipe as I build bespoke R installers for each machine I use. To use the same installer on different computers I recommend -mtune=generic -pipe. See the GCC 8.3 documentation for more on optimization flags.
Next, add and edit the following lines under #Enable features:

USE_ATLAS = YES
ATLAS_PATH="/mingw$(WIN)/lib/"

1 2	USE_ATLAS = YES ATLAS_PATH="/mingw$(WIN)/lib/"

This tells the build to use the OpenBLAS libraries and where to find both the 32 and 64 bit versions.
To use qpdf, add the following under # For building docs/installer, substituting the proper path.

QPDF = /path/to/qpdf

1	QPDF = /path/to/qpdf

PKGBUILD

Adjusting this file is more complicated. Add the highlighted lines below to PKGBUILD in their appropriate locations:

# Maintainer: Jeroen Ooms <jeroen@berkeley.edu>
_realname=r-installer
pkgbase=${_realname}
pkgname="${_realname}"
pkgver=4.0.9000
pkgrel=1
pkgdesc="The R Programming Language"
arch=('any')
makedepends=("${MINGW_PACKAGE_PREFIX}-bzip2"
             "${MINGW_PACKAGE_PREFIX}-gcc"
             "${MINGW_PACKAGE_PREFIX}-gcc-fortran"
             "${MINGW_PACKAGE_PREFIX}-cairo"
             "${MINGW_PACKAGE_PREFIX}-curl"
             "${MINGW_PACKAGE_PREFIX}-icu"
             "${MINGW_PACKAGE_PREFIX}-libtiff"
             "${MINGW_PACKAGE_PREFIX}-libjpeg"
             "${MINGW_PACKAGE_PREFIX}-libpng"
             "${MINGW_PACKAGE_PREFIX}-pcre2"
             "${MINGW_PACKAGE_PREFIX}-tcl"
             "${MINGW_PACKAGE_PREFIX}-tk"
             "${MINGW_PACKAGE_PREFIX}-xz"
             "${MINGW_PACKAGE_PREFIX}-zlib"
             "${MINGW_PACKAGE_PREFIX}-openblas"
             "texinfo"
             "texinfo-tex"
             "sed")
options=('staticlibs')
license=("GPL")
url="https://www.r-project.org/"

# Default source is R-devel (override via $rsource_url)
source=(R-source.tar.gz::"${rsource_url:-https://cran.r-project.org/src/base-prerelease/R-devel.tar.gz}"
    https://curl.haxx.se/ca/cacert.pem
    MkRules.local.in
    shortcut.diff
    create-tcltk-bundle.sh
    blas.diff)

# Automatic untar fails due to embedded symlinks
noextract=(R-source.tar.gz)

sha256sums=('SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP')

prepare() {
  # Verify that InnoSetup is installed
  INNOSETUP="C:/Program Files (x86)/Inno Setup 6/ISCC.exe"
  msg2 "Testing for $INNOSETUP"
  test -f "$INNOSETUP"
  "$INNOSETUP" 2>/dev/null || true

  # Put pdflatex on the path (assume Miktex 2.9)
  msg2 "Checking if pdflatex and texindex can be found..."
  export PATH="$PATH:/c/progra~1/MiKTeX 2.9/miktex/bin/x64"
  pdflatex --version
  texindex --version

  # Extract tarball with symlink workarounds
  msg2 "Extracting R source tarball..."
  rm -rf ${srcdir}/R-source
  mkdir -p ${srcdir}/R-source
  MSYS="winsymlinks:lnk" tar -xf ${srcdir}/R-source.tar.gz -C ${srcdir}/R-source --strip-components=1
  cd "${srcdir}/R-source"

  # Ship the CA bundle
  cp "${srcdir}/cacert.pem" etc/curl-ca-bundle.crt

  # Ship the TclTk runtime bundle
  msg2 "Creating the TclTk runtime bundle"
  mkdir -p Tcl/{bin,bin64,lib,lib64}
  ${srcdir}/create-tcltk-bundle.sh  

  # Add your patches here
  patch -Np1 -i "${srcdir}/shortcut.diff"
  patch -Np1 -i "${srcdir}/blas.diff"
}

build() {
  msg2 "Copying source files for 32-bit build..."
  rm -Rf ${srcdir}/build32
  MSYS="winsymlinks:lnk" cp -Rf "${srcdir}/R-source" ${srcdir}/build32

  # Build 32 bit version
  msg2 "Building 32-bit version of base R..."
  cd "${srcdir}/build32/src/gnuwin32"
  sed -e "s|@win@|32|" -e "s|@texindex@||" -e "s|@home32@||" "${srcdir}/MkRules.local.in" > MkRules.local
  #make 32-bit SHELL='sh -x'
  make 32-bit
  
  # Build 64 bit + docs and installers
  msg2 "Building 64-bit distribution"
  cd "${srcdir}/R-source/src/gnuwin32"
  TEXINDEX=$(cygpath -m $(which texindex))  
  sed -e "s|@win@|64|" -e "s|@texindex@|${TEXINDEX}|" -e "s|@home32@|${srcdir}/build32|" "${srcdir}/MkRules.local.in" > MkRules.local
  make distribution
}

check(){
  # Use cloud mirror for CRAN unit test
  #export R_CRAN_WEB="https://cran.rstudio.com"

  # Run 64 bit checks in foreground
  cd "${srcdir}/R-source/src/gnuwin32"
  echo "===== 64 bit checks ====="
  make check-all
}

package() {
  # Derive output locations
  REVISION=$((read x; echo ${x:10}) < "${srcdir}/R-source/SVN-REVISION")
  CRANDIR="${srcdir}/R-source/src/gnuwin32/cran"

  # This sets TARGET variable
  $(sed -e 's|set|export|' "${CRANDIR}/target.cmd")

  # Copy CRAN release files
  cp "${srcdir}/R-source/SVN-REVISION" "${pkgdir}/SVN-REVISION.${target}"
  cp "${CRANDIR}/NEWS.${target}.html" ${pkgdir}/
  cp "${CRANDIR}/README.${target}" ${pkgdir}/

  # Determine which webpage variant to ship from target (for example "R-3.4.1beta")
  case "$target" in
  *devel|*testing)
    cp "${CRANDIR}/rdevel.html" "${pkgdir}/"
    ;;
  *patched|*alpha|*beta|*rc)
    cp "${CRANDIR}/rpatched.html" "${pkgdir}/"
    cp "${CRANDIR}/rtest.html" "${pkgdir}/"
    ;;
  R-4*)
    cp "${CRANDIR}/index.html" "${pkgdir}/"
    cp "${CRANDIR}/md5sum.txt" "${pkgdir}/"
    cp "${CRANDIR}/rw-FAQ.html" "${pkgdir}/"
    cp "${CRANDIR}/release.html" "${pkgdir}/"
    REVISION="$target"
    ;;
  *)
    echo "Unknown release type: $target"
    exit 1
    ;;
  esac

  # Helper for appveyor script
  echo "set revision=${REVISION}" >> "${CRANDIR}/target.cmd"
  cp "${CRANDIR}/target.cmd" ${pkgdir}/
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

# Maintainer: Jeroen Ooms <jeroen@berkeley.edu>

_realname=r-installer

pkgbase=${_realname}

pkgname="${_realname}"

pkgver=4.0.9000

pkgrel=1

pkgdesc="The R Programming Language"

arch=('any')

makedepends=("${MINGW_PACKAGE_PREFIX}-bzip2"

"${MINGW_PACKAGE_PREFIX}-gcc"

"${MINGW_PACKAGE_PREFIX}-gcc-fortran"

"${MINGW_PACKAGE_PREFIX}-cairo"

"${MINGW_PACKAGE_PREFIX}-curl"

"${MINGW_PACKAGE_PREFIX}-icu"

"${MINGW_PACKAGE_PREFIX}-libtiff"

"${MINGW_PACKAGE_PREFIX}-libjpeg"

"${MINGW_PACKAGE_PREFIX}-libpng"

"${MINGW_PACKAGE_PREFIX}-pcre2"

"${MINGW_PACKAGE_PREFIX}-tcl"

"${MINGW_PACKAGE_PREFIX}-tk"

"${MINGW_PACKAGE_PREFIX}-xz"

"${MINGW_PACKAGE_PREFIX}-zlib"

"${MINGW_PACKAGE_PREFIX}-openblas"

"texinfo"

"texinfo-tex"

"sed")

options=('staticlibs')

license=("GPL")

url="https://www.r-project.org/"

# Default source is R-devel (override via $rsource_url)

source=(R-source.tar.gz::"${rsource_url:-https://cran.r-project.org/src/base-prerelease/R-devel.tar.gz}"

https://curl.haxx.se/ca/cacert.pem

MkRules.local.in

shortcut.diff

create-tcltk-bundle.sh

blas.diff)

# Automatic untar fails due to embedded symlinks

noextract=(R-source.tar.gz)

sha256sums=('SKIP'

'SKIP'

'SKIP')

prepare() {

# Verify that InnoSetup is installed

INNOSETUP="C:/Program Files (x86)/Inno Setup 6/ISCC.exe"

msg2 "Testing for $INNOSETUP"

test -f "$INNOSETUP"

"$INNOSETUP" 2>/dev/null || true

# Put pdflatex on the path (assume Miktex 2.9)

msg2 "Checking if pdflatex and texindex can be found..."

export PATH="$PATH:/c/progra~1/MiKTeX 2.9/miktex/bin/x64"

pdflatex --version

texindex --version

# Extract tarball with symlink workarounds

msg2 "Extracting R source tarball..."

rm -rf ${srcdir}/R-source

mkdir -p ${srcdir}/R-source

MSYS="winsymlinks:lnk" tar -xf ${srcdir}/R-source.tar.gz -C ${srcdir}/R-source --strip-components=1

cd "${srcdir}/R-source"

# Ship the CA bundle

cp "${srcdir}/cacert.pem" etc/curl-ca-bundle.crt

# Ship the TclTk runtime bundle

msg2 "Creating the TclTk runtime bundle"

mkdir -p Tcl/{bin,bin64,lib,lib64}

${srcdir}/create-tcltk-bundle.sh

# Add your patches here

patch -Np1 -i "${srcdir}/shortcut.diff"

patch -Np1 -i "${srcdir}/blas.diff"

}

build() {

msg2 "Copying source files for 32-bit build..."

rm -Rf ${srcdir}/build32

MSYS="winsymlinks:lnk" cp -Rf "${srcdir}/R-source" ${srcdir}/build32

# Build 32 bit version

msg2 "Building 32-bit version of base R..."

cd "${srcdir}/build32/src/gnuwin32"

sed -e "s|@win@|32|" -e "s|@texindex@||" -e "s|@home32@||" "${srcdir}/MkRules.local.in" > MkRules.local

#make 32-bit SHELL='sh -x'

make 32-bit

# Build 64 bit + docs and installers

msg2 "Building 64-bit distribution"

cd "${srcdir}/R-source/src/gnuwin32"

TEXINDEX=$(cygpath -m $(which texindex))

sed -e "s|@win@|64|" -e "s|@texindex@|${TEXINDEX}|" -e "s|@home32@|${srcdir}/build32|" "${srcdir}/MkRules.local.in" > MkRules.local

make distribution

}

check(){

# Use cloud mirror for CRAN unit test

#export R_CRAN_WEB="https://cran.rstudio.com"

# Run 64 bit checks in foreground

cd "${srcdir}/R-source/src/gnuwin32"

echo "===== 64 bit checks ====="

make check-all

}

package() {

# Derive output locations

REVISION=$((read x; echo ${x:10}) < "${srcdir}/R-source/SVN-REVISION")

CRANDIR="${srcdir}/R-source/src/gnuwin32/cran"

# This sets TARGET variable

$(sed -e 's|set|export|' "${CRANDIR}/target.cmd")

# Copy CRAN release files

cp "${srcdir}/R-source/SVN-REVISION" "${pkgdir}/SVN-REVISION.${target}"

cp "${CRANDIR}/NEWS.${target}.html" ${pkgdir}/

cp "${CRANDIR}/README.${target}" ${pkgdir}/

# Determine which webpage variant to ship from target (for example "R-3.4.1beta")

case "$target" in

*devel|*testing)

cp "${CRANDIR}/rdevel.html" "${pkgdir}/"

;;

*patched|*alpha|*beta|*rc)

cp "${CRANDIR}/rpatched.html" "${pkgdir}/"

cp "${CRANDIR}/rtest.html" "${pkgdir}/"

;;

R-4*)

cp "${CRANDIR}/index.html" "${pkgdir}/"

cp "${CRANDIR}/md5sum.txt" "${pkgdir}/"

cp "${CRANDIR}/rw-FAQ.html" "${pkgdir}/"

cp "${CRANDIR}/release.html" "${pkgdir}/"

REVISION="$target"

;;

echo "Unknown release type: $target"

exit 1

;;

esac

# Helper for appveyor script

echo "set revision=${REVISION}" >> "${CRANDIR}/target.cmd"

cp "${CRANDIR}/target.cmd" ${pkgdir}/

}

Step 6: Build R

Launch Rtools40 via msys.exe which creates a shell window. Navigate to the subdirectory where the build files are using Unix style. For example, if the files are in “c:\R\R40”, navigate to “/c/R/R40”. Invoke the build via “./full-build.sh” which will probably run for hours. The process updates all the necessary components, builds both the 32 bit and 64 bit versions of R using OpenBLAS, checks the results for each version, and packages the build into an executable installer file. When done, the desired R-devel-win.exe will be either in the above subdirectory or in /src/R-source/src/gnuwin32/installer, if not both. This the executable which installs R 4+ for Windows with OpenBLAS!

Final notes, for now

Installing and using this modified version of R should pose no problems, nor should installing most packages from source. There are some packages which rely on compiled libraries—such as XML or nlopt—for which extra steps are needed to build from source. However, these are few and may be installed as binaries for now. I’d appreciate thoughts and corrections in the comments; good luck!

10 Responses

AndresT May 14, 2020 at 1:14 AM | Permalink

It gave me the following error:
ERROR: PKGBUILD contains CRLF characters and cannot be sourced.
Jeroen May 14, 2020 at 5:14 AM | Permalink

I think that means your git client is replacing line endings. Try setting: git config –global core.autocrlf false
1. AndresT May 15, 2020 at 7:04 PM | Permalink
  
  Thanks for trying. Still no luck:
  
  ERROR: PKGBUILD contains CRLF characters and cannot be sourced.
  cp: cannot stat ‘src/R-source/src/gnuwin32/installer/*.exe’: No such file or directory
  1. AndresT May 15, 2020 at 7:44 PM | Permalink
    
    It worked when using: git config –global core.autocrlf auto
    instead of false.
    Wouldn’t have been able to figure it out without your help.
    Thanks!
    1. Avi May 16, 2020 at 9:58 PM | Permalink
      
      Fantastic!!
Alain May 16, 2020 at 10:55 AM | Permalink

How did you manage the curl certificate error on Windows?

curl: (60) SSL certificate problem: self signed certificate in certificate chain
1. Avi May 16, 2020 at 9:57 PM | Permalink
  
  Hello, Alain. On my home computer it worked fine. For my work computer, I manually downloaded “https://curl.haxx.se/ca/cacert.pem” to the starting directory and then changed PKGBUILD (line 33 above) to just reference cacert.pem like shortcut.diff. There was one time, and I’m not remembering for what, that instead I had to change a bunch of https to http. I think it was for a package and not base, but you can try that too.
R Weekly 2020-20 drake & ML, presser, xaringanthemer – R-Craft May 17, 2020 at 8:01 PM | Permalink

[…] Building R 4+ for Windows with OpenBLAS […]
nekay June 7, 2020 at 1:58 PM | Permalink

> There are some packages which rely on compiled libraries—such as XML or nlopt—for which extra steps are needed to build from source.

Could you tell me the examples for tackling these types of packages?
1. Avi June 7, 2020 at 2:24 PM | Permalink
  
  Hi. For XML, Jeroen has a method here: <https://github.com/r-windows/docs/blob/master/packages.md#readme>. To build nloptr requires forking r-packages, building nlopt with an adjusted PKGBUILD script, changing some calls in the Makevars.win for nloptr, and then installing it. I haven’t put them all together cleanly yet. I’ll try to this week; if I don’t, email me and I’ll send you my raw notes.

Comments are closed.