Best Practices Working With Spatial Data in R

Lots of great resources:

• https://www.rspatial.org/
• https://www.r-spatial.org/ (note the dash)
• https://datacarpentry.org/geospatial-workshop/

A note about DataCamp:

• Recent mis-handling of sexual assault by DataCamp
• Many instructors advising people not use their courses, and are making them available elsewhere
• See https://noamross.github.io/datacamp-sexual-assault/ for more info

Notes available from https://github.com/achubaty/r-talks/tree/master/spatial-data-best-practices

1. CRAN taskviews:
   • https://cran.r-project.org/web/views/Spatial.html
   • https://cran.r-project.org/web/views/SpatioTemporal.html
2. Reading/loading spatial data
3. Basic plotting
4. Gotchas and best practices

install.packages(c("assertthat", "crayon", "raster", "rgdal", "sp"))

library(sp)
library(raster)
library(crayon)

References

• Bryan, J. 2017. Excuse me, do you have a moment to talk about version control? PeerJ Prepr. 5:1–23. https://peerj.com/preprints/3159v2/
• Visser M. D. et al.. 2015. Speeding up ecological and evolutionary computations in R; essentials of high performance computing for biologists. PLOS Comput. Biol. 11:e1004140.
• Wilson, G. et al. 2014. Best practices for scientific computing. PLoS Biol. 12:e1001745.

Every project should be self-contained and use relative file paths.

myProject/
  |_ data/           ## shared data could be symlinked
  |_ outputs/
  |_ presentations/
  |_ publications/
  |_ scripts/
  |_ src/

Use RStudio projects!

• https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects

it's great if others can use your code, but it's more important that you can reuse it

• comment your code with why (design & purpose), not the what (mechanics)
• use descriptive names (nouns for objects; verbs for functions)
• use consistent style/formatting
  • check formatting with lintr package

• use version control (e.g., git & GitHub)

• don't copy/paste
• don't copy paste
• don't copy/paste
• modularize code for reuse (use functions)

• use assertions & tests

df1 <- data.frame(A = 1:26, B = sample(letters))
head(df1)

assertthat::assert_that(is(df1$B, "character")) ## why?

• pair programming
• code review

• speed and memory improvements
• only do this after code is working correctly

Especially on a shared system 😄

1. only use what you need;
2. improve your code so it's less greedy;
3. communicate your needs with other users.

1. RAM usage;
2. Disk I/O;
3. Disk storage (user and temp folders);
4. CPU time;
5. Network use.

Remove intermediate objects to free RAM.

Save them to disk for retrieval later if you need them.

Have the script quit the rsession when done to free RAM.

exit <- Q <- function(save = "no", status = 0, runLast = TRUE) {
  q(save = save, status = 0, runLast = TRUE)
}

To best use RAM, CPU, disk, and network when others aren't likely using them:

• Schedule your scripts to run at later time (RStudio v1.2)
• Stagger jobs using Sys.sleep()

Identify bottlenecks + high resource use

Determine code run time

startTime <- Sys.time()
reallyLongFunction()
endTime <- Sys.time() - startTime
message(cyan("reallyLongFunction() took ", endTime, "s to run."))

Identify bottlenecks + high resource use

Determine memory usage

f <- "~/data/LandCoverOfCanada2005_V1_4/LCC2005_V1_4a.tif"
r <- raster::raster(f)
object.size(r)

17112 bytes

raster::inMemory(r)

[1] FALSE

Also, use OS tools for monitoring memory use.

Identify bottlenecks + high resource use

Rstudio profiler

• profiling time

  • https://support.rstudio.com/hc/en-us/articles/218221837-Profiling-with-RStudio#profiling-time-example

• profiling memory

  • https://support.rstudio.com/hc/en-us/articles/218221837-Profiling-with-RStudio#profiling-memory-example

Garbage collection is handled by R and your OS will try to free RAM that is no longer being used.

However, you can make scripts more efficient by frequently removing transient/intermediate objects from memory:

1. save to disk often (but not too often!);
2. load from disk when recovering your session (so you don't need to rerun everything each time);
3. cache your intermediate results (reproducible::Cache())

Many raster operations use tempfiles behind the scenes, which will clutter your temp drive (a big problem when the machine isn't rebooted frequently).

## make a note of this location, restart R session, and rerun
raster::tmpDir()

Use rasterOptions() to set tmpdir manually and be sure to cleanup at the end of your script/session.

maxMemory <- 5e+7
scratchDir <- file.path("/tmp/scratch/MPB")

rasterOptions(default = TRUE)
options(rasterMaxMemory = maxMemory, rasterTmpDir = scratchDir)

raster::tmpDir()

...

unlink(raster::tmpDir(), recursive = TRUE)

> ?dataType

Cluster operations on Windows use more RAM than they should because Windows can't fork.

Always close your clusters when done to ensure that RAM gets freed.

cl <- raster::beginCluster()
...
raster::endCluster(cl) ## in a function, put this in `on.exit()`

• RasterStack converting to RasterBrick
• writeRaster() changes the projection
• Factor rasters are a pain (use factorValues())
• Use buffer width 0 before intersection etc.
  • see reproducible::fixErrors() within reproducible::prepInputs

install.packages("sf")    ## will supercede `sp`
install.packages("stars") ## will supercede `raster`

## fast shapefile operations
devtools::install_github("s-u/fastshp")

## fast raster operations
install.packages("fasterize")
install.packages("velox")

## more wrappers for GDAL in R
install.packages("gdalUtils")