Download several datasets given their URLs and destination path

Source: R/mf_download_data.R

This function enables to download datasets. In a data import workflow, this function is typically used after a call to the mf_get_url function. The output value of mf_get_url can be used as input of parameter df_to_dl of mf_download_data.

The download can the parallelized.

mf_download_data(
  df_to_dl,
  path = tempfile("modisfast_"),
  parallel = FALSE,
  num_workers = parallel::detectCores() - 1,
  credentials = NULL,
  verbose = "inform",
  min_filesize = 5000
)

Arguments

df_to_dl

data.frame. Urls and destination files of dataset to download. Typically output of mf_get_url. See Details for the structure

path

string. Target folder for the data to download. Default : temporary folder.

parallel

boolean. Parallelize the download ? Default to FALSE

num_workers

integer. Number of workers in case of parallel download. Default to number of workers available in the machine minus one.

credentials

vector string of length 2 with username and password. optional if the function mf_login was previously executed.

verbose

string. Verbose mode ("quiet", "inform", or "debug"). Default "inform".

min_filesize

integer. Minimum file size expected (in bites) for one file downloaded. If files downloaded are less that this value, the files will be downloaded again. Default 5000.

Value

a data.frame with the same structure of the input data.frame df_to_dl + columns providing details of the data downloaded. The additional columns are :

fileDl

Booloean (dataset downloaded or failure)

dlStatus

Download status : 1 = download ok ; 2 = download error ; 3 = dataset was already existing in destination file

fileSize

File size on disk (in bites)

Details

Parameter df_to_dl must be a data.frame with the following minimal structure :

id_roi

An id for the ROI (character string)

collection

Collection (character string)

name
url

URL of the file to download (character string)

Examples

if (FALSE) { # \dontrun{

### Login to EOSDIS Earthdata with your username and password
log <- mf_login(credentials = c("earthdata_un", "earthdata_pw"))

### Set-up parameters of interest
coll <- "MOD11A1.061"

bands <- c("LST_Day_1km", "LST_Night_1km")

time_range <- as.Date(c("2017-01-01", "2017-01-30"))

roi <- sf::st_as_sf(
  data.frame(
    id = "roi_test",
    geom = "POLYGON ((-5.82 9.54, -5.42 9.55, -5.41 8.84, -5.81 8.84, -5.82 9.54))"
  ),
  wkt = "geom", crs = 4326
)

### Get the URLs of the data
(urls_mod11a1 <- mf_get_url(
  collection = coll,
  variables = bands,
  roi = roi,
  time_range = time_range
))

### Download the data
res_dl <- mf_download_data(urls_mod11a1)

### Import the data as terra::SpatRast
modis_ts <- mf_import_data(dirname(res_dl$destfile[1]), collection = coll)

### Plot the data
terra::plot(modis_ts)
} # }