S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform
The Reclamation Research and Development Office funded an evaluation of file formats for large datasets to use in RISE through the Science & Technology Program. A team of Reclamation scientific and information technology (IT) subject matter experts evaluated multiple file formats commonly utilized for scientific data through literature review and independent benchmarks. The network Common Data Form (netCDF) and Zarr formats were identified as open-source options that could meet a variety of Reclamation use cases. The formats allow for metadata, data compression, subsetting, and appending in a single file using an efficient binary format. Additionally, the Zarr format is optimized for cloud storage applications. While support of both formats would provide the most flexibility, the maturity of the netCDF format led to its prioritization as the preferred RISE file format for large datasets.
This report documents the evaluation and selection of large data file formats for the RISE platform. Additionally, a preliminary list of identified changes to the RISE platform needed to support the netCDF format is provided. The intent is to frame future RISE development by providing a roadmap to support large datasets within the platform.
Resources
2 resources available
-
RISE Item Details Page URL for "S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform "
TEXT/HTML -
PDF File for "S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform "
APPLICATION/PDF
Find Related Datasets
Search by Tags
Click any tag below to search for similar datasets
Complete Metadata
| @type | dcat:Dataset |
|---|---|
| accessLevel | public |
| accrualPeriodicity | irregular |
| bureauCode |
[ "010:10" ] |
| contactPoint |
{ "fn": "RISE Team", "@type": "vcard:Contact", "hasEmail": "mailto:data@usbr.gov" } |
| description | The Reclamation Research and Development Office funded an evaluation of file formats for large datasets to use in RISE through the Science & Technology Program. A team of Reclamation scientific and information technology (IT) subject matter experts evaluated multiple file formats commonly utilized for scientific data through literature review and independent benchmarks. The network Common Data Form (netCDF) and Zarr formats were identified as open-source options that could meet a variety of Reclamation use cases. The formats allow for metadata, data compression, subsetting, and appending in a single file using an efficient binary format. Additionally, the Zarr format is optimized for cloud storage applications. While support of both formats would provide the most flexibility, the maturity of the netCDF format led to its prioritization as the preferred RISE file format for large datasets. This report documents the evaluation and selection of large data file formats for the RISE platform. Additionally, a preliminary list of identified changes to the RISE platform needed to support the netCDF format is provided. The intent is to frame future RISE development by providing a roadmap to support large datasets within the platform. |
| distribution |
[ { "@type": "dcat:Distribution", "title": "RISE Item Details Page URL for "S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform "", "accessURL": "https://data.usbr.gov/catalog/8002/item/128581", "mediaType": "text/html", "description": "Landing page for "S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform "" }, { "@type": "dcat:Distribution", "title": "PDF File for "S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform "", "mediaType": "application/pdf", "description": ""S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform " as a PDF file", "downloadURL": "https://data.usbr.gov/rise/content-rise-public/rise/catalog-item/binary/2023-12-31%20rise%20report%20final.pdf" } ] |
| identifier | https://datainventory.usbr.gov/rise/item/128581 |
| keyword |
[ "Modeling", "Open Data", "Storage" ] |
| landingPage | https://data.usbr.gov/catalog/8002/item/128581 |
| modified | 2024-01-05T19:19:09Z |
| publisher |
{ "name": "Bureau of Reclamation", "@type": "org:Organization" } |
| spatial |
"{"type":"Polygon","coordinates":[[[-180,-90],[-180,90],[180,90],[180,-90],[-180,-90]]]}"
|
| title | S&T Project 22041 Final Report: Evaluation of file formats for storage and transfer of large datasets in the RISE platform |
