Model Cross Validation Software and Pipeline
This code base allows a user to provide a file of different configurations for running the Structural Topic Models, parallelizing its runs in Gitlab in a parallel matrix.
The interim and final outputs are pushed to Artifactory for archival storage and distribution.
Attached Files:
- stm-summaries-main.zip, the zipped code base for the cross validation gitlab pipeline
- Dockerfile, a document with instructions used to create a container image
- stm-summaries-main-File Manifest.csv, a list of the files contained in the zip file and their purpose
Running locally
Note that these scripts are *primarily* provided for posterity purposes.
To run these scripts locally, you must copy processed article files into the `1_initial_article_text_cleaning/tagged_articles/` folder, in the form of `.rds` files.
See the "Data prep and modeling support" child on ScienceBase for more information on processing articles into the appropriate `.rds` format.
Running in GitLab if you import this codebase as a project:
- browse to the Pipelines -> Click New Pipeline
- Select File for variable type and then paste in a chunk of yml similar to the one below:
`seed:` is the seed used to provide replication capabilities
`train_prop:` is the proportion of the data set that should be withheld for training versus testing the model
`K:` is the # of topics to ask the model to fit, a value of 0 is treated as a signal to the stm model to use the algorithm of Lee and Minmo to determine the proper K value
`prevalence:` is the formula for the stm to use for introducing covariates into the stm model run.
Example:
~~~
.snippet:
model_parameters:
- seed: "20250514"
train_prop: "0.75"
K: "0"
prevalence: "~ stm::s(quarter_ind) + water_quantity_ind + water_quality_ind + access_to_the_resource_ind + susceptibility_ind + coping_ind + exposure_ind + fire_ind + drought_ind + heat_ind + flood_ind + hab_ind"
- seed: "20250514"
train_prop: "0.75"
K: "100"
prevalence: "~ stm::s(quarter_ind) + water_quantity_ind + water_quality_ind + access_to_the_resource_ind + susceptibility_ind + coping_ind + exposure_ind + fire_ind + drought_ind + heat_ind + flood_ind + hab_ind"
Find Related Datasets
Search by Tags
Click any tag below to search for similar datasets
Complete Metadata
| accessLevel | public |
|---|---|
| bureauCode |
[ "010:12" ] |
| contactPoint |
{ "fn": "Megan K Hines", "@type": "vcard:Contact", "hasEmail": "mailto:mhines@usgs.gov" } |
| description | This code base allows a user to provide a file of different configurations for running the Structural Topic Models, parallelizing its runs in Gitlab in a parallel matrix. The interim and final outputs are pushed to Artifactory for archival storage and distribution. Attached Files: - stm-summaries-main.zip, the zipped code base for the cross validation gitlab pipeline - Dockerfile, a document with instructions used to create a container image - stm-summaries-main-File Manifest.csv, a list of the files contained in the zip file and their purpose Running locally Note that these scripts are *primarily* provided for posterity purposes. To run these scripts locally, you must copy processed article files into the `1_initial_article_text_cleaning/tagged_articles/` folder, in the form of `.rds` files. See the "Data prep and modeling support" child on ScienceBase for more information on processing articles into the appropriate `.rds` format. Running in GitLab if you import this codebase as a project: - browse to the Pipelines -> Click New Pipeline - Select File for variable type and then paste in a chunk of yml similar to the one below: `seed:` is the seed used to provide replication capabilities `train_prop:` is the proportion of the data set that should be withheld for training versus testing the model `K:` is the # of topics to ask the model to fit, a value of 0 is treated as a signal to the stm model to use the algorithm of Lee and Minmo to determine the proper K value `prevalence:` is the formula for the stm to use for introducing covariates into the stm model run. Example: ~~~ .snippet: model_parameters: - seed: "20250514" train_prop: "0.75" K: "0" prevalence: "~ stm::s(quarter_ind) + water_quantity_ind + water_quality_ind + access_to_the_resource_ind + susceptibility_ind + coping_ind + exposure_ind + fire_ind + drought_ind + heat_ind + flood_ind + hab_ind" - seed: "20250514" train_prop: "0.75" K: "100" prevalence: "~ stm::s(quarter_ind) + water_quantity_ind + water_quality_ind + access_to_the_resource_ind + susceptibility_ind + coping_ind + exposure_ind + fire_ind + drought_ind + heat_ind + flood_ind + hab_ind" |
| distribution |
[ { "@type": "dcat:Distribution", "title": "Digital Data", "format": "XML", "accessURL": "https://doi.org/10.5066/P144GSTD", "mediaType": "application/http", "description": "Landing page for access to the data" }, { "@type": "dcat:Distribution", "title": "Original Metadata", "format": "XML", "mediaType": "text/xml", "description": "The metadata original format", "downloadURL": "https://data.usgs.gov/datacatalog/metadata/USGS.68af3ac2d4be0253bb2c804d.xml" } ] |
| identifier | http://datainventory.doi.gov/id/dataset/USGS_68af3ac2d4be0253bb2c804d |
| keyword |
[ "USGS:68af3ac2d4be0253bb2c804d", "inlandWaters", "society" ] |
| modified | 2026-07-10T00:00:00Z |
| publisher |
{ "name": "U.S. Geological Survey", "@type": "org:Organization" } |
| spatial | -128.6719, 17.9787, -65.3906, 50.0642 |
| theme |
[ "geospatial" ] |
| title | Model Cross Validation Software and Pipeline |
