Demo Codebase for agentic-research-measurement-probes
An agentic AI measurement tool for deep research over local corpora of PDF and Markdown documents. Given a research question, it orchestrates a programmatic AI pipeline to exhaustively evaluate, synthesize, and verify information from your documents, producing a Markdown report with inline footnote citations -- then automatically measures the quality of every citation using LM-judge measurement probes.
The probes are a first-class feature, not an afterthought. After each report section is written, three mutually exclusive probe evaluators run automatically, scoring every citation along distinct quality dimensions: faithfulness (does the source support the claim?), completeness (is the source's full message represented without cherry-picking?), and sufficiency (does the source carry the evidentiary burden the claim requires, or does the author overreach?). Probe results are stored alongside the report as a structured audit trail, enabling quantitative measurement of AI-generated research quality.
Resources
1 resource available
-
agentic-research-measurement-probes Github Repository
FILE
Find Related Datasets
Search by Tags
Click any tag below to search for similar datasets
Complete Metadata
| @type | dcat:Dataset |
|---|---|
| accessLevel | public |
| accrualPeriodicity | irregular |
| bureauCode |
[ "006:55" ] |
| contactPoint |
{ "fn": "Michael Majurski", "hasEmail": "mailto:michael.majurski@nist.gov" } |
| description | An agentic AI measurement tool for deep research over local corpora of PDF and Markdown documents. Given a research question, it orchestrates a programmatic AI pipeline to exhaustively evaluate, synthesize, and verify information from your documents, producing a Markdown report with inline footnote citations -- then automatically measures the quality of every citation using LM-judge measurement probes. The probes are a first-class feature, not an afterthought. After each report section is written, three mutually exclusive probe evaluators run automatically, scoring every citation along distinct quality dimensions: faithfulness (does the source support the claim?), completeness (is the source's full message represented without cherry-picking?), and sufficiency (does the source carry the evidentiary burden the claim requires, or does the author overreach?). Probe results are stored alongside the report as a structured audit trail, enabling quantitative measurement of AI-generated research quality. |
| distribution |
[ { "title": "agentic-research-measurement-probes Github Repository", "accessURL": "https://github.com/usnistgov/agentic-research-measurement-probes", "description": "Github repository of the agentic-research-measurement-probes demo" } ] |
| identifier | ark:/88434/mds2-4158 |
| issued | 2026-03-27 |
| keyword |
[ "Agentic-AI Measurement", "Deep Research", "LLM" ] |
| landingPage | https://data.nist.gov/od/id/mds2-4158 |
| language |
[ "en" ] |
| license | https://www.nist.gov/open/license |
| modified | 2026-03-26 00:00:00 |
| programCode |
[ "006:045" ] |
| publisher |
{ "name": "National Institute of Standards and Technology", "@type": "org:Organization" } |
| theme |
[ "Information Technology:Software research" ] |
| title | Demo Codebase for agentic-research-measurement-probes |
