Dioptra Test Platform

Source code, documentation, and examples of use of the source code for the Dioptra Test Platform. Dioptra is a software test platform for assessing the trustworthy characteristics of artificial intelligence (AI). Trustworthy AI is: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair - with harmful bias managed1. Dioptra supports the Measure function of the NIST AI Risk Management Framework by providing functionality to assess, analyze, and track identified AI risks. Dioptra provides a REST API, which can be controlled via an intuitive web interface, a Python client, or any REST client library of the user's choice for designing, managing, executing, and tracking experiments. Details are available in the project documentation available at https://pages.nist.gov/dioptra/. Use Cases We envision the following primary use cases for Dioptra: - Model Testing: -- 1st party - Assess AI models throughout the development lifecycle -- 2nd party - Assess AI models during acquisition or in an evaluation lab environment -- 3rd party - Assess AI models during auditing or compliance activities - Research: Aid trustworthy AI researchers in tracking experiments - Evaluations and Challenges: Provide a common platform and resources for participants - Red-Teaming: Expose models and resources to a red team in a controlled environment Key Properties Dioptra strives for the following key properties: - Reproducible: Dioptra automatically creates snapshots of resources so experiments can be reproduced and validated - Traceable: The full history of experiments and their inputs are tracked - Extensible: Support for expanding functionality and importing existing Python packages via a plugin system - Interoperable: A type system promotes interoperability between plugins - Modular: New experiments can be composed from modular components in a simple yaml file - Secure: Dioptra provides user authentication with access controls coming soon - Interactive: Users can interact with Dioptra via an intuitive web interface - Shareable and Reusable: Dioptra can be deployed in a multi-tenant environment so users can share and reuse components

[
  "006:55"
]

{
  "fn": "Harold Booth III",
  "hasEmail": "mailto:harold.booth@nist.gov"
}

[
  {
    "title": "Dioptra Github Repository",
    "format": "Github Repository",
    "accessURL": "https://github.com/usnistgov/dioptra",
    "description": "The USNIST Github location where source code is located."
  }
]

[
  "AI; Trustworthy AI; Test; Evaluation; Adversarial Machine Learning; Machine Learning; TEVV"
]

[
  "en"
]

[
  "006:045"
]

{
  "name": "National Institute of Standards and Technology",
  "@type": "org:Organization"
}

[
  "Information Technology:Cybersecurity"
]