SAMATE Logo NIST Logo The SAMATE Project Department of Homeland Security

Resources from the Software Assurance Reference Dataset

You will find here documents from test suites and miscellaneous resources related to SARD.


* DISCLAIMER: These are documents produced by other entities and NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic.


These documents describe the Securely Taking On New Executable Software of Uncertain Provenance (STONESOUP) C and Java test cases that were created by the The Intelligence Advanced Research Projects Activity (IARPA) specifically for use in testing static analysis tools. The documents are intended for anyone who wishes to use the test cases for their own testing purposes, or who would like to have a greater understanding of the test cases design.

STONESOUP Phase 3 test cases were released to NIST as a virtual machine, which can be downloaded here. Phase 1 is also avaiable for download here.

This product contains or makes use of Intelligence Advanced Research Projects Activity (IARPA) data from the STONESOUP program. Any product, report, publication, presentation, or other document including or referencing the IARPA data herein should include this statement.

Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA or the U.S. Government.

Please find the documents below.

  • Overview.pdf (1p): gives a big picture of the IARPA STONESOUP Program.

  • Test Case and Test Suite Material:
    • Test Case Creation Guide.pdf (39p): describes how the test cases are organized, including the naming convention, file structure, and metadata documentation.

    • Weaknesses Documentation.pdf (673p): a number of software snippets was developed to provide discrete tests of specific weaknesses, performing no further meaningful processing. These weakness variants form the basis from which the STONESOUP is generated.

    • TEXAS User Guide.pdf (23p) and Communication API Guide.pdf (80p): The Test and Evaluation eXecution and Analysis System (TEXAS) is designed and developed to test a Performer technology’s ability to detect and mitigate software vulnerabilities and exploit through static analysis and run time countermeasures.

    • System Design Document.pdf (39p): The scope of this document is to cover the system design of the "Test and Evaluation, eXecution, Analysis System" (TEXAS) for the STONESOUP Phase 3 Test and Evaluation activity.

    • Test and Evaluation Phase 3 Final Report.pdf (493p): This document presents the final main report of the STONESOUP project in detail. Test and Evaluation were performed by Columbia University, GrammaTech and Kestrel Institute.

    • Test Generation Report (69p): This report discusses how Test and Evaluation team implemented the tasks described in the STONESOUP Phase 3 Test Data Generation Plan (TGP). The TGP describes the composition of test cases, and includes the test case naming standard, which allows a user to identify the behavior of each test case based on the name of the test case.

  • Participants Reports:
    • Kestrel Institute Report.pdf (162p, Ref: AFRL-RY-WP-TR-2015-0019): The research team from Kestrel Institute, Kestrel Technology, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and Dynamic Object Language Labs Inc (DOLL) produced the Vulnerabilities In Bytecode Removed by Anaysis, Nuanced Confinement and Diversification (VIBRANCE). The VIBRANCE tool starts with a vulnerable Java application and automatically hardens it against SQL injection, OS command injection, file path traversal, numeric errors, denial of service, and other attacks. For a large class of attacks, the protection added by VIBRANCE blocks the attacks and safely continues execution.

    • Grammatech Report.pdf (257p, Ref: AFRL-RY-WP-TR-2015-0017): Describes the results of the research and development of the Preventing Exploits Against Software of Uncertain Provenance (PEASOUP), a technology that enables the safe execution of software executables.

    • MINESTRONE Report.pdf (58p, Ref: AFRL-RY-WP-TR-2015-0002): MINESTRONE is a novel architecture that integrates static analysis, dynamic confinement, and code diversification techniques to enable the identification, mitigation and containment of a large class of software vulnerabilities.

^ Juliet Documents *

These documents describe the Juliet Test Suite C/C++ and Java test cases that were created by the NSA’s Center for Assured Software (CAS) specifically for use in testing static analysis tools. These are intended for anyone who wishes to use the test cases for their own testing purposes, or who would like to have a greater understanding of how the test cases were created. Please find the links below:

Juliet Test Suite can be downloaded as standalone files here for v1.3 test cases, and here for v1.2 and v1.1 test cases. Also, Juliet test cases are individually avaiable in the SARD, following the documents below:

^ SARD Downloads

Here is the It contains every test case with files, flaws and associations.

You can also download sard_schema.xsd that describes the manifest's structure. contains all SARD test cases, including the full manifest.