The SAMATE Project Department of Homeland Security

Other Assurance Tool Test Collections


DISCLAIMER: Certain trade names and company products are mentioned in the text or identified. In no case does such identification imply recommendation or endorsement by the National Institute of Standards and Technology (NIST), nor does it imply that the products are necessarily the best available for the purpose.

By selecting almost any of these links, you will be leaving NIST webspace. We provide these links because they may have information of interest to you. No inferences should be drawn because some sites are referenced, or not, from this page. There may be other web sites that are more appropriate for your purpose. NIST does not necessarily endorse the views expressed, or concur with the assertions presented on these sites. Further, NIST does not endorse any commercial products that may be mentioned on these sites.

In addition to the SARD, we provide our users with a list of other assusrance tool collections and benchmarks that we are aware of. Test collections in this list must be designed to assess the capabilities of assurance tools. Tool test collections can include requirements analysis tool tests, design model anaysis tool tests, source code analysis tool tests, static and dynamic binary analysis tool tests. Test collections on this list may include a harness or a test framework.

  • Software-artifact Infrastructure Repository (SIR): This repository provides Java and C programs for use in experimentation with testing and analysis techniques, and materials facilitating that use. The primary purpose of the tests and testing framework is generation of experiments in software fault testing. That said, extension of the test suites and test objects beyond its current capability is encouraged by the developers.
    • Test Suite Type: Source Code for Specific Versions of Open Source Applications
    • Number of Programs: 85
    • Number of Bugs: 572+
    • Average Number of Lines of Code: 38,825
    • Language: C, Java, C++, C#
    • Supports Multiple Versions of Test Cases: Yes
    • Test Harness: Yes

  • FaultBench: is a set of real, subject Java programs for comparison and evaluation of actionable alert identification techniques (AAITs) that supplement automated static analysis.
    • Test Suite Type: Source Code (via CVS) for Multiple Versions of Open Source Applications
    • Number of Programs: 6
    • Number of Bugs: 780
    • Average Number of Lines of Code: 4973
    • Language: Java
    • Supports Multiple: Yes
    • Test Harness: No

  • OWASP Benchmark Project is a suite of synthetic test cases designed to evaluate the speed, coverage, and accuracy of vulnerability detection tools. It includes both test cases with weaknesses and without weaknesses (to test for false positives). The test suite is updated periodically.
    • Test Suite Type: Source Code
    • Number of Programs: Over 2500
    • Number of Bugs: Over 1300
    • Average Number of Lines of Code: 53 (As of version 1.2)
    • Language: Java
    • Supports Multiple: Yes
    • Test Harness: Yes


    If you would like to suggest other tool testing resources for inclusion on this list, please contact SAMATE.