SAMATE Logo NIST Logo The SAMATE Project Department of Homeland Security

Test Suites

Test cases in SARD can be combined and form multiple test suites and all are present in this page. Please use the links below to quick access to each section:

^ Stand-alone Suites

Download Publication Date Title Version Description Contributor # of Cases
See Description Column May. 2015 IARPA STONESOUP Phase 3 - Virtual Machine 3.0 A collection of C and Java test cases based on 16 widely-used open-source software in which vulnerabilities have been seeded. It comes bundled in a virtual machine for ease of use.

This product contains or makes use of Intelligence Advanced Research Projects Activity (IARPA) data from the STONESOUP program. Any product, report, publication, presentation, or other document including or referencing the IARPA data herein should include this statement.

All documents related to the STONESOUP program can be found at the documents page.

NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic.

Download the VM in 2GB slices below:
Part 00 Download testsuite Part 01 Download testsuite Part 02 Download testsuite Part 03 Download testsuite Part 04 Download testsuite Part 05 Download testsuite Part 06 Download testsuite Part 07 Download testsuite Part 08 Download testsuite Part 09 Download testsuite Part 10 Download testsuite Part 11 Download testsuite
IARPA C: 4582

Java: 3188
Download testsuite May. 2013 Juliet Test Suite for C/C++ 1.2 A collection of test cases in the C/C++ language. It contains examples for 118 different CWEs.

All documents related to the Juliet Test Suite can be found at the documents page.

This software is not subject to copyright protection and is in the public domain. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic.
NSA Center for Assured Software 61,387
Download testsuite May. 2013 Juliet Test Suite for Java 1.2 A collection of test cases in the Java language. It contains examples for 112 different CWEs.

All documents related to the Juliet Test Suite can be found at the documents page.

This software is not subject to copyright protection and is in the public domain. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic.
NSA Center for Assured Software 25,477

^ SARD Suites

Results: 34 Test Suites.


Test Suite ID View
Download
Manifest
Creation Date Title Description Contributor # of Cases
101 View testsuite    Download testsuite    Download manifest 2015-03-16 C Test Suite for Source Code Analyzer v2 - Secure This test suite replaces test suite 46 based on a collaboration with Alexander Hoole from University of Victoria, BC, Canada. The new test cases provided by these test suites contain the following improvements: removal of targeted weaknesses from 13 “GOOD” test cases in test suite 46, removal of extraneous weaknesses, replacement of test cases to align with the CWE’s specified in NIST SP 500-268 v1.1, creation of additional test cases to provide consistent BAD/GOOD pairings, application of minor improvements to code, renaming of files and the addition of FLAW/FIX comments to assist automation, and insertion of improved metadata to assist researchers using SARD. Please refer to the test case metadata fields to view additional information for each test case. Note: Some test cases have been deprecated and replaced with fixed versions since this test suite has been initially published. Aurelien Delaitre 102
100 View testsuite    Download testsuite    Download manifest 2015-03-16 C Test Suite for Source Code Analyzer v2 - Vulnerable This test suite replaces test suite 45 based on a collaboration with Alexander Hoole from University of Victoria, BC, Canada. The new test cases provided by these test suites contain the following improvements: removal of extraneous weaknesses, replacement of test cases to align with the CWE’s specified in NIST SP 500-268 v1.1, creation of additional test cases to provide consistent BAD/GOOD pairings, application of minor improvements to code, renaming of files and the addition of FLAW/FIX comments to assist automation, and insertion of improved metadata to assist researchers using SARD. Please refer to the test case metadata fields to view additional information for each test case. Note: Some test cases have been deprecated and replaced with fixed versions since this test suite has been initially published. Aurelien Delaitre 102
99 View testsuite    Download testsuite    Download manifest 2014-08-01 wordpress-2.0 Content management system based on PHP and MySQL. Contains CVEs. SAMATE Team Staff 17
98 View testsuite    Download testsuite    Download manifest 2014-08-01 openfire-3.6.0 Real time collaboration server that uses XMPP (Jabber). Contains CVEs. SAMATE Team Staff 12
97 View testsuite    Download testsuite    Download manifest 2014-08-01 jspwiki-2.5.124 WikiWiki engine built around JEE components (Java, servlets, JSP). Contains CVEs. SAMATE Team Staff 3
96 View testsuite    Download testsuite    Download manifest 2014-08-01 jetty-6.1.16 Web server and javax.servlet container with support for SPDY, WebSocket, OSGi, JMX, JNDI, JAAS, along with other integrations. Contains CVEs. SAMATE Team Staff 6
95 View testsuite    Download testsuite    Download manifest 2014-08-01 apache-tomcat-5.5.13 Open source software implementation of the Java Servlet and JavaServer Pages technologies. Contains CVEs. SAMATE Team Staff 37
94 View testsuite    Download testsuite    Download manifest 2014-08-01 wireshark-1.8.0 Network traffic analyzer containing CVEs. SAMATE Team Staff 85
93 View testsuite    Download testsuite    Download manifest 2014-08-01 wireshark-1.2.0 Network traffic analyzer containing CVEs. SAMATE Team Staff 44
92 View testsuite    Download testsuite    Download manifest 2014-08-01 dovecot-1.2.0 IMAP and POP3 email server for Linux/UNIX-like systems. Contains CVEs. SAMATE Team Staff 9
91 View testsuite    Download testsuite    Download manifest 2014-08-01 chrome-5.0.375.54 Google web browser containing CVEs. SAMATE Team Staff 10
90 View testsuite    Download testsuite    Download manifest 2014-08-01 asterisk-10.2.0 VoIP communication system with chat, conferencing, instant messaging, fax and other features. Contains CVEs. SAMATE Team Staff 15
89 View testsuite    Download testsuite    Download manifest 2014-06-09 A Taxonomy of Buffer Overflows Kendra Kratkiewicz developed a taxonomy of C buffer overflows and 291 test cases representing this taxonomy. Each test case has three flawed versions (with overflows just outside, moderately outside, and far outside the buffer) and a patched version (without buffer overflow). Examples of using these are in "A Taxonomy of Buffer Overflows for Evaluating Static and Dynamic Software Testing Tools" 2005. Eric Rosenberg 1164
88 View testsuite    Download testsuite    Download manifest 2014-06-09 Testing Exploitable Buffer Overflows From Open Source Code Zitser, Lippmann, and Leek extracted 14 model programs from internet applications (BIND, Sendmail, WU-FTP) with known buffer overflows. These models have the portion of code with the overflows. Patched versions are also included. Examples of using these are in "Using Exploitable Buffer Overflows From Open Source Code" 2004. Eric Rosenberg 28
87 View testsuite    Download testsuite    Download manifest 2013-05-15 Juliet Test Suite for Java (v1.2) This is a collection of test cases in the Java language. It contains examples for 112 different CWEs. NOTE: This package contains only individual test cases. We recommend to download the full test suite at the top of the "Test Suite" page. All documents related to the Juliet Test Suite can be found at the documents page SAMATE Team Staff 25477
86 View testsuite    Download testsuite    Download manifest 2013-05-15 Juliet Test Suite for C/C++ (v1.2) This is a collection of test cases in the C/C++ language. It contains examples for 118 different CWEs. NOTE: This package contains only individual test cases. We recommend to download the full test suite at the top of the "Test Suite" page. All documents related to the Juliet Test Suite can be found at the documents page SAMATE Team Staff 61387
81 View testsuite    Download testsuite    Download manifest 2013-02-08 Basic CWE Effectiveness, CWE-121: Stack-based Buffer Overflow, for C. These allow a prospective user to understand that a capability is effective in locating CWE-121: Stack-based Buffer Overflow in the most basic situations in C code. Michael Koo 5
69 View testsuite    Download testsuite    Download manifest 2011-04-08 Juliet Test Suite for Java (v1.0 - Deprecated) This is a collection of test cases in the Java language. It contains examples for 106 different CWEs. NOTE: This package contains only individual test cases. We recommend to download the full test suite at the top of the "Test Suite" page. All documents related to the Juliet Test Suite can be found at the documents page SAMATE Team Staff 14184
68 View testsuite    Download testsuite    Download manifest 2011-04-08 Juliet Test Suite for C/C++ (v1.0 - Deprecated) This is a collection of test cases in the C/C++ language. It contains examples for 116 different CWEs. NOTE: This package contains only individual test cases. We recommend to download the full test suite at the top of the "Test Suite" page. All documents related to the Juliet Test Suite can be found at the documents page SAMATE Team Staff 45309
65 View testsuite    Download testsuite    Download manifest 2010-02-04 Java Test Suite for Source Code Analyzer - weakness suppresion This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RO-2 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 10
64 View testsuite    Download testsuite    Download manifest 2010-02-04 Java Test Suite for Source Code Analyzer - false positive This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RM-6 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 27
63 View testsuite    Download testsuite    Download manifest 2010-02-04 Java Test Suite for Source Code Analyzer - weakness This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RM-1 through SCA-RM-5 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 27
62 View testsuite    Download testsuite    Download manifest 2008-10-02 Defence R&D Canada 25 C++ test cases (plus a main including all of them) created in 2006 by Frederic Michaud and Frederic Painchaud, Defence Research & Development Canada, http://www.drdc-rddc.gc.ca/ SAMATE Team Staff 26
59 View testsuite    Download testsuite    Download manifest 2007-12-06 C++ Test Suite for Source Code Analyzer - weakness suppresion This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RO-2 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 14
58 View testsuite    Download testsuite    Download manifest 2007-12-06 C++ Test Suite for Source Code Analyzer - false positive This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RM-6 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 39
57 View testsuite    Download testsuite    Download manifest 2007-12-06 C++ Test Suite for Source Code Analyzer - weakness This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RM-1 through SCA-RM-5 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 41
47 View testsuite    Download testsuite    Download manifest 2007-02-05 C Test Suite for Source Code Analyzer - weakness suppresion This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RO-2 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 21
46 View testsuite    Download testsuite    Download manifest 2007-02-05 C Test Suite for Source Code Analyzer - false positive This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RM-6 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 73
45 View testsuite    Download testsuite    Download manifest 2007-01-24 C Test Suite for Source Code Analyzer - weakness This test suite tests against Source Code Security Analyzer based on functional requirements SCA-RM-1 through SCA-RM-5 specified in "Source Code Security Analysis Tool Functional Specification" Michael Koo 77
31 View testsuite    Download testsuite    Download manifest 2006-10-24 Web Applications in PHP The PHP Test cases Romain Gaucher 15
27 View testsuite    Download testsuite    Download manifest 2006-10-18 MS Eric D. 25
17 View testsuite    Download testsuite    Download manifest 2006-08-09 CANDIDATE Source Code Analysis Tool Functional Specification Test Suite This test suite contains all test cases that can be used to test a general purpose, production source code analysis tool implementation against the SAMATE Source Code Analysis Tool Functional Specification. SAMATE Team Staff 34
9 View testsuite    Download testsuite    Download manifest 2006-07-11 Test suite (2006/07/11 18:32:50) Redge Bartholomew 5
6 View testsuite    Download testsuite    Download manifest 2006-06-23 ABM 1.0.1 Fortify Software\'s Analyzer BenchMark v. 1.0.1 Jeff Meister 112

^ Applications

Web apps

Results: 1 apps.

Application ID View
Download
Manifest
Name Date added Version Language Origin SLOC # of files Size Added by # of Test cases
1 View application    Download application    Download manifest WordPress 2015-04-01 2.0 PHP https://wordpress.org/wordpress-2.0.zip 24192 178 590kB Charles Oliveira 1

Mobile apps

Results: 1 apps.

Application ID View
Download
Manifest
Name Date added Version Language Origin SLOC # of files Size Added by # of Test cases
2 View application    Download application    Download manifest Card Board Sample 2015-04-16 1.0 Java https://github.com/googlesamples/cardboard-java 1947 28 305kB Charles Oliveira 1

Standalone apps

No Standalone apps at the moment.

^ Archives

Download Publication Date Title Version Description Contributor # of Cases
Download testsuite Nov. 2012 IARPA STONESOUP Phase 1 - Memory Corruption for C 1.0 A collection of test cases in the C language. It contains examples of memory corruption issues, including input triggering the vulnerability. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the STONESOUP program can be found at the documents page. IARPA 212
Download testsuite Nov. 2012 IARPA STONESOUP Phase 1 - Null Pointer Dereference for C 1.0 A collection of test cases in the C language. It contains examples of null pointer mishandling, including input triggering the vulnerability. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the STONESOUP program can be found at the documents page. IARPA 115
Download testsuite Nov. 2012 IARPA STONESOUP Phase 1 - Injection for Java 1.0 A collection of test cases in the Java language. It contains examples of various injection issues, including input triggering the vulnerability. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the STONESOUP program can be found at the documents page. IARPA 36
Download testsuite Nov. 2012 IARPA STONESOUP Phase 1 - Numeric Handling for Java 1.0 A collection of test cases in the Java language. It contains examples of numeric mishandling, including input triggering the vulnerability. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the STONESOUP program can be found at the documents page. IARPA 59
Download testsuite Nov. 2012 IARPA STONESOUP Phase 1 - Tainted Data for Java 1.0 A collection of test cases in the Java language. It contains examples of tainted data mishandling, including input triggering the vulnerability. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the STONESOUP program can be found at the documents page. IARPA 35
Download testsuite Sep. 2012 Juliet Test Suite for Java 1.1.1 A collection of test cases in the Java language. It contains examples for 113 different CWEs. This software is not subject to copyright protection and is in the public domain. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic.

v1.1.1 supersedes v1.1. It added methods needed for building test cases after adding/removing test cases. Does not affect using test cases as is. All documents related to the Juliet Test Suite can be found at the documents page.

NSA Center for Assured Software 23,957
Download testsuite Jul. 2012 Juliet Test Suite for C/C++ 1.1 A collection of test cases in the C/C++ language. It contains examples for 119 different CWEs. This software is not subject to copyright protection and is in the public domain. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the Juliet Test Suite can be found at the documents page. NSA Center for Assured Software 57,099
Download testsuite Dec. 2010 Juliet Test Suite for C/C++ 1.0 A collection of test cases in the C/C++ language. It contains examples for 116 different CWEs. This software is not subject to copyright protection and is in the public domain. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the Juliet Test Suite can be found at the documents page. NSA Center for Assured Software 45,324
Download testsuite Dec. 2010 Juliet Test Suite for Java 1.0 A collection of test cases in the Java language. It contains examples for 106 different CWEs. This software is not subject to copyright protection and is in the public domain. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guaranties, expressed or implied, about its quality, reliability, or any other characteristic. All documents related to the Juliet Test Suite can be found at the documents page. NSA Center for Assured Software 13,801