The SAMATE Project Department of Homeland Security

SARD Manual


Software Assurance Reference Dataset (SARD) Manual

Preamble

The purpose of the reference dataset is to provide researchers (particularly in the SAMATE Project), software assurance tool developers, and end users with a set of artifacts with known software security errors and fixes for them. The artifacts will be designs, source code, binaries, etc., that is, from all phases of the software lifecycle. The samples include "synthetic" (written to test), collected from "the wild" (production), and academic (from university researchers). This dataset will also contain real, production software applications with known bugs and vulnerabilities. This will allow developers to test their methods and end users to evaluate a tool when considering it. The dataset intends to encompass a wide variety of vulnerabilities, languages, platforms, and compilers. There is more information about the ideas behind the reference dataset in the Software Assurance Reference Dataset philosophy page. To access the set itself, visit https://samate.nist.gov/SARD/.

The dataset is a large effort with more than 170 000 test cases. It has benefited from many contributors. The groups of contributions are detailed on the Acknowledgments and Test Case Descriptions page.

Eligibility of a Test Case

Any software artifact with security vulnerabilities is welcome to be submitted. Samples of avoiding or mitigating such vulnerabilities are also welcome. Although we intend to have security errors from the whole software lifecycle, this dataset concentrates on source code for now.

Submit Test Cases

A test case consists of one or more files, which manifest the security error, and metadata about the file(s), such as the weakness type, language, etc. Contact us to submit test cases.

View, Search, and Download Test Cases

Any user can view or search then download test cases. The view/download screen and its subsequent screens present all the test cases in the SARD, except by default Deprecated cases. You can download selected test cases on a page, download the entire SARD, or download those on a page.

Clicking a Test Case ID displays that test case.

You can search for test cases according to certain search criteria, such as test case id, test case description, language, weakness type, string in the test file, etc. As above, you can download selected test cases from the set of test cases found, download all test cases found, or all the test cases on a page.

Test Suite

A test suite is a pre-defined collection of test cases. Anyone can view and download entire test suites.

  • Display/Download
Test suite on the navigation bar will lead to a screen that can display and down test suite.

Review Process

When a test case is first added, its status is "Candidate". After review, the status of a test case could be "Accepted". If a test case needs to be withdrawn, it is marked "Deprecated". It is still available, for historical purposes, but should not be used in any new work. See Test Case Status - What it Means for details of the review process.

Enhancements

The SARD is continuing to evolve. Suggested near-term enhancements include:

Major Subsystems

These are major subsystems to be added, which will require many changes.

  • Enhance SARD architecture to support "complex" test cases, for example,
    1. if test cases use a large infrastructure (like web app scanner), they could share it
    2. a single, large case may have many pieces which are updated from time to time. the best we could do now is (a) one huge test case, which is deprecated when a piece changes or (b) a test suite composed of test cases - the user would have to know to collect them up and combine them to use them.
  • Support Easy Running of Downloaded Test Cases
    • Indicate which file(s) is the argument to the test script
      • Add "target file(s)" field to test case metadata
    • Indicate the expected result (like good or bad, and where)!
      • Populate the Expected Output field in the metadata
      • As above, have download collect this info in some easily (mechanically) accessible way.
    • The sample tool script should have an option to do all the downloaded target files at once, that is, in one tool invocation.
  • Present all downloaded test cases as one big program. That is, there is one main().
    • "Compile" instructions would have to indicate each subdirectory, or, everything would have to put in one directory.
    • This could be an option. With this, a source code analyzer would only have to be started once, which may save considerable time.
    • Each test case would have to be a procedure, which could be called by a (synthesized) main().
    • This only applies to source code artifacts, not binaries, designs, etc.

Display Features

  • Link from contributor
In a test case page, e.g., https://samate.nist.gov/SARD/?7, link from Contributor to the SARD Acknowledgments.

Search Features

  • Add ability to search test cases by:
    • test case id (ranges)
    • file size
    • Full-text search of description - natural language search, Boolean operators.
    • other metadata
    • other code metrics? LOC (Line of Code)? Cyclomatic complexity? Keywords?
  • Optional case-sensitive search
  • Allow Boolean connections between search criteria

Other Enhancements

  • Users should be able to download information about all the test suites. (7 May 2007)
    • They can download all test cases, with a manifest, but not the test suites. That manifest, or another manifest, could include all test suites and their information.
  • Statistics of visit for each web page (available through Special pages, Popular pages) and each test case
    • It is partially done. There is statistics about visitors, pages but not really on test cases.
  • When we can search by contributor, enhance the SARD Acknowledgments page with links to search and show the test cases of the contributor.
  • Add button to select (or download) all test cases in search or display set. With that user can search, then easily download them all
  • It's possible to upload the same file twice for a given test case.
  • Refreshing the screen after having submitted a test case may submit the test cases again with a different test case ID.
    • Due to POST variables... don't know yet what to do (Romain)
  • The registration process should use a randomly generated image (e.g., letter and numbers to type by user or Captcha) to verify human interaction. The risk is that someone uses a script to register hundreds of users.

Internal Enhancements

These are internal to the SARD. They are not visible to users.

  • Use a source code control system. Let SAMATE members make small changes by themselves.
  • Document the tests (for regression testing)

History

The collection was created in 2005 and was originally called the Standard Reference Dataset, abbreviated SRD [Black, "Software Assurance And Tool Evaluation", SERP 2005]. The name was quickly changed to Software Reference Dataset to be more specific, then to SAMATE Reference Dataset to be less presumptuous.

In 2013, we were notified that the abbreviation SRD conflicted with the federally legislated Standard Reference Data (SRD). About the same time we came to desire an acronym that was less common for web searches. Several rounds of soliciting suggestions, brain storming, polling, and discussion produced over two dozen possible new names. The new name, Software Assurance Refence Dataset, was suggested by Bertrand Stivalet and was announced at the Static Analysis Tool Exposition (SATE) V Experience Workshop on Friday, 14 March 2014.