The SAMATE Project Department of Homeland Security

Static Analysis Tool Exposition VI (SATE VI)

Last update: 10/17/2017


Introduction

The Static Analysis Tool Exposition (SATE) is designed to advance research (based on large test sets) in, and improvement of, static analysis tools that find security-relevant defects in source code. Briefly, participating tool makers run their static analyzer on a set of programs, then researchers led by NIST analyze the tool reports. Everyone shares results and experiences at a workshop. The analysis report is made publicly available later.

SATE's purpose is NOT to evaluate nor choose the "best" tools. Rather, it is aimed at exploring the following characteristics of tools: relevance of warnings to security, their correctness, and prioritization. Its goals are:

  • To enable empirical research based on large test sets,
  • To encourage improvement of tools,
  • To boost public awareness of tools by objectively demonstrating their use on real software.

SATE VI is the sixth occurrence of SATE. There is information about and results from SATE V , SATE IV , SATE 2010 , SATE 2009  and SATE 2008  on-line.


Changes Since SATE V

  • The C and Java Tracks have been merged into the new Classic Track.
  • We have a new Mobile Track, focusing on mobile applications.
  • SATE VI will be using rolling releases for the test cases, so all tracks may not run at the same time.
  • On the Classic Track, we injected realistic vulnerabilities into large test cases to assess the ability of tools to find bugs that matter. The type of these vulnerabilities will be shared along with the test cases.

Tracks

Preparation work is ongoing for SATE VI. We will add information about the tracks and test cases as it becomes available. Note that we will be using rolling releases for the test cases, so the different tracks may not all run at the same time.


Classic Track

Contact: aure 'at' nist.gov

Release target: Winter 2017/2018

The Classic Track combines the C and Java Tracks from the past SATEs. In SATE VI, we injected realistic vulnerabilities into large test cases to assess the ability of tools to find bugs that matter. The SAMATE team will only analyze warnings related to these injected bugs.

For participants interested in a dry run, we provide below the name and version of the base programs we used for our test cases. These programs are not the test cases and do not contain the injected bugs. The test cases containing the bugs will be released in the next few months.

Other test cases are likely to be added, and we will list them on this page when ready.


Ockham Sound Analysis Criteria Track

Contact: paul.black 'at' nist.gov

Release target: October 2017

Main page: https://samate.nist.gov/SATE6OckhamCriteria.html

The Ockham Criteria Track focuses on sound static analysis tools. We will analyze all findings reported by participating tools to assess their soundness.

After consultation with potential participants, we decided to use a new version of Juliet, 1.3, as test cases. Although the 1.3 cases are done, the suite needs a little work to release.


Mobile Track

Contact: michael.ogata 'at' nist.gov

Release target: October 2017

The mobile track includes several mobile application test cases. Most of the test cases are ready for analysis by tools. For this first foray into the mobile space, we are focusing on the Android operating system. For each test case, both source code and deployable APK files are provided. While SATE focuses primarily on static analysis, we invite participants in the mobile track to submit analysis of any and all kinds, including dynamic and behavioral analysis.


Organizing Meeting

The organizing meeting was held on May 31, 2017. The recording of the event is available below.

Notes:

The SATE V report is in the final stage of the publication process and should be released within two months.

Additional information will be available on this website after the organizing meeting. Should you have any inquiry, please contact us:

  • Classic track: aure 'at' nist.gov
  • Ockham track: paul.black 'at' nist.gov
  • Mobile track: michael.ogata 'at' nist.gov
  • General inquiries: aure 'at' nist.gov
Views