The SAMATE Project Department of Homeland Security

Introduction to SAMATE


The NIST SAMATE (Software Assurance Metrics And Tool Evaluation) project is dedicated to improving software assurance by developing methods to enable software tool evaluations, measuring the effectiveness of tools and techniques, and identifying gaps in tools and methods. This project supports the Department of Homeland Security's Software Assurance Tools and R&D Requirements Identification Program - in particular, Part 3, Technology (Tools and Requirements), the identification, enhancement and development of software assurance tools. The scope of the SAMATE project is broad: ranging from operating systems to firewalls, SCADA to web applications, source code security analyzers to correct-by-construction methods.


Support Tool Evaluation

One of our goals is to establish a methodology for evaluating software assurance tools. We do this by developing tool specifications, test plans, and test sets. The results provide information for tool developers to improve tools, for users to make informed choices about acquiring and using software tools, and for interested parties to understand tool capabilities. Our efforts include:

Source Code Security Analyzers – This class of software tools examines source code files for security weaknesses and potential vulnerabilities. We published a specification as NIST Special Publication 500-268 v1.1 and a draft test plan for source code security analyzers as NIST Special publication 500-270.

Web Vulnerability Scanners – These tools crawl a web application’s pages and search for vulnerabilities by simulating attacks on it. A specification is published as NIST Special Publication 500-269. A test framework for web application scanners appeared in a paper entitled “Building a Test Suite for Web Application Scanners” and published in 41st Hawaii International Conference on System Sciences (HICSS), January 2008.

A new effort on Binary Code Scanners - Similar to source code security analyzers, this class of tool analyzes a compiled binary application, including libraries, and provides a report of code weakness over the entire application.

The Software Assurance Reference Dataset (SARD) - A community repository of example code and other artifacts to help end users evaluate tools and developers test their methods. Currently, the SARD consists of over 1800 test cases which encompass a wide variety of flaws, languages, platforms, and compilers.

Third annual Static Analysis Tools Exposition, which is in progress. The goals are to

  • enable empirical research based on large test sets,
  • encourage improvement of tools, and
  • speed tool adoption by objectively demonstrating their use on real software.

Briefly, we pick a set of programs. Tool makers run their tools on them and return the tool reports. We perform a limited analysis of the reports and note interesting aspects. We and the participants report our experience and results at a workshop. We make the test set, tool reports, and results publicly available later.


To learn, bring a community together, and share results, we host or co-host workshops, conferences sessions, and other meetings from time to time. Here are the latest.

Static Analysis Workshop (SAW) featuring Static Analysis Tools Exposition (SATE 2008), June 2008, Tucson, AZ.

Static Analysis Summit II (SAS II), November 2007, Fairfax, VA.

Static Analysis Summit (SAS), June 2006, Gaithersburg, MD.

Workshop on Software Security Assurance Tools, Techniques, and Metrics (SSATTM), November 2005, Long Beach, CA.

Workshop on Defining the State of the Art in Software Security Tools, August 2005, Gaithersburg, MD.

Studies on Software Assurance

Effect of Static Analysis tools on Software Security: Preliminary Investigation, Third Workshop on Quality of Protection (QoP), Oct 2007.

SAMATE and Evaluating Static Analysis Tools, International Conference on Reliable Software Technologies – Ada Europe, June 2007.

SAMATE Publications is a complete list of papers, workshops, and presentations.

Other Work

We started a new effort for capturing and sharing facts about a piece of software, including its claims. For example, a software fact sheet may have information about the product’s pedigree, development process, testing comprehension, security, safety, and quality.


SAMATE began in Fall, 2004.