Introduction to SAMATE

Software Assurance covers both the property and the process to achieve it. From CNSS, National Information Assurance Glossary, CNSS Instruction No. 4009, April 6, 2015, page 115 and Celia Paulsen and Robert Byers, NISTIR 7298 Rev. 3, Glossary of Key Information Security Terms, July 2019, Software Assurance is:

• The level of confidence that software functions as intended and is free of vulnerabilities, either intentionally or unintentionally designed or inserted as part of the software throughout the lifecycle.
• The planned and systematic set of activities that ensure that software life cycle processes and products conform to requirements, standards, and procedures.

The NIST Software Assurance Metrics And Tool Evaluation (SAMATE) project is dedicated to improving software assurance by developing methods to enable software tool evaluations, measuring the effectiveness of tools and techniques, and identifying gaps in tools and methods. Our major efforts include defining bug classes, collecting a corpus of programs with known bugs, and enabling a better understanding of tool effectiveness.

The Software Assurance Reference Dataset (SARD) is a growing collection of almost two hundred thousand test programs with documented weaknesses. Test cases vary from small synthetic programs to large applications. The programs are in C, C++, Java, PHP, and C#, and cover over 150 classes of weaknesses. The Acknowledgments and Test Case Descriptions page describes the content. The Manual explains how to use the SARD website.

The Static Analysis Tool Exposition (SATE) is a recurring study designed to advance research in static analysis tools that find security-relevant weaknesses in source code. Briefly, NIST provides a set of programs to tool makers, then they run their tools and return tool outputs for analysis. Tool makers and organizers share their experiences and observations at a workshop. The analysis report is made publicly available later. As of 2021, we have had six SATE events.

The Bugs Framework (BF) is a structured, complete, orthogonal, and language-independent classification of software weaknesses (bugs). Each BF class is a taxonomic category of a kind of bugs, defined by all possible cause to consequence transitions, a set of operations, and a set of attributes. BF allows unambiguous descriptions of software vulnerabilities.

The AI Bug Finder is a modular and expandable test bed for evaluating AI-based methods for finding bugs in source code.

Other work includes:

• Mobile Application Tool Testing focuses on tools that check mobile applications.
• Source Code Security Analysis specifications, background, etc.
• Web Application Scanners – These tools crawl a web application’s pages and search for vulnerabilities by simulating attacks on it. A specification is published as NIST Special Publication 500-269.
• Software Assurance Tool Taxonomy.
• SAMATE Publications is a complete list of papers, workshops, and presentations.