The SAMATE Project

NIST Workshop on

Software Measures and Metrics to Reduce Security Vulnerabilities


Tuesday, 12 July 2016

Green Auditorium

  National Institute of Standards and Technology  

Gaithersburg, Maryland, USA

Final Report

The final workshop report is available as NIST SP 500-320. It is available at DOI: 10.6028/NIST.SP.500-320.


The Federal Cybersecurity Research and Development Strategic Plan seeks to fundamentally alter the dynamics of security, reversing adversaries' asymmetrical advantages. Achieving this reversal is the mid-term goal of the plan, which calls for "sustainably secure systems development and operation." Part of the mid-term (3-7 years) goal is "the design and implementation of software, firmware, and hardware that are highly resistant to malicious cyber activities ..." and reduce the number of vulnerabilities in software by orders of magnitude. Measures of software play an important role.

Industry requires evidence to tell how vulnerable a piece of software is, what techniques are most effective in developing software with far fewer vulnerabilities, determine the best places to deploy countermeasures, or take any of a number of other actions. This evidence comes from measuring, in the broadest sense, or assessing properties of software. With useful metrics, it is straight-forward to determine which software development technologies or methodologies lead to sustainably secure systems.

The goal of this workshop is to gather ideas on how the Federal Government can best use taxpayer money to identify, improve, package, deliver, or boost the use of software measures and metrics to significantly reduce vulnerabilities. We call for position statements from one to three paragraph long. Position statements may be on any subject like the following:

  • existing measures of software that can make a difference in three to seven years,
  • means of validating software measures or confirming their efficacy (meta-measurements),
  • quantities (properties) in software that can be measured,
  • standards (in both étalon and norme senses) needed for software measurement,
  • cost vs. benefit of software measurements,
  • surmountable barriers to adoption of measures and metrics,
  • areas or conditions of applicability (or non-applicability) of measures,
  • software measurement procedures (esp. automated ones), or
  • sources of variability or uncertainty in software metrics or measures.

The output of this workshop and other efforts is a plan for how best the Federal Government can employ taxpayer money to significantly curtail software vulnerabilities in the mid-term. See for details.

Important Dates

  • 22 May: deadline to submit statements Submissions are closed.
  • 8 June: invitations to present sent Sent
  • 27 June: deadline for non-citizens to register (no on-site registration)
  • 5 July: deadline for US citizens to register (no on-site registration)
  • 12 July: Workshop


The workshop was at the U.S. National Institute of Standards and Technology (NIST) in Gaithersburg, Maryland.

Final Agenda

The program was a mix of presentations based on position statements and discussion. The workshop went from 9 am to 4:30 pm with breaks.

0900 Welcome, safety, and schedule, Paul E. Black, NIST

0910 Federal Cybersecurity Research and Development Strategic Plan, Greg Shannon, White House Office of Science and Technology Policy,

0915 Opening Remarks, William F. Guthrie, Chief, Statistical Engineering Division, NIST,

0930 Measuring Software Analyzability, Andrew Walenstein, BlackBerry,

1000 Dealing with Code that is Opaque to Static Analysis, James Kupsch, University of Wisconsin-Madison,

1030         break

1050 Composing processes for secure development using process control measures, William Nichols, Software Engineering Institute,

1110 Measure Early and Measure Often – SWAMP, Miron Livny,

1130         lunch

1300 CISQ Measures of Secure, Resilient Software, Dr. Bill Curtis, Executive Director, Consortium for IT Software Quality (CISQ),

1320 Mostly Sunny with a Chance of Cyber-doom, David Flater, NIST,

1340 Dynamically Proving That Security Issues Exist, Dr Andrew V. Jones, Vector Software,

1400 Charge to Breakout Groups, breakout into six groups

1420         break

1450 breakout reports: 6 × 5 minutes

1520 Toward Evidence-Based Low Defect Software Production, James Kirby Jr., US Naval Research Laboratory,

1540 Using Malware Analysis to Reduce Design Weaknesses, Carol Woody, Ph.D., Software Engineering Institute,

1600 Summary - Our Next Steps, Paul E. Black, NIST

1630 Workshop ends

General Chairs

Elizabeth Fong   (National Institute of Standards and Technology)

Paul E. Black   (National Institute of Standards and Technology)

Thomas D. Hurt   (Office of the Deputy Assistant Secretary of Defense for Systems Engineering - Joint Federated Assurance Center (JFAC) lead)

Program Committee

Paul E. Black

David Flater

Elizabeth Fong

D. Richard Kuhn

W. Timothy Polk

Cybersecurity National Action Plan (CNAP)

This page's URL is