NIST Response to SP 500-268 - 02/16/2007-01

[SAMATE Home | IntrO TO SAMATE | SARD | SATE | Bugs Framework | Publications | Tool Survey | Resources]

General comment
---------------

I have a general concern over the scope of this specification and how it will be perceived by readers.

The specification focusses on what has become known as retrospective "raise the floor" style detection of weaknesses. It also concentrates on programs written in the most popular (at least by volume of use) programming langauges. The organisers of the SAMATE effort have often stated that this is intended to address the most pressing need and to have the most impact on the largest number of projects, with which I agree completely.

The specification, though, is largely silent on the other end of the spectrum - the "ceiling" as it were. I do not for a moment believe that this silence is in any way malicious, but that the ceiling is simply out of scope for this document.

Section 1.2, for example, explicitly notes that Appendix A only addresses C, C++ and Java. How is a reader to interpret this section? Perhaps:

1) That there are no other languages?
2) That there are no other language more suitable for secure programming than those already mentioned?
3) That there are no tools for any other languages?
4) That "popularity" of a language is more imporant that its fitness-for-purpose for high-integrity software development?

and so on.

NIST: THESE ARE VALUABLE OBSERVATIONS. WE NOW CLARIFY WHY C, C++ AND JAVA ARE CHOSEN IN SECTION 1.2 (SCOPE)

The final clause in para 1 of 1.2 is also disappointing: "...although it is recognized that particular weaknesses may exist in other languages as well." This may leave the reader with entirely the wrong end of the proverbial stick. Is it not equally important to point out that other languages can _prevent_ several significant classes of weakness?

NIST: WE NOW MAKE THE READER AWARE OF THE VALUE OF MORE SECURE LANGUAGES IN SECTION 1.2. THE INTENT OF THIS SPECIFICATION IS WHAT CAN BE DONE, NOT TO DESCRIBE A BEST PRACTICE.

Compliance
----------

Section 2.2 lists mandatory requirements that a tool "must be able to accomplish." SCA-RM-1 says "Identify of the code weaknesses listed in Appendix A."

Can a langauge/tool claim compliance with SCA-RM-1 by preventing all such weaknesses, rathing than identifying their presence?

NIST: STRICTLY SPEAKING, NO. THE SPEC ASKS IF A TOOL CAN FIND CERTAIN WEAKNESSES, NOT IF A METHODOLOGY CAN KEEP THEM OUT OF FINISHED SOFTWARE. HOWEVER, WE SEE VALUE IN REFERRING TO THIS SPEC AND CLAIMING THAT A LANGUAGE/TOOL ABSOLUTELY PRECLUDES ALL SUCH WEAKNESSES.

What if tool X doesn't process C, C++ or Java? Can I claim compliance at all with this specification? 

NIST: AGAIN, STRICTLY SPEAKING, NO, NOT AT THIS TIME. IF THIS SPEC GROWS TO ENCOMPASS REQUIREMENTS FOR LANGAUGES BEYOND THESE THESE, PERHAPS IT CAN.

Minor comments
--------------

The word "conformant" appears in several places. This not listed in any dictionary that I have ever seen.

NIST: THANK YOU. WE ADDED THAT DEFINITION TO THE GLOSSARY

Section 1.5 lists "Weakness Suppression System" twice.

NIST: FIXED

- Rod Chapman, SPARK Team, Praxis High Integrity Systems