NIST Response to SP 500-268 - 02/05/2007-01
From SAMATE
NIST: OUR RESPONSES TO COMMENT SP 500-268 - 02/05/2007-01 ON THE DRAFT Source Code Security Analysis TOOL SPECIFICATION ARE INTERSPERSED WITH THE MESSAGE.
Thanks! Here's another comment. If you point to CWE definitions, then what do you do about tools that find only a few cases in a category? I know of no existing code scanner that can find all common ways in which stack overflows happen. In fact, I recently played with a commercial scanner and it found only 1 out of 5 more or less common overflows in tests that I crafted. It found nothing wrong in 4 out of 5 tests, and I believe that it's one of the best code scanners available. I'm certain that I would find a similar situation in most of the other CWEs you listed. Will there be standard tests defined for tools to pass? It seems likely (and true according to Rice's theorem) that someone could always come up with a more obfuscated example (note that I don't consider the tests I used to be obfuscated) that the tools couldn't find. So, when does a tool meet the criteria? How well is good enough?
It seems to me that you have no choice but to limit the general case to some test cases, and possibly grade the tools on how many cases they can discover (hoping that vendors don't include code specific for the tests you use).
Regards,
Pascal
NIST: SINCE WE KNOW THAT NO TOOL CAN FIND ALL POSSIBLE INSTANCES OF A WEAKNESS, WE WILL BE NARROWING THE SCOPE OF THE GENERAL CASE. WHAT WILL THAT TELL YOU ABOUT THE TOOL? AS WITH CONFORMANCE TESTING, IT WILL TELL YOU WHAT A TOOL CAN CATCH UNDER PARTICULAR CIRCUMSTANCES, BUT NOT ALL CIRCUMSTANCES. AS WE STATE IN THE SPECIFICATION, IT DOES NOT MEAN THAT A TOOL WILL CATCH ALL CASES OF A WEAKNESS, PARTICULARLY THOSE THAT FALL OUTSIDE THE DOMAIN OF THE SPEC. HOWEVER, RIGHT NOW, THERE IS NO WAY TO EVEN MEASURE TOOLS AGAINST EACH OTHER WITHIN A CONSTRAINED SET OF OF WEAKNESS CASES. SO WE SEE VALUE IN THIS EFFORT IF WE CODE OUR TEST CASES TO REFLECT THE ESSENTIAL “DNA” OF A WEAKNESS, AND THEN LAYER OBFUSCATING CODE ON TOP OF IT.
NIST: THE MORE INTERESTING STUDY WILL BE IN IDENTIFYING WHICH TYPES OF OBFUSCATIONS ( WHAT WE CALL “CODE COMPLEXITIES” IN OUR SPECIFICATION) A TOOL CAN HANDLE, AND WHAT COMPLEXITIES TOOLS (AS A WHOLE) HAVE DIFFICULTY WITH.