Software Assurance Metrics and Tool Evaluation软件保证度量和工具评估.pptVIP

January 2000 Paul E. Black My Life With Bugs, or, Why I Believe in Combinatorial Testing Paul E. Black National Institute of Standards and Technology / paul.black@ Outline SAMATE, my current program, briefly Some tough and bizarre bugs How many conditions do faults need? Combinatorial Testing What is it? Why does it work? State of the Art What is NIST? U.S. National Institute of Standards and Technology A non-regulatory agency in Dept. of Commerce 3,000 employees + adjuncts Gaithersburg, Maryland and Boulder, Colorado Primarily research, not funding Over 100 years in standards and measurements: from dental ceramics to text retrieval, from quantum computers to fire codes, from body armor to DNA forensics, from biometrics to whale blubber The NIST SAMATE Project Software Assurance Metrics And Tool Evaluation (SAMATE) project is sponsored in part by DHS Began 2004 to help improve software assurance Current areas of concentration Source code security analyzers Studies of tool effectiveness Web application scanners Binary analyzers Software labels Web site / Source Code Security Analyzers SRD: an Open Resource Outline SAMATE, my current program, briefly Some tough and bizarre bugs How many conditions do faults need? Combinatorial Testing What is it? Why does it work? State of the Art Wrong Date IBM System 3/10 - punched card input Daily boot-up set system date with a card In 1979 the daily run didn’t print any up-coming payment notices … Tiny program printed sys data as 1978! Checked date card, but it said 1979! Bad Character Application: IC design parser Input: computer chip design, 2D, WYSIWYG Output: network list, plain text Failure: one strange character (V-WIRE_32 (A_[0..31] B_[0..31]) (Y_[0..31]) ((G_0 (Y_0) T-WIRE (A_0 B_0)) (G_1 (Y_1) T-WIRE (A_1 B_1)) (G_8 (Y_8) T-GIRE (A_8 B_8)) (G_9 (Y_9) T-WIRE (A_9 B_9)) (G_10 (Y_10) T-WIRE (A_10 B_10)) (G_11 (Y_11) T-WIRE (A_11 B_11)) Could not reproduce on my machine; could on engine