As they say: “If you can’t measure it, you can’t manage it.”

There’s a serious push in the industry right now for security metrics. People really want to know what works — because this ain’t it. But where can we find hard data?

What about fuzzers — the automated, randomized testers that have been so good at finding bugs through sheer brute force?

I had a hypothesis, borne of an optimism that’s probably a bit out of place in our field: I believe, after ten years of pushing for more secure code, software quality has increased across the board — at least in things that have been under active attack. And, in conjunction with Adam Cecchetti and Mike Eddington of Deja Vu Security (and the Peach Project), we developed an experiment to try to show this.

We fuzzed Office and OpenOffice. We fuzzed Acrobat, Foxit, and GhostScript. And we fuzzed them all longitudinally — going back in time, smashing 2003, 2007, and 2010.

175,334 crashes later, we have some…interesting data.  Here’s a view of just what happened to Office — and OpenOffice — between 2003 (when the Summer of Worms hit) and 2010.

Things got better! This isn’t the end all and be all of this approach.  There’s lots that could be going wrong.  But we see similar collapses in PDF crash rates too.  Slides follow below, and a paper will probably be assembled at some point.  But the most important thing we’re doing is — we’re releasing data!  This is easily some of the most fun public data I’ve played with.  I’m looking forward to seeing what sort of visualizations come of them.  Here’s one I put together, looking at individual crash files and their combined effect on Word/Writer 2003/2007/2010:

I’ve wanted to do a study like this for a very long time, and it’s been pretty amazing finally doing a collaboration with DD and Adam!  Thanks also to the vendors — Microsoft, the OpenOffice Team, Adobe, Foxit, and the guys behind GhostScript.  Everyone’s been cool.

Please send feedback if you have thoughts.  I’d love to post some viz based on this data!  And now, without further ado…

• CanSecWest Slides
• Crash Summary (TSV) (MySQL Dump)
• Crash Profile Spreadsheet (XLS) (Google Spreadsheet)
• (Very Rough) Crash Profile Generator

Science!  It works.  Or, in this case, it hints of an interesting new tool with its own set of downsides 😉