Tips for more effective fuzz testing with AFL++ (nullprogram.com)

103 points by luu •

These are good tips, thank you

I used to think writing safe code in C/C++ was a matter of skill. However, after finding more than a few issues with AFL at a FAANG in code written by engineers with the highest engineering title, I changed my mind. Incredible tool.

It is a matter of skill to a large extent, though. A skilled programmer will write fewer bugs on average than an unskilled one. They'll know the patterns to avoid and have a higher bar for quality. Nobody can write 100% bug-free code, independent of programming language. Also, job title might correlate with skill but it doesn't rule out incompetence at the highest levels.

I think it's mostly a testing thing. Valgrind is a friend a lot of people haven't heard of for c/c++. Setting up good regression tests is an art. Maintaining them is even harder.

It doesn't really matter, the most skilled programmers are usually working on harder stuff, so the bug rate remains fairly consistent.

Sometimes writing shitty code is a result of competence, not incompetence. I've seen much better programmers than I ever will be writing terrible code because it was the fastest way to get a prototype running and find out whether the problem was worth writing good code for.

Writing safer C is a matter of skill.

Writing safe C is a matter of either radically different (much, much slower) process or better tooling or both. Skill is not sufficient.

>I used to think writing safe code in C/C++ was a matter of skill.

It's a matter of how many fucks the devs gave about doing quality work. You can find inexperienced people deliver quality work simply because they're passionate and they care (Half-Life was made in part by people with no prior game or coding experience), and you can find supposed rockstars deliver slop because for whatever reason they don't care, they just want to punch in, collect paychecks, then punch out.

FAANG stopped being a benchmark of skill a while ago. It's a skill of griding leetcode and 7 stage interviews to get it, then resting and vesting till you can jump ship to something better or get mass layd off.

> It's a matter of how many fucks the devs gave about doing quality work.

It's more a matter of how much management cares about quality, and how much time and how many resources they're willing to devote to it.

And that's a function of how much the quality of the code actually matters in the market.

FANG doesn't have resources, but indie coders and game studios do?

Notice I said how much management is willing to devote, not how much they have.

I think you’re a bit optimistic about the quality of a triple-A game’s codebase that was rushed to a holiday release date. I have slightly more tolerance for bugginess in a game.

You can have all the resources in the world but it won't matter if you spend them on something else.

Has there ever existed a C or C++ codebase of significant size that hasn't had exploitable memory corruption issues? I don't think we can keep saying it's a question of the dev's passion or skill or whatever, if no dev has ever managed to do it.

That's not to say memory corruption issues are the only kind of issues, and rewriting a mature C/C++ codebase into a memory safe language may well introduce more issues than it fixes...

I think that good use of static analysis and fuzz testing is a part of the skill to detail-oriented programming languages.

This is a great list.

I had been playing with and building fuzzers long before AFL, but its release and its ability to explore the code space in a smart way fundamentally changed the game. Never once have I improved a codebase so it could be fuzzed and didn't get a payoff of it finding bugs.

And the field is relatively young. Like the other year built a fuzzer with a transformer model as the internal learning engine and other cool stuff like https://antithesis.com

I know fuzzing has been around forever, but it is still surprising how many engineers I encounter who have never used one. I hate fixing bugs and love tools that just find bugs and gave me exact reproducible cases. It was nice to see GoLang get a built in fuzzer to try to help it become more mainstream.

And these days with AI where you need ways it both can't gamify (a unit test that simply returns 'pass') as well as automatic ways to eval generate code fuzzing is a great fit. And not to mention that it is cheaper to fuzz all night than continuously burn tokens looking for ways to crash the code.

Hacker News