Introduction

Fuzzcheck is a crate to easily find bugs in Rust code. Given a test function:

fn test_always_true(x: SomeType) -> bool {
    // ...
}

You can use fuzzcheck to find a value x: SomeType that causes test_always_true(x) to return false or crash 💥.

Technically, it is called an evolutionary, structure-aware fuzzing engine. It is coverage-guided by default, but can also use other observations to guide the evolution of the fuzz test.

Goal 🦄

Fuzzcheck’s goal is to be used as part of your development routine. You can quickly test small or complex function with it. It can find difficult corner-case bugs, minify complex test cases into easy-to-debug ones. It can find stack overflows and inputs that cause excessive computations or allocations.

Over time, by using fuzzcheck, you can automatically build a large corpus of values that trigger most edge cases in your code. This will help ensure that no new bugs are introduced when refactoring.

Requirements 🎟

Currently, it is only possible to use fuzzcheck on Linux and macOS. I'd like to add Windows support to it, and it shouldn't be complicated, but I need some help with it.

You also need a nightly version of the Rust compiler. This requirement is unlikely to change soon.

Design ⚙️

Fuzzcheck works by repeatedly running the test function test_always_true(x) with automatically generated values for x. For each invocation of test_always_true(x), it gathers observations about the code that was run and determines whether new observations were discovered. This enables it to build a list of values that are known to be interesting to test, which we call the “pool of interesting values”.

Then, fuzzcheck repeatedly draws values from that pool, slightly mutates them, and then feeds them to test_always_true again. It continues this process indefinitely, analysing the observations (e.g. code coverage) triggered by each test case in order to discover more and more interesting values to test.

Thus, fuzzcheck is composed of three components working together:

• a Mutator<T>, which can generate arbitrary values of type T, either from scratch or starting from a known interesting value.
• a Sensor, whose role is to gather observations about the execution of a test function. By default, this uses the code coverage information generated by the -C instrument-coverage option of the Rust compiler. But different kinds of observations can be used.
• a Pool, which listens to the observations from the Sensor and determines which values are interesting to test. It internally ranks the different values that were tested and ensures that the most interesting ones are mutated and re-tested more often.

In pseudo-code:

loop {
    let value = pool.get_mut();
    mutator.mutate(value);
    let observations = sensor.record(|| {
        test_function(value);
    });
    if pool.is_interesting(value, observations) {
        pool.add(value, observations);
    }
}

Example 👀

We can illustrate fuzzcheck’s strengths with a simple but unrealistic example. What follows are not instructions to use fuzzcheck (these start from the next section), but rather an example showing how fuzzcheck is different from other testing tools such as quickcheck or proptest. Imagine we have the following function:

fn this_should_always_succeed(xs: &[u8]) -> bool {
    if xs.len() >= 4 && xs[0] == 96 && xs[1] == 1 && xs[2] < 78 && xs[3] == 189 {
        false
    } else {
        true
    }
}

We can find a vector of bytes that makes the function fail by running the following test:

#[cfg(test)]
mod tests {
    #[test]
    fn fuzz_test() {
        let result = fuzzcheck::fuzz_test(super::this_should_always_succeed)
            .default_options()
            .stop_after_first_test_failure(true)
            .launch();
        assert!(!result.found_test_failure);
    }
}

We run:

cargo fuzzcheck tests::fuzz_test

which launches the fuzz test on the library target. Fuzzcheck progressively finds values which satisfy each condition inside fn this_should_always_succeed until, after about 2000 iterations, it prints:

Failing test case found. 
Saving at "fuzz/artifacts/tests::fuzz_test/642e66bf463956ed.json"

It found the bug and serialized the failing test case to the file 42e66bf463956ed.json, which contains:

[96,1,24,189]

The time to find this bug, on my machine, was 12 milliseconds.

While this was a simplistic example, the same process can be used for large functions that take very complex input types as arguments. However, note that the test function should run fairly fast, ideally in less than a tenth of a millisecond.