Be Fussy About Fuzz Testing to Avoid Software Fails
Summarize With:
While the cost and effort that goes into software testing are high, what is enormously higher is the cost of a software failure. Did you know? Tricentis' 2018 Software Fail Watch projected that a whopping 3.6 billion people were affected by software failures last year, resulting in economic losses of over $1.7 trillion and a downtime of 268 years! There is no way you can take software testing lightly — for the end result is not just a financial loss but more significantly, a loss of name and customer trust.
The Buzz Around Fuzz Testing
Among the myriad types of software testing being undertaken by developers throughout the software development life cycle, fuzzing or fuzz testing has picked up steam of late. An automated software testing technique, fuzz testing involves inputting invalid, unexpected, or random data to a software and monitoring it for crashes, memory leaks, or failing assertions. By demonstrating the presence of bugs rather than their absence, fuzz testing exposes hidden vulnerabilities in a software.
Using fuzz testing to discover coding errors and security gaps in software involves feeding enormous amounts of random data to software in an attempt to make it crash. However, running a fuzzer — which can either be a file or a protocol — even for several weeks, and not finding a bug does not certify the software bug-free; after all, the software may still fail for an input that has not been executed, yet.
Here's how you fussy you should be with fuzz testing:
- Automate values: Since fuzz testing relies on the assumption that there are bugs in every software waiting to be discovered, using an automated program that feeds random inputs into the software is the most appropriate way to spot them. Rather than attempting to list down values that are likely to provoke a crash — which might take you ages — an automated fuzz test inputs a large number of illogical values that a normal programmer would never think of.
- Feed a large number of inputs: In order to spot vulnerabilities, use a fuzzer that produces a large number of inputs in a relatively short time. By using a toolchain that automates otherwise manual and tedious tasks, you can enable automated generation of failure-inducing inputs. For eg., google's OSS-fuzz project produced around 4 trillion test cases a week.
- Conduct dynamic program analysis: Using a static program analysis for fuzz testing might just analyze a software without executing it and report problems that actually don't exist. Hence it is advisable to carry out fuzz testing in combination with dynamic program analysis so as to generate an input that truly witnesses the stated issue.
- Automate bug triage: To effectively expose bugs and fix or patch the security critical ones with higher priority, you can group a large number of failure-inducing inputs based on their root cause and then prioritize each individual bug based on severity.
- Use sanitizers: To make a fuzzer more sensitive to failures, inject assertions that crash the program when a failure is detected. You can choose from different types such sanitizers to spot different kinds of bugs: deadlocks, undefined behavior, memory leaks, control-flow, etc.
- Employ test case reduction: In order to isolate that part of the input that is inducing the failure, a test case reduction (or automated input minimization) tool would eliminate as many input values as possible while still reproducing the original bug. This is especially helpful in cases where the failure-inducing input is large and hence proving difficult for testers to understand the root cause of the bug.
- Avoid random plugging: If your fuzz testing program reveals several bugs in your software, plugging them as they appear may not be the right way as random plugging might reduce the robustness of the software. Consider fundamentally hardening the file format through the use of checksums, XML, and grammar-based file formats.
- Verify everything: The most common mistake many testers make is to assume that because an instance of the software outputted the correct value once, it will output the same correct value every time. But that can be risky: the instance could have been overwritten, it could have been corrupted, it could have been modified by another program that had a bug or it could even have been intentionally altered in an effort to enhance the software's security. Hence, assume nothing. Verify everything.
Avoid Failures:
In a world where organizations are losing millions due to software fails (UK-based Provident Financial lost $2.4 billion worth of market value), fuzz testing greatly improves the confidence that an application is robust and secure against unexpected input. Although fuzz testing is simple, it often reveals serious bugs, defects, and potential avenues of attack that must be fixed before the software is shipped. By automatically injecting random permutations of data into a target software until one of those permutations exposes a weakness, Fuzz Testing is helping testers discover software faults and rectify them early in the cycle. Just a few bytes out of place can bring the entire software product down. In such a scenario, fuzz testing helps to prevent such disastrous outcomes. And that's not fuzzy logic.