March 6, 2024 – Reading time: 9 minutes
In times of advancing complexity of technology and growing concerns about cybersecurity, the need for robust software testing methods is increasing. Fuzzing as a negative testing method is an additional approach to address cybersecurity vulnerabilities and software robustness issues Its adoption is also driven by commercial and contractual constraints as well as stakeholder requirements. This article introduces fuzzing and describes its importance for improving software testing and quality assurance. The discussion covers the specific benefits of fuzzing, including its role in automated testing, identifying potential security vulnerabilities and improving software robustness. Then we discuss the challenges of applying this technique in the world of embedded systems and how to address them.
Introduction to Fuzzing
The stability and security of software is becoming increasingly important due to cybersecurity and vulnerabilities concerns in today’s technology landscape. In particular, the predicted 16% annual economic growth of the IoT industry[1] and the expected 35% increase in software sales in the automotive industry over the next five years[2] underlines the need for robust software testing methods. The testing and validation strategy must keep pace with increasing software complexity and requirements. At this point, we aim to promote the growth of fuzzing in embedded systems, a dynamic software testing technique also known as fuzz testing.
Fuzzing, in its original conception, involves providing invalid or random data as inputs to a software system. The main goal is to detect errors, invalid system states or behaviors, caused by memory leaks or buffer overflows. Tracing the history of fuzzing, its evolution has been unusual. From its inception in the 1980s as a primitive idea of a monkey typing on a keyboard, fuzzing has evolved into a highly sophisticated, automated testing technique. With the increasing complexity and integration of software systems, the need for advanced fuzzing methods has grown onsiderably over the years.
Three main fuzzing methodologies have emerged: mutation based, generation based, and guided fuzzing. Mutation based fuzzing involves altering existing data samples, known as the test corpus, to create new test inputs based on selected rules or random decisions. By continuously modifying these samples, mutation based fuzzing can effectively probe for unforeseen issues in the software being tested. This methodology evolved to generation based fuzzing to reflect the structure of real world data. Here the fuzzing engine generates new data based on predefined models. Guided fuzzing, a quite recent advancement, employs a feedback loop to adapt and refine the fuzzing process based on previous results. Mutation based fuzzing is often considered as a good start in a wide range of environments due to its simplicity. Guided fuzzing, while resource intensive and target dependent, is highly effective in increasing code coverage.
With cyber security expenses doubling in the last five years[3], the pressure to identify and mitigate vulnerabilities early in the software development lifecycle has intensified. Fuzzing has been proven multiple times to be effective in uncovering unknown security vulnerabilities (see Shellshock[4], Heartbleed[5]) and is therefore an invaluable tool, enabling developers and testers to proactively address potential security and stability issues. The relevance of fuzzing is highlighted by the results from tools like ClusterFuzz, which detected an average of 12 bugs per week in Chromium (Google Chrome) in 2023 and so far, found more than 25,000 bugs in Google products.
In addition to its technical benefits discussed later in this article, fuzzing is increasingly recognized as a crucial requirement from both an economic and contractual standpoint. Various standards and regulations now mandate the implementation of fuzzing in certain sectors, for example in the automotive industry. Here the regulatory framework ISO 21434 can be named, which specifically
recommends the use of fuzzing during cyber security validation. This will be generalized with the upcoming EU Cyber Resilience Act, which emphasizes the need for robust measures to strengthen the resilience of digital products. Fuzzing would be a viable tool in complying with this regulation, ensuring that digital products are resilient against a wide range of cyber threats. Similarly, OEMs started to mandate fuzzing as a part of their penetration testing requirements. This contractual obligation increases the confidence that all embedded systems within vehicles are thoroughly tested against potential cyber threats, thereby increasing the security and safety of the overall vehicle. It may therefore come as no surprise that fuzzing is being used in some parts of the automotive industry test landscape in the form of a black box approach. However, these methods are usually only mutation-based and test system interfaces and do not implement the newer generations of fuzzing.
From an economic perspective, the early detection of security vulnerabilities is critical in preventing substantial economic damages. Security breaches can lead to severe financial losses, legal liabilities, and reputational damage. The existing testing methods can be extended by fuzzing. This enables development teams to proactively identify and address vulnerabilities and significantly reduce the risk of security incidents.
Benefits of Fuzzing
As explained in the previous chapter, there are already incentives for fuzzing from an economic and contractual perspective. But let’s now look at the specific technical advantages that this testing approach has to offer.
1. Automated testing
One of the strengths of fuzzing is its ability to automate the testing process. This automation facilitates extensive coverage and testing efficiency that would be impractical, if not impossible, with manual methods. As only a fuzzing harness is needed, the time and resources required are lower compared to other test methods.
2. Detecting security vulnerabilities
Fuzzing excels in uncovering memory leaks and buffer overflows, which could lead to potential security vulnerabilities. In principle, these could also be found by static code analysis, but fuzzing differs in terms of execution and methodology, and therefore also in terms of effectiveness. By systematically exposing the software to unexpected and edge-case scenarios, fuzzing improves the likelihood of vulnerability detection before they can be exploited. It therefore makes sense to integrate fuzzing into continuous integration (CI) systems and executing fuzzing in a loop. This proactive approach is crucial in securing the software development lifecycle even beyond software deployment.
3. Monitoring for unexpected system behavior
Another key benefit of fuzzing is its ability to reveal unexpected system behaviors, including crashes and undefined behaviors. These issues might remain hidden under normal testing conditions as these are mostly for verification. The automated approach of fuzzing generates negative test cases.
4. Validating using properties
The negative test approach can be extended to include the definition of properties in the form of assertions. In this case, the test corpus is systematically tested against these properties and fuzzing can contribute to the validation of the software functionality. For example, the system behavior is validated even if the initial test vectors have inadvertently omitted cases.
5. Improving software robustness
Monitoring for crashes, timeouts and runtime violations is an essential aspect of fuzzing. This not only improves the software’s robustness but also increases the confidence that it can handle a wide range of operational scenarios and stress conditions without failure. As sufficiently complex systems reach hardly achievable complete test coverage, continuous fuzzing is invaluable and provides a way to enable long-term evaluation with an ever-increasing test corpus due to its automated nature.
6. Incorporating regression testing
By adding problematic inputs identified during fuzzing or other means to the test corpus, it effectively accomplishes regression testing. This ensures that previously identified issues are consistently tested against in subsequent software versions and prevents recurrences of the same bugs. At the same time, the reverse process can be used to derive new dedicated test cases from the corpus. As the test corpus grows over time, it becomes a valuable resource for testing new variants and even projects. This is further facilitated using standardized interfaces (e.g. AUTOSAR).
7. Testing beyond code
Fuzzing is unique in that it expands the scope by testing the implementation and not just source code. This approach is important in identifying issues that arise not just from coding errors but from the interaction of software and hardware.
Bringing fuzzing to the embedded world
The transition of fuzzing into the embedded domain, for example in automotive contexts, presents a unique set of challenges that differ from traditional software environments in several key aspects. These challenges require tailored solutions and a deep understanding of embedded systems.
One of the foremost challenges is platform dependence. Many automotive systems are built on platforms like AUTOSAR, which require that fuzzing tools and techniques need to be compatible with its architecture. Additionally, the hardware dependence due to complex system-on-chips need to be considered. These systems integrate multiple components into a single chip with various peripheral modules, which complicates the fuzzing process as the interaction between these components need to be emulated or stubbed during fuzzing. Implementing the required feedback loop for guided fuzzing is not straightforward. Limited debug access in many embedded systems hinders the ability of guided fuzzing techniques to observe the control flow and monitor the system’s responses. Finally, physical limitations, such as latency, may play a critical role as real-time properties of these safety systems add another layer of complexity.
Another challenge is the common statefulness of these systems, which needs to be reflected by the test corpus and differs from the expectations of traditional fuzzing tools. The initial integration of fuzzing into embedded systems is complex. Integrating fuzzing into existing build processes and continuous integration systems can be non-trivial, particularly in automotive systems where build processes and component dependencies are complex. Hardware-in-the-loop (HIL) tests and virtual equivalents are commonly used and involve a complex test setup. This requires the fuzzing process to be integrated to interact with actual hardware in test benches, further complicating the fuzzing process and increasing the test cycle times.
Given these challenges and differences, the solution for implementing fuzzing in the embedded world, particularly in automotive applications, needs to be system specific. With a good understanding of the system’s architecture and operational environment these challenges can be solved. Tailoring fuzzing techniques to meet these specific requirements is crucial for effective and efficient testing in this complex and critical field.
