In this assignment you will use two static analysis tools to automatically detect potential defects.
The first static analysis tool is Facebook's Infer, which focuses on memory errors, leaks, race conditions, and API issues. Infer is open source.
The second one is SpotBugs, which is a static tool for analyzing specifically Java programs. It can be used to detect a wide range of bugs (spanning more than 400 patterns), including those related to bad practice, (single-threaded and multi-threaded) correctness, security, and even performance! SpotBugs is open source.
You may work with a partner for this assignment. If you do you must use the same partner for all sub-components of this assignment. Use Gradescope's partner selection feature. Only one partner needs to submit the report on Gradescope, but if you both do, nothing fatal happens.
You should use the setup from HW0 to run Infer.
As an optional alternative, many users report that Facebook's Infer tool does not run on the Windows Subsystem for Linux (WSL) or similar shortcuts for using Ubuntu- or Linux-like interfaces. Headless Virtual Box configurations (instructions) are reported to work very well. Officially, however, the HW0 setup is the supported configuration for the class.
It is your responsibility to download, compile, run and analyze the subject program and associated tools (or use the precompiled one: we recommend using the precompiled version since it is known to work with the HW0 setup). Getting the code and tools to work in some manner is part of the assignment. You can post on the forum for help and compare notes bemoaning various architectures (e.g., windows vs. mac vs. linux, etc.). Ultimately, however, it is your responsibility to read the documentation for these programs and tools and use some elbow grease to make them work.
We will make use of the lighttpd webserver (pronounced "lighty"), version 1.4.17, as one subject program for this homework. A local mirror copy of lighttpd-1.4.17.tar.gz is available, but you can also get it from the original website. It is about 55,000 lines of code in about 90 files. While somewhat small for this class, some analysis tool licenses have LOC limits or scalability issues, so it was chosen as an indicative compromise.
While not as large or popular as apache, at various points lighttpd has been used by YouTube, xkcd and Wikimedia. Much like apache, old verisons of it have a number of known security vulnerabilities.
The Common Vulnerabilities and Exposures system is one approach for tracking security vulnerabilities. A CVE is basically a formal description, prepared by security experts, of a software bug that has security implications.
There are at least ten CVEs associated with lighttpd 1.4.17 tracked in various lists (such as cvedetails or mitre). For example, CVE-2014-2324 has the description "Multiple directory traversal vulnerabilities in (1) mod_evhost and (2) mod_simple_vhost in lighttpd before 1.4.35 allow remote attackers to read arbitrary files via a .. (dot dot) in the host name, related to request_check_hostname." You can dig into the information listed in, or linked from, a CVE (or just look at subsequent versions of the program where the bug is fixed!) to track down details. Continuing the above example, mod_evhost refers to source file mod_evhost.c, mod_simple_vhost refers to file mod_simple_vhost.c, and request_check_hostname is in file request.c. You will need such information when evaluating the whether or not a tool finds these security bugs.
ActiveMQ is message broker, written in Java, which supports multiple industry standards. We will make use of it, version 5.15.0, as another subject program for this homework. A local mirror copy of activemq-version.tar.gz is available, which you can also obtain from the original website.
There are at least ten CVEs associated with activemq-5.15.0. For example, CVE-2023-46604 has the description "The Java OpenWire protocol marshaller is vulnerable to Remote Code Execution. This vulnerability may allow a remote attacker with network access to either a Java-based OpenWire broker or client to run arbitrary shell commands by manipulating serialized class types in the OpenWire protocol to cause either the client or the broker (respectively) to instantiate any class on the classpath. Users are recommended to upgrade both brokers and clients to version 5.15.16, 5.16.7, 5.17.6, or 5.18.3 which fixes this issue." You can find details about CVEs, similar to lighttpd above.
The Infer tool is a static analyzer — it detects bugs in programs without running them. The primary website is fbinfer.com.
Unfortunately, some versions of Infer can be obnoxious to build and install, despite their handy installation guide. Also, many users report that Infer does not run on Windows Subsystem for Linux (WSL) or similar setups; a headless Virtual Box configuration (instructions) is recommended.
Instead (but see above about "your responsibility"), a precompiled, runs-on-the-HW0-setup (Ubuntu 16.04.2 LTS GNU/Linux 4.4.0-34-generic x86_64) version of Infer is available locally here. You must be signed into your University of Michigan account to access this. (warning: 265 MB; you will likely want to use scp to transfer the .tar.gz file to your HW0 setup and unpack it there). Once you have transferred and unpacked it, the main binary can be found at infer-linux64-v0.13.0/infer/bin/infer. You can use either the pre-compiled one or compile it yourself for full credit (any version at all of Infer is full credit).
Once you have Infer built or downloaded, applying it to lighttpd should be as simple as:
$ sudo apt install make $ sudo apt install python-minimal $ cd lighttpd-1.4.17 $ sh configure $ /path/to/infer/bin/infer run -- make
That should produce output similar to (but everything is fine if you get very different numbers):
make[1]: Leaving directory '/home/weimer/src/lighttpd-1.4.17'
Found 88 source files to analyze in /home/weimer/src/lighttpd-1.4.17/infer-out
Starting analysis...
legend:
"F" analyzing a file
"." analyzing a procedure
FFFFFFFFFF.....F...FF....F..FF.F..F....................................................................................FF.................................................F...........F..................F..................F...........................................................................F....................................................................F........................................................F.......F.................F...............F.......FF.............F...................F.............F.........F...F.................F...................................F............FF.F.....F.......................F.....FF..............F..F........FF..........FF.............FF.......FF.F....F......F......FFF..............F.........F...F......F...........F.......FF..........F.F...........F...F..F.......F..F...F........................F..F.........F....F........F.....F..F..........F............F....F...................F................................................................................................................................................
Found 308 issues
src/joblist.c:19: error: NULL_DEREFERENCE
pointer `srv->joblist->ptr` last assigned on line 16 could be null and is dereferenced at line 19, column 2.
17. }
18.
19. > srv->joblist->ptr[srv->joblist->used++] = con;
20.
21. return 0;
...
Summary of the reports
NULL_DEREFERENCE: 145
DEAD_STORE: 94
MEMORY_LEAK: 65
RESOURCE_LEAK: 3
QUANDARY_TAINT_ERROR: 1
(Before you worry about getting different numbers, double-check the prose above: it is fine to get different numbers. Similarly, it is common for this tool to only report a few "types" of defects: if you only see a few "types" of defects, you are running the tool correctly, even if SpotBugs reports more "types" of defects.) You will have to read through the output carefully and analyze the reported defects. Some will be true positives (i.e., real bugs in the code) and some will be false positives (i.e., spurious warnings that do not correspond to real bugs).
Running Infer on jfreechart-1.5.0 is similarly direct.
$ cd jfreechart-1.5.0
$ /path/to/infer/bin/infer run -- mvn compile
Capturing in maven mode...
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building JFreeChart 1.5.0
...
Found 640 source files to analyze in /home/weimer/src/jfreechart-1.5.0/infer-out
Starting analysis...
...
Found 69 issues
src/main/java/org/jfree/data/xml/DatasetReader.java:73: error: RESOURCE_LEAK
resource of type `java.io.FileInputStream` acquired to `in` by call to `FileInputStream(...)` at line 72 is not released after line 73.
71. throws IOException {
72. InputStream in = new FileInputStream(file);
73. > return readPieDatasetFromXML(in);
74. }
...
Summary of the reports
THREAD_SAFETY_VIOLATION: 43
NULL_DEREFERENCE: 22
RESOURCE_LEAK: 4
While times will vary, some students have reported that running Infer on jfreechart can take five hours.
You can find Infer's output in the infer-out folder.
SpotBugs, similar to Infer, is also a static program analyzer, but it is specialized to Java. That is, while it can detect a wide range of bugs in Java code, it (as of now) cannot analyze programs written in other languages. SpotBugs is a successor of FindBugs; the latter at this point is abandoned, but the former is actively maintained (at least up to this point). The official SpotBugs website is https://spotbugs.github.io/ and the source code can be found here: https://github.com/spotbugs/spotbugs.
For the Winter 2025 semester, we make SpotBugs reports available for ActiveMQ and a few other Java programs, such that students can have access to such reports to complete this homework without having to install and build SpotBugs.
Below are some additional subject programs that you may choose to in this homework. Note that these programs are written in different languages. You may choose one in Java for SpotBugs.
Note that the report requires you to choose an additional program (from the list above) and analyze it.
In this section we detail previous student issues and resolutions:
Question: When I run infer.exe run -- make or infer run -- mvn compile I get errors like InferModules__SqliteUtils.Error or Maven command failed.
Answer: The most common issue is that Infer does not always run well on Windows Subsystem for Linux (WSL) or similar shortcuts to get a Linux- or Ubuntu-like interface on another OS. We strongly recommend a headless Virtual Box setup (instructions).
Question: When I try to run Infer, I get cannot execute binary file: Exec format error..
Answer: One student reports: "Finally got it. Turns out I was using a 32 bit processor (i386) so even when I set up my vm as 64 bit, it couldn’t run any x86-64 binaries. Fixed it by installing a 64 bit vdi. https://appuals.com/fix-cannot-execute-binary-file-exec-format-error-ubuntu/
Question: I see Maven command failed: *** mvn compile -P infer-capture when I try to run Infer.
Answer: Some students have seen success with:
sudo apt-get install cobertura maven sudo apt-get install openjdk-8-jdkOthers reported that "I ended up having to setup an Ubuntu 16.04 VM in VirtualBox".
You must write a detailed PDF report reflecting on your experiences with these static analysis tools. Your report must include your University email address(es). In particular, all of the following are required:
The grading staff will select a small number of excerpts from particularly high-quality or instructive reports and share them with the class. If your report is selected you will receive extra credit.
Students are often anxious about a particular length requirement for this report. Unfortunately, some students include large screenshots and others do not, so raw length counts are not as useful as one might hope. Instead, I will say that in HW4 (and HW6, upcoming) we often see varying levels "insight" or "critical thinking" from students. I know that's the sort of wishy-washy phrasing that students hate to hear ("How can I show insight?"). But some of the questions (e.g., "what does cost mean in this report?") highlight places where some students give one direct answer and some students consider many nuances. Often considering many nuances is a better fit (but note that if you make things too long you lose points for not being verbose or readable -- yes, this is tough).
Let us consider an example from the previous homework. Suppose we had asked you whether mutation testing worked or not. Some students might be tempted to respond with something like "Yes, mutation testing worked because it put test suite A ahead of test suite B, and we know A is better than B because it has more statement coverage." That's a decent answer ... but it overlooks the fact that statement coverage is not actually the ground truth. (It is somewhat akin to saying "yes, we know the laser range finder is good because it agrees with my old bent ruler".) Students who give that direct answer get most of the credit, but students who explain that nuance, perhaps finding some other ways to indicate whether mutation testing worked or not, and what that even means, would get the most credit (and will also have longer reports). Students are often concerned about length, but from a grading perspective, the real factor is the insight provided.
Submit a single PDF report via Gradescope. You must include your name and UM email id (as well as your partner's name and email id, if applicable).
There is no explicit format (e.g., for headings or citations) required. For example, you may either use an essay structure or a point-by-point list of question answers.
In this section we detail previous student issues and resolutions:
Question: When I try to run infer on lighttpd, it dies when trying to build the first file with an error like:
External Error: *** capture command failed: *** make *** existed with code 2 Run the command again with `--keep-going` to try and ignore this error.
Answer: Some students have reported that being careful to run all of the commands, such as with this exact sequences, works:
wget https://eecs481.org/hw4/infer-linux64-v0.13.0.tar.gz wget https://eecs481.org/hw4/lighttpd-1.4.17.tar.gz tar xzf infer-linux64-v0.13.0.tar.gz tar xzf lighttpd-1.4.17.tar.gz cd lighttpd-1.4.17 sh configure ../infer-linux64-v0.13.0/infer/bin/infer run -- make
Question: When I try to run infer, I get some output but then Fatal error: out of memory. What can I do?
Answer: You may need to assign your virtual machine more memory (see HW0 for setup). You may also need to choose a different subject progam. Some students have reported this when analyzing cpython — perhaps a different program would work for you.
Question: When I try to run infer on libpng, it dies when trying to build the first file with an error like:
External Error: *** capture command failed: *** make *** existed with code 2 Run the command again with `--keep-going` to try and ignore this error.
Answer: One student reported that being careful to install all of the required build utilities, such as with this exact sequences, resolved the issue:
sudo apt install make sudo apt install python-minimal
Question: When I try to run infer on a program (e.g., lighttpd), it seems to produce no reports or output when I run infer run -- make. Instead, if I look very carefully at the output, hidden near the bottom is a warning like:
** Error running the reporting script:
Answer: You must have your directories set up so that infer/bin/infer is "next to" other files like infer/lib/python/report.py. Infer uses those extra scripts to actually generate human-readable reports. If you tried to copy the infer binary somewhere else, it won't work. Make sure you have all of the components of infer in consistent locations.
Question: I'm not certain why "false positives" and "false negatives" are relevant for comparing the tools. I'm also not certain how we tell if something is a false positive or a false negative. Can you elaborate?
Answer: We can elaborate a bit, but I will note that this aspect of the assignment is assessing your mastery of course concepts. That is, why false positives and false negative might be important, and how to distinguish between them, are critical software engineering concepts and might come up on the exam as well. You may want to double-check your notes on these, including on the readings. Now for more detail:
Suppose you are able to determine the false positive rate of one tool — or approximate it. For example, suppose you find that Tool #1 produces twice as many false positives as Tool #2. Well, then you might combine that with some of the reading for the class. For example, the FindBugs reading notes "Our ranking and false positive suppression mechanisms are crucial to keeping the displayed warnings relevant and valuable, so that users don’t start ignoring the more recent, important war" (among other comments on false alarms), while the Coverity reading notes "False positives do matter. In our experience, more than 30% easily cause problems. People ignore the tool. True bugs get lost in the false. A vicious cycle starts where ..." among other comments on false alarms. You might also check out the Parnin and Orso reading, and so on.
Something similar could be considered for false negatives. To give a prose example rather than a reading list this time, a report might include a claim like: "Many developers will dislike a tool that claims to find Race Conditions but actually misses 99% of them. If the tool has that many false negatives, developers will feel they cannot gain confidence in the quality of the software and will instead turn to other techniques, such as testing, that increase confidence in quality assurance." I'm not saying that is a good or a bad argument, but it is an example of the sort of analytic text or line of reasoning that might be applicable here.
Students often wonder: "How do I know if the tool is missing a bug?" Unfortunately, that's a real challenge. There are at least two ways students usually approach that problem, and both require labor or effort. Similarly, determining if a report is a false alarm usually requires reading it and comprehending the code nearby.
I can't really say much more here without giving away too much of what we expect from you on this part of the assignment, but I can reiterate the soundness and completeness (false positives and false negatives) are significant concepts in EECS 481 and that you should include them, coupled with your knowledge of the human element of such tools, in your assessment of the tools.