Static Analysis of Python Reference Counts
I've written a static analysis tool that can detect reference-counting errors made in Python extension modules written in C. We'll run the tool on all such code in Fedora 17 and make an effort to fix as many problems as time allows.
- Name: Dave Malcolm
- Email: firstname.lastname@example.org
- Targeted release: Fedora 17
- Last updated: 2012-03-05
- Percentage of completion: 75%
The code works, and has found real bugs, but still contains bugs itself. It's only been run on a small subset of the Python code in Fedora.
- the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
- wrote an automated script for running the tool on a mock build, and generating a triaged report on the issues found
- created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472
- only run it on source files that include <Python.h> (implemented in git; not yet in a tarball release)
- automated running it on all code in Fedora using mock, injecting the plugin
IN PROGRESS: I'm working through the builds, going through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.
Detailed status can be seen via the tracker bug and via a status file covering both bugs filed and those SRPMs for which bugs have not yet been filed (with reasons)
- 54 bugs filed for src.rpms, where the checker found genuine problems (14%)
- 50 src.rpms not requiring a bug to be filed (13%)
- 78 src.rpms waiting on fix for C++ support (21%)
- 17 src.rpms waiting on better SWIG support (4%)
- 172 src.rpms requiring other followup work (46%)
out of 369 total src.rpms (that link against libpython2.7)
This is the continuation of the "Static Analysis of CPython Extensions" Fedora 16 feature.
Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.
Unfortunately, such wrapper code must manually manage the reference-counts of objects, and mistakes here can lead to /usr/bin/python leaking memory or segfaulting. There's also plenty of code out there that doesn't check for errors.
In Fedora 16, we shipped an initial version of a static analysis tool I've written (gcc-with-cpychecker), implementing some basic checks.
The latest version of the checker can now detect reference-counting bugs, along with paths through code that doesn't properly handle errors from the Python extension API, and I've already used it to patch some significant memory leaks.
Benefit to Fedora
We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugs. These typically manifest as memory leaks. The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).
My hope was to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).
Unfortunately it's not possible to get the signal:noise ratio good enough in time for Fedora 17 for that.
The plan now is to automate running it on all of the C extension modules in Fedora 17, and to analyze the results. Initially bugs would be filed against the tool itself (gcc-python-plugin), and I would then triage them; genuine bugs would be reassigned to the appropriate components, and I'd try to fix the high-value ones, sending fixes upstream. However, this is a large task, and I'm likely to need help from package owners and other Python developers. False positives would thus remain as bugs in the checker itself, and I'd work on fixing them.
Work to be done:
- there's a gcc-4.7 incompatibility that will need a couple of days to fix
- automate running it on all code
- go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.
How To Test
It's not clear that we need this section; the feature covers a distro-wide bug-fixing push.
I *have* written an extensive selftest suite for the checker itself, which is run when it is built.
Non-technical end-users of Fedora should see no difference (other than more a robust operating system).
For examples of the output from the checker, see: http://dmalcolm.livejournal.com/6560.html
This is implemented via a GCC plugin that embeds Python; the checker itself is implemented in Python.
Given that this "Feature" is essentially a bug-sweep (using a new tool), we'll do as much as we can by the deadline. Any that's been done is an improvement to Fedora, but if the amount doesn't look impressive, we can drop this as a feature.
Upstream documentation: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html