Features/StaticAnalysisOfPythonRefcounts

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(Current status)
 
(17 intermediate revisions by 2 users not shown)
Line 19: Line 19:
 
== Current status ==
 
== Current status ==
 
* Targeted release: [[Releases/17 | Fedora 17 ]]  
 
* Targeted release: [[Releases/17 | Fedora 17 ]]  
* Last updated: 2012-01-23
+
* Last updated: 2012-05-29
* Percentage of completion: 30%
+
* Percentage of completion: 100%
  
The code works, and has found real bugs, but still contains bugs itself.  It's only been run on a small subset of the Python code in Fedora.
+
The code works, and has found real bugs, but still contains bugs itself.  It's been run on all of the Python code in Fedora, but doing so has sometimes uncovered bugs in the checker.
  
Major TODO items remaining:
+
Completed items:
* there's a gcc-4.7 incompatibility that will need a couple of days to fix
+
* the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
* automate running it on all code
+
* wrote an automated script for running the tool on a mock build, and generating [http://people.fedoraproject.org/~dmalcolm/gcc-python-plugin/2012-02-10/gstreamer-python-0.10.19-2.fc15/ a triaged report on the issues found]
* go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.
+
* created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472
 +
* only run it on source files that include <Python.h> (implemented in git; not yet in a tarball release)
 +
* automated running it on all code in Fedora using mock, injecting the plugin
  
 +
IN PROGRESS:
 +
I'm working through the builds, going through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.
 +
 +
Detailed status can be seen via [https://bugzilla.redhat.com/showdependencytree.cgi?id=789472 the tracker bug] and via [http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob_plain;f=misc/fedora/bugreports.txt a status file covering both bugs filed and those SRPMs for which bugs have not yet been filed (with reasons)]
 +
 +
Everything in Fedora 17 linked against libpython2.7:
 +
* 74 bugs filed for src.rpms, where the checker found genuine problems (20%)
 +
* 71 src.rpms not requiring a bug to be filed (19%)
 +
* 78 src.rpms waiting on fix for C++ support (21%)
 +
* 18 src.rpms waiting on better SWIG support (4%)
 +
* 13 src.rpms waiting on better Cython support (3%)
 +
* 117 src.rpms requiring other followup work (31%)
 +
out of 370 total src.rpms (that link against libpython2.7)
 +
 +
Within the [[Critical_path_package|critical path]]:
 +
* 12 bugs filed for src.rpms, where the checker found genuine problems (3%)
 +
** {{bz|790973}} NEW        - Bugs found in python-krbV-1.0.90-4.fc15 using gcc-with-cpychecker static analyzer
 +
** {{bz|790979}} NEW        - Memory leaks and crashers found in python bindings in rpm-4.9.1.2-12.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|790983}} NEW        - Segfault under low-memory conditions found in libxml2-2.7.8-6.fc16 using gcc-with-cpychecker static analyzer
 +
** {{bz|791180}} NEW        - Bugs found in anaconda-17.8-1.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|791359}} NEW        - Bug found in deltarpm-3.6-0.7.20110223git.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|794989}} NEW        - Bugs found in libpwquality-1.0.0-2.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|794991}} NEW        - Memory leak in PyErr_SetTDBError() found in libtdb-1.2.9-14.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|800075}} NEW        - Memory leaks and possible crashers found in newt-0.52.14-2.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|800086}} NEW        - Bugs found in pyOpenSSL-0.12-2.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|800146}} ASSIGNED  - Bugs found in python-ethtool-0.7-2.fc16 using gcc-with-cpychecker static analyzer
 +
** {{bz|800200}} NEW        - Bug found in yum-metadata-parser-1.1.4-6.fc17 using gcc-with-cpychecker static analyzer
 +
** {{bz|809945}} NEW        - Bug found in python-markupsafe-0.11-4.fc17 using gcc-with-cpychecker static analyzer
 +
* 4 src.rpms not requiring a bug to be filed (1%)
 +
** dbus-python-0.83.0-9.fc17: Only false positives
 +
** python-pycurl-7.19.0-9.fc15: Only false positives
 +
** pygpgme-0.2-2.fc17: Only in module initialization
 +
** python-nss-0.12-3.fc17: Only in module initialization
 +
* 2 src.rpms waiting on fix for C++ support (0%)
 +
** libimobiledevice-1.1.1-5.fc17: FIXME: C++
 +
** pycryptopp-0.5.29-3.fc17: FIXME: C++
 +
* 12 src.rpms requiring other followup work (3%)
 +
** libsemanage-2.1.6-2.fc17: FIXME: build.log has: error: File /builddir/build/SOURCES/libsemanage-rhat.patch is smaller than 13 bytes
 +
** cryptsetup-1.4.1-2.fc17: FIXME: checker got confused by PyObjectResult, and some tracebacks
 +
** gnome-python2-2.28.1-8.fc17: TODO
 +
** libtalloc-2.0.7-4.fc17: TODO
 +
** gdb-7.4.50.20120120-17.fc17: TODO
 +
** kernel-3.3.0-0.rc3.git5.1.fc17: TODO
 +
** python-2.7.2-18.fc17: TODO: this one will probably require special-casing
 +
** libselinux-2.1.9-7.fc17: TODO: appears to have failed to build
 +
** policycoreutils-2.1.10-21.fc17: TODO: appears to have failed to build
 +
** libdmtx-0.7.2-6.fc17: FIXME: tracebacks:
 +
** pyparted-3.8-3.fc17: FIXME: did not see rpmbuild -bb in build.log
 +
** pyliblzma-0.5.3-6.fc17: FIXME: 4 tracebacks during build
 +
 +
Outside of the [[Critical_path_package|critical path]]:
 +
* 62 bugs filed for src.rpms, where the checker found genuine problems (16%)
 +
* 67 src.rpms not requiring a bug to be filed (18%)
 +
* 76 src.rpms waiting on fix for C++ support (20%)
 +
* 18 src.rpms waiting on better SWIG support (4%)
 +
* 13 src.rpms waiting on better Cython support (3%)
 +
* 105 src.rpms requiring other followup work (28%)
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
  
Line 45: Line 104:
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
Fedora is already a great environment for doing Python development - having a good-quality static analysis tool integrated into Fedora's build system for python extension modules will make Fedora even more compelling for Python developers(Naturally the tool will be Free Software, and thus usable on other platforms; but we'll have it first).
+
We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugsThese typically manifest as memory leaks.  The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).
 
+
The presence of the tool should also make it easier to fix certain awkward bugs, and make it easier to support secondary CPU architectures.
+
  
 
== Scope ==
 
== Scope ==
Line 104: Line 161:
 
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
Fedora now ships with a <code>gcc-with-cpychecker</code> variant of GCC, which adds additional compile-time checks to Python extension modules written in C, detecting various common problems (e.g. reference counting mistakes).  This variant is itself written in Python.
+
(assuming we achieve this:) To prevent memory leaks, all of the Python extension modules in Fedora 17 have been run through a [https://fedorahosted.org/gcc-python-plugin/ static analysis tool] that can [http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html detect reference-counting bugs].
  
 
== Comments and Discussion ==
 
== Comments and Discussion ==
Line 110: Line 167:
  
  
[[Category:FeaturePageIncomplete]]
+
[[Category:FeatureAcceptedF17]]
 
<!-- When your feature page is completed and ready for review -->
 
<!-- When your feature page is completed and ready for review -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->

Latest revision as of 13:46, 29 May 2012


Contents

[edit] Static Analysis of Python Reference Counts

[edit] Summary

I've written a static analysis tool that can detect reference-counting errors made in Python extension modules written in C. We'll run the tool on all such code in Fedora 17 and make an effort to fix as many problems as time allows.

[edit] Owner

  • Email: dmalcolm@redhat.com

[edit] Current status

  • Targeted release: Fedora 17
  • Last updated: 2012-05-29
  • Percentage of completion: 100%

The code works, and has found real bugs, but still contains bugs itself. It's been run on all of the Python code in Fedora, but doing so has sometimes uncovered bugs in the checker.

Completed items:

  • the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
  • wrote an automated script for running the tool on a mock build, and generating a triaged report on the issues found
  • created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472
  • only run it on source files that include <Python.h> (implemented in git; not yet in a tarball release)
  • automated running it on all code in Fedora using mock, injecting the plugin

IN PROGRESS: I'm working through the builds, going through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

Detailed status can be seen via the tracker bug and via a status file covering both bugs filed and those SRPMs for which bugs have not yet been filed (with reasons)

Everything in Fedora 17 linked against libpython2.7:

  • 74 bugs filed for src.rpms, where the checker found genuine problems (20%)
  • 71 src.rpms not requiring a bug to be filed (19%)
  • 78 src.rpms waiting on fix for C++ support (21%)
  • 18 src.rpms waiting on better SWIG support (4%)
  • 13 src.rpms waiting on better Cython support (3%)
  • 117 src.rpms requiring other followup work (31%)

out of 370 total src.rpms (that link against libpython2.7)

Within the critical path:

  • 12 bugs filed for src.rpms, where the checker found genuine problems (3%)
    • RHBZ #790973 NEW - Bugs found in python-krbV-1.0.90-4.fc15 using gcc-with-cpychecker static analyzer
    • RHBZ #790979 NEW - Memory leaks and crashers found in python bindings in rpm-4.9.1.2-12.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #790983 NEW - Segfault under low-memory conditions found in libxml2-2.7.8-6.fc16 using gcc-with-cpychecker static analyzer
    • RHBZ #791180 NEW - Bugs found in anaconda-17.8-1.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #791359 NEW - Bug found in deltarpm-3.6-0.7.20110223git.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #794989 NEW - Bugs found in libpwquality-1.0.0-2.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #794991 NEW - Memory leak in PyErr_SetTDBError() found in libtdb-1.2.9-14.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #800075 NEW - Memory leaks and possible crashers found in newt-0.52.14-2.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #800086 NEW - Bugs found in pyOpenSSL-0.12-2.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #800146 ASSIGNED - Bugs found in python-ethtool-0.7-2.fc16 using gcc-with-cpychecker static analyzer
    • RHBZ #800200 NEW - Bug found in yum-metadata-parser-1.1.4-6.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #809945 NEW - Bug found in python-markupsafe-0.11-4.fc17 using gcc-with-cpychecker static analyzer
  • 4 src.rpms not requiring a bug to be filed (1%)
    • dbus-python-0.83.0-9.fc17: Only false positives
    • python-pycurl-7.19.0-9.fc15: Only false positives
    • pygpgme-0.2-2.fc17: Only in module initialization
    • python-nss-0.12-3.fc17: Only in module initialization
  • 2 src.rpms waiting on fix for C++ support (0%)
    • libimobiledevice-1.1.1-5.fc17: FIXME: C++
    • pycryptopp-0.5.29-3.fc17: FIXME: C++
  • 12 src.rpms requiring other followup work (3%)
    • libsemanage-2.1.6-2.fc17: FIXME: build.log has: error: File /builddir/build/SOURCES/libsemanage-rhat.patch is smaller than 13 bytes
    • cryptsetup-1.4.1-2.fc17: FIXME: checker got confused by PyObjectResult, and some tracebacks
    • gnome-python2-2.28.1-8.fc17: TODO
    • libtalloc-2.0.7-4.fc17: TODO
    • gdb-7.4.50.20120120-17.fc17: TODO
    • kernel-3.3.0-0.rc3.git5.1.fc17: TODO
    • python-2.7.2-18.fc17: TODO: this one will probably require special-casing
    • libselinux-2.1.9-7.fc17: TODO: appears to have failed to build
    • policycoreutils-2.1.10-21.fc17: TODO: appears to have failed to build
    • libdmtx-0.7.2-6.fc17: FIXME: tracebacks:
    • pyparted-3.8-3.fc17: FIXME: did not see rpmbuild -bb in build.log
    • pyliblzma-0.5.3-6.fc17: FIXME: 4 tracebacks during build

Outside of the critical path:

  • 62 bugs filed for src.rpms, where the checker found genuine problems (16%)
  • 67 src.rpms not requiring a bug to be filed (18%)
  • 76 src.rpms waiting on fix for C++ support (20%)
  • 18 src.rpms waiting on better SWIG support (4%)
  • 13 src.rpms waiting on better Cython support (3%)
  • 105 src.rpms requiring other followup work (28%)

[edit] Detailed Description

This is the continuation of the "Static Analysis of CPython Extensions" Fedora 16 feature.

Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.

Unfortunately, such wrapper code must manually manage the reference-counts of objects, and mistakes here can lead to /usr/bin/python leaking memory or segfaulting. There's also plenty of code out there that doesn't check for errors.

In Fedora 16, we shipped an initial version of a static analysis tool I've written (gcc-with-cpychecker), implementing some basic checks.

The latest version of the checker can now detect reference-counting bugs, along with paths through code that doesn't properly handle errors from the Python extension API, and I've already used it to patch some significant memory leaks.

[edit] Benefit to Fedora

We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugs. These typically manifest as memory leaks. The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).

[edit] Scope

My hope was to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).

Unfortunately it's not possible to get the signal:noise ratio good enough in time for Fedora 17 for that.

The plan now is to automate running it on all of the C extension modules in Fedora 17, and to analyze the results. Initially bugs would be filed against the tool itself (gcc-python-plugin), and I would then triage them; genuine bugs would be reassigned to the appropriate components, and I'd try to fix the high-value ones, sending fixes upstream. However, this is a large task, and I'm likely to need help from package owners and other Python developers. False positives would thus remain as bugs in the checker itself, and I'd work on fixing them.

Work to be done:

  • there's a gcc-4.7 incompatibility that will need a couple of days to fix
  • automate running it on all code
  • go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

[edit] How To Test

It's not clear that we need this section; the feature covers a distro-wide bug-fixing push.

I *have* written an extensive selftest suite for the checker itself, which is run when it is built.

[edit] User Experience

Non-technical end-users of Fedora should see no difference (other than more a robust operating system).

For examples of the output from the checker, see: http://dmalcolm.livejournal.com/6560.html

[edit] Dependencies

This is implemented via a GCC plugin that embeds Python; the checker itself is implemented in Python.

[edit] Contingency Plan

Given that this "Feature" is essentially a bug-sweep (using a new tool), we'll do as much as we can by the deadline. Any that's been done is an improvement to Fedora, but if the amount doesn't look impressive, we can drop this as a feature.

[edit] Documentation

Upstream documentation: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html

[edit] Release Notes

(assuming we achieve this:) To prevent memory leaks, all of the Python extension modules in Fedora 17 have been run through a static analysis tool that can detect reference-counting bugs.

[edit] Comments and Discussion