Features/EasierPythonDebugging

= Easier Python Debugging =

Summary
The  debugger has been extended so that it can report detailed information on the internals of the Python 2 and Python 3 runtimes. To enable this feature, the python-debuginfo package must be installed (or python3-debuginfo for Python 3). Then backtraces involving Python will show mixed C and Python-level information on what such processes are doing, without requiring expertise in the use of.

We have also extended  with new commands, aimed at Python developers, allowing them to see what a process is doing at the Python level.

We believe this ability is unique to Fedora, and will be valuable for Python developers seeking additional visibility into their CPython processes.

Owner

 * Name: David Malcolm


 * Email: 

Current status

 * Targeted release: Fedora 13
 * Last updated: 2010-04-06
 * Percentage of completion: 100% - the feature is complete for Python 2, but the necessary build for Python 3 is still in the testing repository (please test, and mark with "karma"!).
 * The new gdb commands work on both i386 and x86_64
 * Automatic prettyprinting of backtraces works fully on i386 but much less well on x86_64 (see below)
 * There are known bugs in the python3-debuginfo build which need fixing before it is usable for Python 3

DONE: It also contains numerous other bug fixes, and added the various new gdb commands detailed below.
 * Code sent upstream as http://bugs.python.org/issue8032 and v5 of the patch was committed upstream as revision 79548 (an older version of the code can be seen at http://fedorapeople.org/gitweb?p=dmalcolm/public_git/libpython.git;a=summary, but this git repository is out-of-date)
 * Proof-of-concept works with both Python 2.6 and Python 3.1
 * v1: handles lists, tuples, dicts, strings, int, long, unicode, stack frames, instances of old-style classes
 * v5 of the patch has numerous bugfixes. It is in the Python 2 build, is upstream and in rawhide.  This limits the amount of data that is scraped to 1KB per line, avoiding a problem seen in complex real-world programs where gdb would wait for 30 seconds, scrape pages and pages of debug data, then print it.


 * Initial version sent to archer mailing list for feedback here.

Unfortunately, getting at frame information is being affected by this bug on x86_64, which greatly diminishes the effectiveness of the hooks on that architecture.

WAITING FOR TESTING: Version 5 of the patch for Python 3 build is still in "testing", the relevant update is:
 * https://admin.fedoraproject.org/updates/python3-3.1.2-4.fc13

Detailed Description
We ship Python wrappers for numerous libraries implemented in C and C++. Bugs (either in the libraries themselves, or in the usage of those libraries) can lead to complicated backtraces from gdb, and it can be hard to figure out what's going on at the python level.

For example, see this complex backtrace (relating to bug 536786) shows a segmentation fault somewhere inside a complicated call stack involving Python and other libraries, and it's not at all clear what's going on.

Walking through the stack frames, going up from the bottom (textually), or down from the top (numerically):
 * frames 26 and below show a pygtk application starting up.
 * An event comes in frame 24/25, and is dispatched into pulsecore (frames 23->18; pstream_packet_callback, pa_context_simple_ack_callback) which:
 * calls a Python callback (down to frame 15),
 * ...which invokes python code down to frame 3.
 * ...where it calls back into native code; whereupon the segfault happens, calling Py_DecRef on some object pointer.

Note that as it stands, all we see from the backtrace is that python code was run: we have no way as-is of telling what that python code was.

In the above example, it happens that there is a bug in the application's Python code, which is sufficiently serious to cause a SIGSEGV error. This example uses the  module, which is designed to expose machine-level details. It's fairly easily to write a one-liner of python code using this module which causes the python process to immediately fail with either a SIGSEGV or SIGABRT.

When using "native" C/C++ libraries, it's sadly common for bugs in the library to leads to SIGSEGV errors that immediately cause the whole python process to terminate. Beyond that, poorly-designed error-handling in such libraries uses  or   at the C level, which immediately terminates the entire process. It's useful to be able to determine what was "really" going on when this happens.

A trickier problem is when a threading assertion fails: many libraries make assumptions about threads and locks, and allow the programmer to register callbacks, but imposes conditions upon the kind of code run in those callbacks. When the threads and callback-registration hooks are wrapped at the python level, these conditions continue to be required at the Python level, but mistakes here often lead to low-level error-handling that's difficult to debug.

For example, the GTK widget library requires that all communication with the X server happen within a GDK lock, to avoid garbling the single "conversation" between the process and the X server. The common way to implement this in a multi-threaded application is to restrict all calls to GTK to a single "primary" thread. See attachment 379251 to rhbug:543278 bug 543278 for an example of where a secondary thread in an application violates this, which leads to a low-level  failure in the main thread: frames 16 to 28 of this backtrace are running Python code, but it's not at all clear from the backtrace _what_ said code is actually doing.

Current state-of-the-art for debugging CPython backtraces
Python already has a gdbinit file with plenty of domain-specific hooks for debugging CPython, and we ship it in our  subpackage. If you copy this to  you can then use "pyframe" and other commands to debug things, and figure out where we are in Python code from gdb. I used it when deciphering the example backtraces referred to above.

Unfortunately:
 * this script isn't very robust - it effectively injects "print to stderr" calls into the process being debugged. If the data in the "inferior" process is corrupt, attempting to print it can lead to a SIGSEGV within that process.
 * you have to go into gdb manually and run these commands by hand, and it's hard to do this correctly; any mistakes when doing this will typically cause a SIGSEGV in the inferior process; see e.g. bug 532552
 * the script is written in the gdb language and is thus hard to work with and extend

Proposal
gdb should provide rich information on what's going on at the Python level automatically. I plan to hook this in using gdb-archer, and make it automatic:
 * Biggest win: automatically display python frame information in PyEval_EvalFrameEx in gdb backtraces, including in ABRT:
 * python source file, line number, and function names
 * values of locals, if available
 * name of function for wrapped C functions

See Alex's work: http://blogs.gnome.org/alexl/2008/11/18/gdb-is-dead-long-live-gdb/ and more recently: http://blogs.gnome.org/alexl/2009/09/21/archer-gdb-macros-for-glib/

I'd want to have the python backtrace work integrated with the glib backtrace work: pygtk regularly shows me backtraces with a mixture of both

Alex's work is in in glib git: http://git.gnome.org/browse/glib/commit/?id=efe9169234e226f594b4254618f35a139338c35f which does a: gdb.backtrace.push_frame_filter (GFrameFilter)

See http://tromey.com/blog/?p=522 for info on this.

This needs a more recent version of gdb than in F-12; I'll need to build a local copy of "archer-tromey-python" branch of gdb to work on this.

Archer upstream: http://sourceware.org/gdb/wiki/ProjectArcher

Benefit to Fedora
Backtraces from gdb (such as those from ABRT) that involve python code will show what's going on at the Python level, as well as at the C level. This will make it much easier for developers to read backtraces when a library wrapped by python encounters a bug (e.g. PyGTK)

For python developers, it should be possible to attach to a running python process using gdb, then run  to get an overview of all C and Python code running in all threads within that process - I believe this ability would be unique to Fedora, and be valuable for Python developers seeking additional visibility into their CPython processes.

Scope
This will require extensions to the  srpm, and analogous changes to the   srpm.

It may well require co-ordination with the  srpm (such as API changes), and with the   changes written by Alex referred to above.

How To Test
Ideas for test cases/coverage: [david@brick ~]$ python -c "import ctypes; ctypes.string_at(0xffffffff)" Segmentation fault (core dumped) [david@brick ~]$ python -c "import ctypes; ctypes.string_at(0x0)" python: Objects/stringobject.c:115: PyString_FromString: Assertion `str != ((void *)0)' failed. Aborted (core dumped)
 * try attaching to a running (multithreaded) python process and ensure that  generates meaningful results
 * ensure it plays well with Alex's GLib/GTK work; debug a multithreaded pygtk app
 * ensure it fails gracefully if python-debuginfo isn't installed
 * ensure that it fails gracefully if the inferior process has corrupted data (e.g. overwrites on the heap)
 * ensure that it fails gracefully if the inferior process has a corrupted stack
 * ensure that it works well under ABRT. It's easy to write one-liner python scripts that abuse the   module in such a way as to cause   to segfault/abort:
 * repeat all of the above for  and

In each case, gdb should give you meaningful information at the Python level, as well as at the C level.

User Experience
Here's an example session from running python within gdb: [david@brick ~]$ gdb --args python GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later  This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /usr/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python2.6.debug...done. done. (gdb) run Starting program: /usr/bin/python [Thread debugging using libthread_db enabled] Python 2.6.2 (r262:71600, Jan 25 2010, 13:22:47) [GCC 4.4.2 20100121 (Red Hat 4.4.2-28)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

At this point, Python is running inside gdb. Let's create a class with a serious bug in it: >>> class Foo: ...    def bar(self): ...        from ctypes import string_at ...        string_at(0xDEADBEEF) # this code will cause Python to segfault ... >>> f = Foo >>> >>> # Let's assign some data of various kinds to the instance: >>> f.someattr = 42 >>> f.someotherattr = {'one':1, 'two':2L, 'three':[, (None,), (None, None)]} >>> >>> # Now let's trigger the segfault >>> f.bar At this point we've generated a segmentation fault inside Python.

Let's see the old behavior of a backtrace, using the "bt" command: Program received signal SIGSEGV, Segmentation fault. __strlen_sse2 at ../sysdeps/i386/i686/multiarch/strlen.S:87 87		pcmpeqb	(%esi), %xmm0 Current language: auto The current source language is "auto; currently asm". (gdb) bt Note that although we can see that there's a problem inside ctypes, it's hard to see what triggered the problem within the Python code.
 * 1) 0 __strlen_sse2  at ../sysdeps/i386/i686/multiarch/strlen.S:87
 * 2) 1 0x07113d30 in PyString_FromString (str=0xdeadbeef ) at Objects/stringobject.c:116
 * 3) 2 0x00167e18 in string_at (ptr=0xdeadbeef , size=-1) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:5348
 * 4) 3 0x0018247f in ffi_call_SYSV  at src/x86/sysv.S:61
 * 5) 4 0x001822b0 in ffi_call (cif=, fn= , rvalue= , avalue= ) at src/x86/ffi.c:213
 * 6) 5 0x00171315 in _call_function_pointer (pProc=0x167de0 , argtuple=0xb7f3d02c, flags=4357, argtypes=0xb7f45a4c, restype=0x80f3dc4, checker=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:815
 * 7) 6 _CallProc (pProc=0x167de0 , argtuple=0xb7f3d02c, flags=4357, argtypes=0xb7f45a4c, restype=0x80f3dc4, checker=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:1162
 * 8) 7 0x0016a6f2 in CFuncPtr_call (self=0xb7f9d5dc, inargs=0xb7f3d02c, kwds=0x0) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:3857
 * 9) 8 0x070c478c in PyObject_Call (func=0xb7f9d5dc, arg=0xb7f3d02c, kw=0x0) at Objects/abstract.c:2492
 * 10) 9 0x0716069c in do_call (f=0x80f37bc, throwflag=0) at Python/ceval.c:3917
 * 11) 10 call_function (f=0x80f37bc, throwflag=0) at Python/ceval.c:3729
 * 12) 11 PyEval_EvalFrameEx (f=0x80f37bc, throwflag=0) at Python/ceval.c:2389
 * 13) 12 0x07162642 in PyEval_EvalCodeEx (co=0xb7f3bda0, globals=0xb7f3768c, locals=0x0, args=0x80ec788, argcount=1, kws=0x80ec78c, kwcount=0, defs=0xb7f45d78, defcount=1, closure=0x0) at Python/ceval.c:2968
 * 14) 13 0x07160983 in fast_function (f=0x80ec644, throwflag=0) at Python/ceval.c:3802
 * 15) 14 call_function (f=0x80ec644, throwflag=0) at Python/ceval.c:3727
 * 16) 15 PyEval_EvalFrameEx (f=0x80ec644, throwflag=0) at Python/ceval.c:2389
 * 17) 16 0x07161b79 in fast_function (f=0x80eb1cc, throwflag=0) at Python/ceval.c:3792
 * 18) 17 call_function (f=0x80eb1cc, throwflag=0) at Python/ceval.c:3727
 * 19) 18 PyEval_EvalFrameEx (f=0x80eb1cc, throwflag=0) at Python/ceval.c:2389
 * 20) 19 0x07162642 in PyEval_EvalCodeEx (co=0xb7f2e578, globals=0xb7fc70b4, locals=0xb7fc70b4, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968
 * 21) 20 0x071627a3 in PyEval_EvalCode (co=0xb7f2e578, globals=0xb7fc70b4, locals=0xb7fc70b4) at Python/ceval.c:522
 * 22) 21 0x0717d94b in run_mod (mod=, filename= , globals=0xb7fc70b4, locals=0xb7fc70b4, flags=0xbffff2fc, arena=0x80e8628) at Python/pythonrun.c:1335
 * 23) 22 0x0717f4a6 in PyRun_InteractiveOneFlags (fp=0x5b5420, filename=0x71c3e7d " ", flags=0xbffff2fc) at Python/pythonrun.c:840
 * 24) 23 0x0717f6ab in PyRun_InteractiveLoopFlags (fp=0x5b5420, filename=0x71c3e7d " ", flags= ) at Python/pythonrun.c:760
 * 25) 24 0x0717f7eb in PyRun_AnyFileExFlags (fp=0x5b5420, filename=, closeit=0, flags=0xbffff2fc) at Python/pythonrun.c:729
 * 26) 25 0x0718c212 in Py_Main (argc=1, argv=0xbffff3f4) at Modules/main.c:599
 * 27) 26 0x080485c7 in main (argc=1, argv=0xbffff3f4) at Modules/python.c:23

Fedora 13's python-debuginfo and python3-debuginfo packages now provide the gdb visualization hooks, installed in a location where gdb will automatically load them. (When I recorded the session, I had to import them by hand in gdb like this: (gdb) python >import sys >sys.path.append('/home/david/coding/python-gdb') >import libpython >reload(libpython) > >end

With the gdb visualizations imported, let's rerun the "bt" command. Notice how this time we get rich information on the Python code: (gdb) bt   <_ctypes.SimpleType at remote 0x80f3dc4>, checker= ) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:815 Note that before in frame 5, gdb merely reported  .  It is now able to tell us that we have a (long, int) 2-tuple:    (this is actually   but it has no way to know what base you want the number in).
 * 1) 0 __strlen_sse2  at ../sysdeps/i386/i686/multiarch/strlen.S:87
 * 2) 1 0x07113d30 in PyString_FromString (str=0xdeadbeef ) at Objects/stringobject.c:116
 * 3) 2 0x00167e18 in string_at (ptr=0xdeadbeef , size=-1) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:5348
 * 4) 3 0x0018247f in ffi_call_SYSV  at src/x86/sysv.S:61
 * 5) 4 0x001822b0 in ffi_call (cif=, fn= , rvalue= , avalue= ) at src/x86/ffi.c:213
 * 6) 5 0x00171315 in _call_function_pointer (pProc=0x167de0 , argtuple=(3735928559L, -1), flags=4357, argtypes=(, ), restype=

Similarly, gdb is now telling us the types of the various objects. For example, in the baseline backtrace in frame 5 gdb merely reported, but with this visualizer it is now able to tell us we have.

<_ctypes.SimpleType at remote 0x80f3dc4>, checker= ) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/callproc.c:1162 In frame 9 and below, we have a Python frame "f". In the old backtrace it was merely reported as , but gdb is now able to tell us the Python file, line number, function, and locals:
 * 1) 6 _CallProc (pProc=0x167de0 , argtuple=(3735928559L, -1), flags=4357, argtypes=(, ), restype=
 * 1) 7 0x0016a6f2 in CFuncPtr_call (self=0xb7f9d5dc, inargs=(3735928559L, -1), kwds= ) at /usr/src/debug/Python-2.6.2/Modules/_ctypes/_ctypes.c:3857
 * 2) 8 0x070c478c in PyObject_Call (func=, arg=(3735928559L, -1), kw= ) at Objects/abstract.c:2492
 * 3) 9 0x0716069c in do_call (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:3917

We can use this to trace the Python-level stacktrace for this thread within gdb.

{'Union': <_ctypes.UnionType at remote 0x17c120>, 'c_wchar': <_ctypes.SimpleType at remote 0x80fb32c>, 'c_bool': <_ctypes.SimpleType at remote 0x80fab54>, 'c_double': <_ctypes.SimpleType at remote 0x80f7a0c>, 'CFUNCTYPE': <function at remote 0xb7f3264c>, '__path__': ['/usr/lib/python2.6/ctypes'], 'byref': <builtin_function_or_method at remote 0xb7f40eec>, 'pointer': <builtin_function_or_method at remote 0xb7f40d6c>, 'alignment': <builtin_function_or_method at remote 0xb7f40eac>, '_memmove_addr': 4962832, 'c_longlong': <_ctypes.SimpleType at remote 0x80f8544>, 'c_short': <_ctypes.SimpleType at remote 0x80f407c>, 'get_errno': <builtin_function_or_method at remote 0xb7f39f4c>, '__file__': '/usr/lib/python2.6/ctypes/__init__.pyc', '_calcsize': <builtin_function_or_method at remote 0xb7f96bec>, 'c_ulong': <_ctypes.SimpleType at remote 0x80f5974>, 'c_int': <_ctypes.SimpleType at remote 0x80f5124>, 'c_int32': <_ctypes.SimpleType at remote 0x80f5124>, 'memmove': <CFunctionType at remote 0xb7f9d4a4>, '_sys': <module at remote 0xb7fa308c>, '_cast': <CFunctionType at remote 0xb7f9d574>, 'addressof': <builtin_function_or_method at remote 0xb7f40f0c>, 'ArgumentError':, 'c_buffer': <function at remote 0xb7f32614>, 'c_longdouble': <_ctypes.SimpleType at remote 0x80f821c>, 'cdll': <LibraryLoader at remote 0xb7f459ac>, 'memset': <CFunctionType at remote 0xb7f9d50c>, 'string_at': <function at remote 0xb7f32a04>, 'sizeof': <builtin_function_or_method at remote 0xb7f40ecc>, '_FUNCFLAG_PYTHONAPI': 4, 'create_string_buffer': <function at remote 0xb7f325dc>, 'set_errno': <builtin_function_or_method at remote 0xb7f40d2c>, '_pointer_type_cache': {<_ctypes.SimpleType at remote 0x80f9ec4>: <_ctypes.PointerType at remote 0x80fb864>, <_ctypes.SimpleType at remote 0x80fb32c>: <_ctypes.PointerType at remote 0x80fb504>, <NoneType at remote 0x72061e0>: <_ctypes.SimpleType at remote 0x80fa6a4>}, '_Pointer': <_ctypes.PointerType at remote 0x17bd00>, 'create_unicode_buffer': <function at remote 0xb7f326bc>, 'c_long': <_ctypes.SimpleType at remote 0x80f5124>, 'c_char_p': <_ctypes.SimpleType at remote 0x80fa37c>, '__builtins__': {'bytearray': , 'IndexError': , 'all': <builtin_function_or_method at remote 0xb7fafccc>, 'help': <_Helper at remote 0xb7f814ec>, 'vars': <builtin_function_or_method at remote 0xb7fb280c>, 'SyntaxError': , 'unicode': , 'sorted': <builtin_function_or_method at remote 0xb7fb274c>, 'isinstance': <builtin_function_or_method at remote 0xb7fb22cc>, 'copyright': <_Printer at remote 0xb7f81d6c>, 'NameError': , 'BytesWarning': , 'dict': , 'input': <builtin_function_or_method at remote 0xb7fb224c>, 'oct': <builtin_function_or_method at remote 0xb7fb246c>, 'bin': <builtin_function_or_method at remote 0xb7fafd8c>, 'SystemExit': , 'StandardError': , 'format': <builtin_function_or_method at remote 0xb7fb20ac>, 'repr': <builtin_function_or_method at remote 0xb7fb268c>, 'UnicodeDecodeError': , 'False': , 'RuntimeWarning': , 'bytes': , 'iter': <builtin_function_or_method at remote 0xb7fb230c>, 'reload': <builtin_function_or_method at remote 0xb7fb264c>, 'Warning': , 'round': <builtin_function_or_method at remote 0xb7fb26cc>, 'dir': <builtin_function_or_method at remote 0xb7faff4c>, 'cmp': <builtin_function_or_method at remote 0xb7fafe4c>, 'set': , 'list': , 'reduce': <builtin_function_or_method at remote 0xb7fb260c>, 'intern': <builtin_function_or_method at remote 0xb7fb228c>, 'issubclass': <builtin_function_or_method at remote 0xb7fb22ec>, 'apply': <builtin_function_or_method at remote 0xb7fafd4c>, 'EOFError': , 'locals': <builtin_function_or_method at remote 0xb7fb238c>, 'BufferError': , 'slice': , 'FloatingPointError': , 'sum': <builtin_function_or_method at remote 0xb7fb278c>, 'buffer': , 'getattr': <builtin_function_or_method at remote 0xb7fb20cc>, 'abs': <builtin_function_or_method at remote 0xb7fafc8c>, 'exit': <Quitter at remote 0xb7fd1d2c>, 'print': <builtin_function_or_method at remote 0xb7fb256c>, 'IndentationError': , 'True': , 'FutureWarning': , 'ImportWarning': , 'None': <NoneType at remote 0x72061e0>, 'hash': <builtin_function_or_method at remote 0xb7fb218c>, 'len': <builtin_function_or_method at remote 0xb7fb234c>, 'credits': <_Printer at remote 0xb7f8156c>, 'frozenset': , '__name__': '__builtin__', 'ord': <builtin_function_or_method at remote 0xb7fb24ec>, 'super': , 'TypeError': , 'license': <_Printer at remote 0xb7f8170c>, 'KeyboardInterrupt': , 'UserWarning': , 'filter': <builtin_function_or_method at remote 0xb7fb206c>, 'range': <builtin_function_or_method at remote 0xb7fb25ac>, 'staticmethod': , 'SystemError': , 'BaseException': , 'pow': <builtin_function_or_method at remote 0xb7fb252c>, 'RuntimeError': , 'float': , 'GeneratorExit': , 'StopIteration': , 'globals': <builtin_function_or_method at remote 0xb7fb210c>, 'divmod': <builtin_function_or_method at remote 0xb7faff8c>, 'enumerate': , 'Ellipsis': , 'LookupError': , 'open': <builtin_function_or_method at remote 0xb7fb24ac>, 'quit': <Quitter at remote 0xb7fd120c>, 'basestring': , 'UnicodeError': , 'zip': <builtin_function_or_method at remote 0xb7fb284c>, 'hex': <builtin_function_or_method at remote 0xb7fb21cc>, 'long': , 'next': <builtin_function_or_method at remote 0xb7fb244c>, 'int': , 'chr': <builtin_function_or_method at remote 0xb7fafe0c>, '__import__': <builtin_function_or_method at remote 0xb7fafc6c>, 'type': , '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", 'Exception': , 'tuple': , 'UnicodeTranslateError': , 'reversed': , 'UnicodeEncodeError': , 'IOError': , 'hasattr': <builtin_function_or_method at remote 0xb7fb214c>, 'delattr': <builtin_function_or_method at remote 0xb7faff0c>, 'setattr': <builtin_function_or_method at remote 0xb7fb270c>, 'raw_input': <builtin_function_or_method at remote 0xb7fb25ec>, 'PendingDeprecationWarning': , 'compile': <builtin_function_or_method at remote 0xb7fafecc>, 'ArithmeticError': , 'str': , 'property': , 'MemoryError': , 'ImportError': , 'xrange': , 'KeyError': , 'coerce': <builtin_function_or_method at remote 0xb7fafe8c>, 'SyntaxWarning': , 'file': , 'EnvironmentError': , 'unichr': <builtin_function_or_method at remote 0xb7fb27cc>, 'id': <builtin_function_or_method at remote 0xb7fb220c>, 'OSError': , 'DeprecationWarning': , 'min': <builtin_function_or_method at remote 0xb7fb242c>, 'UnicodeWarning': , 'execfile': <builtin_function_or_method at remote 0xb7fb202c>, '__package__': <NoneType at remote 0x72061e0>, 'complex': , 'bool': , 'ValueError': , 'NotImplemented': <NotImplementedType at remote 0x72061e8>, 'map': <builtin_function_or_method at remote 0xb7fb23cc>, 'any': <builtin_function_or_method at remote 0xb7fafd0c>, 'max': <builtin_function_or_method at remote 0xb7fb240c>, 'object': , 'TabError': , 'callable': <builtin_function_or_method at remote 0xb7fafdcc>, 'ZeroDivisionError': , 'eval': <builtin_function_or_method at remote 0xb7faffcc>, '__debug__': , 'ReferenceError': , 'AssertionError': , 'classmethod': , 'UnboundLocalError': , 'NotImplementedError': , 'AttributeError': , 'OverflowError': }, '_FUNCFLAG_USE_ERRNO': 8, '_memset_addr': 4962944, '_dlopen': <builtin_function_or_method at remote 0xb7f40e0c>, '__name__': 'ctypes', 'RTLD_LOCAL': 0, 'c_int16': <_ctypes.SimpleType at remote 0x80f407c>, '_SimpleCData': <_ctypes.SimpleType at remote 0x17bde0>, 'wstring_at': <function at remote 0xb7f32a3c>, 'c_void_p': <_ctypes.SimpleType at remote 0x80fa6a4>, 'set_conversion_mode': <builtin_function_or_method at remote 0xb7f40dec>, 'PyDLL': , 'DEFAULT_MODE': 0, 'LittleEndianStructure': <_ctypes.StructType at remote 0x17c040>, 'c_uint64': <_ctypes.SimpleType at remote 0x80f8d54>, 'c_ulonglong': <_ctypes.SimpleType at remote 0x80f8d54>, '_FUNCFLAG_USE_LASTERROR': 16, '_cast_addr': 1490912, 'ARRAY': <function at remote 0xb7f3279c>, 'c_ushort': <_ctypes.SimpleType at remote 0x80f48ac>, '__doc__': 'create and manipulate C data types in Python', '_check_size': <function at remote 0xb7f32684>, 'CDLL': , '_wstring_at': <CFunctionType at remote 0xb7f9d644>, 'c_ubyte': <_ctypes.SimpleType at remote 0x80f9564>, 'RTLD_GLOBAL': 256, 'c_char': <_ctypes.SimpleType at remote 0x80f9ec4>, 'c_uint32': <_ctypes.SimpleType at remote 0x80f5974>, 'c_float': <_ctypes.SimpleType at remote 0x80f71fc>, 'SetPointerType': <function at remote 0xb7f32764>, 'resize': <builtin_function_or_method at remote 0xb7f40dcc>, '_c_functype_cache': {(<_ctypes.SimpleType at remote 0x80f5124>, , 1): <_ctypes.CFuncPtrType at remote 0x8100b14>, (<_ctypes.SimpleType at remote 0x80fa6a4>, (<_ctypes.SimpleType at remote 0x80fa6a4>, <_ctypes.SimpleType at remote 0x80f5124>, <_ctypes.SimpleType at remote 0x80f5974>), 1): <_ctypes.CFuncPtrType at remote 0x80fd84c>}, '---Type to continue, or q to quit--- _os': <module at remote 0xb7fa314c>, '_wstring_at_addr': 1494192, 'cast': <function at remote 0xb7f329cc>, 'c_int8': <_ctypes.SimpleType at remote 0x80f9a14>, 'c_byte': <_ctypes.SimpleType at remote 0x80f9a14>, 'c_int64': <_ctypes.SimpleType at remote 0x80f8544>, 'c_voidp': <_ctypes.SimpleType at remote 0x80fa6a4>, '_string_at_addr': 1474016, '_FUNCFLAG_CDECL': 1, 'pythonapi': <PyDLL at remote 0xb7f45a0c>, 'PYFUNCTYPE': <function at remote 0xb7f327d4>, '_CFuncPtr': <_ctypes.CFuncPtrType at remote 0x17bb40>, '_endian': <module at remote 0xb7fa3944>, '__package__': 'ctypes', 'c_uint16': <_ctypes.SimpleType at remote 0x80f48ac>, 'BigEndianStructure': <_swapped_meta at remote 0x81006ec>, 'pydll': <LibraryLoader at remote 0xb7f459ec>, '__version__': '1.1.0', 'Structure': <_ctypes.StructType at remote 0x17c040>, 'c_uint': <_ctypes.SimpleType at remote 0x80f5974>, 'py_object': <_ctypes.SimpleType at remote 0x80f3dc4>, 'c_wchar_p': <_ctypes.SimpleType at remote 0x80fae7c>, '_string_at': <CFunctionType at remote 0xb7f9d5dc>, 'c_size_t': <_ctypes.SimpleType at remote 0x80f5974>, 'c_uint8': <_ctypes.SimpleType at remote 0x80f9564>, 'LibraryLoader':, 'Array': <_ctypes.ArrayType at remote 0x17bc20>, 'POINTER': <builtin_function_or_method at remote 0xb7f40d4c>}, locals= , args=0x80ec788, argcount=1, kws=0x80ec78c, kwcount=0, defs=0xb7f45d78, defcount=1, closure= ) at Python/ceval.c:2968 The above is probably overkill: gdb is now able to tell us the value of "globals", giving us lots of insight into the namespace.   File, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:3802 File, line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:3727   File , line 2, in bar (self=<Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, string_at=<function at remote 0xb7f32a04>), throwflag=0) at Python/ceval.c:2389 In the above frames, notice how gdb is now able to tell us that this instance of an old-style class is of type "Foo" and the current values of its attributes (I deliberately picked a mixture above in order to show support for dictionaries, lists, tuples, ints, longs etc). {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals= {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure= ) at Python/ceval.c:2968   {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals=    {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}) at Python/ceval.c:522 {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, locals= {'f': <Foo({'someattr': 42, 'someotherattr': {'three': [, (<NoneType at remote 0x72061e0>,), (<NoneType at remote 0x72061e0>, <NoneType at remote 0x72061e0>)], 'two': 2L}}) at remote 0xb7f3946c>, '__builtins__': <module at remote 0xb7fa3074>, '__package__': <NoneType at remote 0x72061e0>, '__name__': '__main__', 'Foo': <classobj at remote 0xb7f3817c>, '__doc__': <NoneType at remote 0x72061e0>}, flags=0xbffff2fc, arena=0x80e8628) at Python/pythonrun.c:1335
 * 1) 10 call_function (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:3729
 * 2) 11 PyEval_EvalFrameEx (f=File /usr/lib/python2.6/ctypes/__init__.py, line 492, in string_at (ptr=3735928559L, size=-1), throwflag=0) at Python/ceval.c:2389
 * 3) 12 0x07162642 in PyEval_EvalCodeEx (co=0xb7f3bda0, globals=
 * 1) 13 0x07160983 in fast_function (f=
 * 1) 14 call_function (f=
 * 1) 15 PyEval_EvalFrameEx (f=
 * 1) 16 0x07161b79 in fast_function (f=File, line 1, in , throwflag=0) at Python/ceval.c:3792
 * 2) 17 call_function (f=File, line 1, in , throwflag=0) at Python/ceval.c:3727
 * 3) 18 PyEval_EvalFrameEx (f=File, line 1, in , throwflag=0) at Python/ceval.c:2389
 * 4) 19 0x07162642 in PyEval_EvalCodeEx (co=0xb7f2e578, globals=
 * 1) 20 0x071627a3 in PyEval_EvalCode (co=0xb7f2e578, globals=
 * 1) 21 0x0717d94b in run_mod (mod=, filename= , globals=
 * 1) 22 0x0717f4a6 in PyRun_InteractiveOneFlags (fp=0x5b5420, filename=0x71c3e7d " ", flags=0xbffff2fc) at Python/pythonrun.c:840
 * 2) 23 0x0717f6ab in PyRun_InteractiveLoopFlags (fp=0x5b5420, filename=0x71c3e7d " ", flags= ) at Python/pythonrun.c:760
 * 3) 24 0x0717f7eb in PyRun_AnyFileExFlags (fp=0x5b5420, filename=, closeit=0, flags=0xbffff2fc) at Python/pythonrun.c:729
 * 4) 25 0x0718c212 in Py_Main (argc=1, argv=0xbffff3f4) at Modules/main.c:599
 * 5) 26 0x080485c7 in main (argc=1, argv=0xbffff3f4) at Modules/python.c:23

We are installing pretty-printing hooks into gdb for the types (PyObject*) and (PyFrameObject*).

If you need to override this behavior to see the underlying data, simply dereference the pointer as normal (we're pretty-printing the pointer types, not the types themselves).

For example, the pretty-printer is invoked for this value: (gdb) p (PyObject*)0x8405df4 $3 =

But you can see the underlying value thus: (gdb) p *(PyObject*)0x8405df4 $4 = {ob_refcnt = 23, ob_type = 0x7203420}

Similarly, this PyObject* value is pretty-printed: (gdb) p ((PyFunctionObject*)0x8405df4)->func_code $8 =, and   packages.

Contingency Plan
The contingency plan would be to remove the additional .py files, deactivating the feature.

Documentation
See the "Detailed Description" section above; this feature page contains much information.

Automatic printing of Python representation
From Fedora 13 onwards, gdb will automatically print Python representations of Python objects in backtraces and when printing data.

For example: (gdb) frame 9 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': , 'gsd.hints': 0, 'timer.active_time':, 'timer.total_time': }], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated), throwflag=0) at Python/ceval.c:3850 3850			retval = PyEval_EvalFrameEx(f,0);
 * 1) 9 0x06e239eb in fast_function (f=

New gdb commands
From python-2.6.4-25.fc13 and python3-3.1.2-4.fc13 onwards, Fedora's python debug packages add new commands to gdb.

py-list
List the Python source code (if any) for the current frame in the selected thread. The current line is marked with a ">" (gdb) py-list 901       if options.profile: 902           options.profile = False 903           profile_me 904           return 905   >906        u = UI 907       if not u.quit: 908           try: 909               gtk.main 910           except KeyboardInterrupt: 911               # properly quit on a keyboard interrupt...

Use  to list at a different line number within the python source, and   to list a specific range of lines within the python source.

py-up and py-down
The "py-up" and "py-down" commands are analagous to gdb's regular "up" and "down" commands, but try to move at the level of Python frames, rather than C frames.

gdb is not always able to read the relevant frame information, depending on the optimization level.

They emit the frame number (at the C level) within the thread.

For example: (gdb) py-up u = UI (gdb) py-up main.start_game (gdb) py-up Unable to find an older python frame so we're at the top of the python stack.
 * 1) 37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game
 * 1) 40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=<module at remote 0xb771b7f4>)

Going back down: (gdb) py-down u = UI (gdb) py-down (gdb) py-down (gdb) py-down (gdb) py-down swallower.run_dialog(self.dialog) (gdb) py-down gtk.main (gdb) py-down (gdb) py-down Unable to find a newer python frame and we're at the bottom of the python stack.
 * 1) 37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game
 * 1) 34 (unable to read python frame information)
 * 1) 23 (unable to read python frame information)
 * 1) 19 (unable to read python frame information)
 * 1) 14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__':, 'gsd.hints': 0, 'timer.active_time': , 'timer.total_time': }], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated)
 * 1) 11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>)
 * 1) 8 (unable to read python frame information)

py-bt
The  command attempts to display a Python-level backtrace of the current thread.

For example: (gdb) py-bt gtk.main swallower.run_dialog(self.dialog) u = UI main.start_game
 * 1) 8 (unable to read python frame information)
 * 2) 11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>)
 * 1) 14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__':, 'gsd.hints': 0, 'timer.active_time': , 'timer.total_time': }], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated)
 * 1) 19 (unable to read python frame information)
 * 2) 23 (unable to read python frame information)
 * 3) 34 (unable to read python frame information)
 * 4) 37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game
 * 1) 40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=<module at remote 0xb771b7f4>)

The frame numbers correspond to those displayed by gdb's standard  command.

py-print
The  command looks up a Python name and tries to print it. It looks in locals within the current thread, then globals, then finally builtins:

(gdb) py-print self local 'self' = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4> (gdb) py-print __name__ global '__name__' = 'gnome_sudoku.dialog_swallower' (gdb) py-print len builtin 'len' = <built-in function len> (gdb) py-print scarlet_pimpernel 'scarlet_pimpernel' not found

py-locals
The  command looks up all Python locals within the current Python frame in the selected thread, and prints their representations: (gdb) py-locals self = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4> d = <gtk.Dialog at remote 0x98faaa4>

Other possibilities
You can of course use other gdb commands.

For example, the  command takes you directly to a particular frame within the selected thread. We can use it to go a specific frame shown by  like this: (gdb) py-bt (output snipped) main (gdb) frame 68 2665				x = call_function(&sp, oparg); (gdb) py-list 1543       # Run the tests in a context manager that temporary changes the CWD to a 1544        # temporary and writable directory. If it's not possible to create or 1545       # change the CWD, the original CWD will be used. The original CWD is 1546       # available from test_support.SAVEDCWD. 1547       with test_support.temp_cwd(TESTCWD, quiet=True): >1548           main
 * 1) 68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in
 * 1) 68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in, throwflag=0) at Python/ceval.c:2665

The  command will give you a list of the threads within the process, and you can use the   command to select a different one.

For example (gdb) info threads 105 Thread 0x7fffefa18710 (LWP 10260) sem_wait  at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 104 Thread 0x7fffdf5fe710 (LWP 10259) sem_wait  at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
 * 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select  at ../sysdeps/unix/syscall-template.S:82

You can use  or (  for short) to run a command on all threads. You can use this with  to see what every thread is doing at the Python level: (gdb) t a a py-bt

Thread 105 (Thread 0x7fffefa18710 (LWP 10260)): self.__block.acquire self._acquire_restore(saved_state) cond.wait f
 * 1) 5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528)
 * 1) 8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858a90>, saved_state=(1, 140737213728528))
 * 1) 12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f
 * 1) 16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528)

Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)): self.__block.acquire self._acquire_restore(saved_state) cond.wait f
 * 1) 5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272)
 * 1) 8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858860>, saved_state=(1, 140736940992272))
 * 1) 12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f
 * 1) 16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272)

Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)): time.sleep(0.01) _wait
 * 1) 5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait
 * 1) 8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=<ConditionTests(_testMethodName='test_notify', _resultForDoCleanups=<TestResult(_original_stdout=<cStringIO.StringO at remote 0xc191e0>, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=<file at remote 0x7ffff7fc6340>, _stdout_buffer=<cStringIO.StringO at remote 0xc9c7f8>, _stderr_buffer=<cStringIO.StringO at remote 0xc9c790>, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={ : <instancemethod at remote 0xd750e0>, : <instancemethod at remote 0xd75160>, : <instancemethod at remote 0xd75060>, : <instancemethod at remote 0xd751e0>, <type at remote 0x7f19e0...(truncated)

(etc, output snipped)

Release Notes

 * Python: the  debugger has been extended so that it can report detailed information on the internals of the Python 2 and Python 3 runtimes.  Backtraces involving Python will now by default show mixed C and Python-level information on what such processes are doing, without requiring expertise in the use of

Comments and Discussion

 * See Talk:Features/EasierPythonDebugging