Features/MemoryDebuggingTools

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(Current status)
m (Current status: add link to bug 634660)
 
(37 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<!-- All fields on this form are required to be accepted by FESCo.
 
We also request that you maintain the same order of sections so that all of the feature pages are uniform.  -->
 
 
<!-- The actual name of your feature page should look something like: Features/YourFeatureName.  This keeps all features in the same namespace -->
 
 
 
= Memory Debugging Tools =
 
= Memory Debugging Tools =
  
Line 12: Line 7:
  
 
== Owner ==
 
== Owner ==
<!--This should link to your home wiki page so we know who you are-->
 
 
* Name: [[User:dmalcolm|Dave Malcolm]]
 
* Name: [[User:dmalcolm|Dave Malcolm]]
 
<!-- Include you email address that you can be reached should people want to contact you about helping with your feature, status is requested, or  technical issues need to be resolved-->
 
 
* Email: <dmalcolm@redhat.com>
 
* Email: <dmalcolm@redhat.com>
  
 
== Current status ==
 
== Current status ==
* Targeted release: [[Releases/{{FedoraVersion||next}} | {{FedoraVersion|long|next}} ]]  
+
* Targeted release: [[Releases/14 | Fedora 14]]  
* Last updated: 2010-07-08
+
* Last updated: 2010-09-16
* Percentage of completion: 20%
+
* Percentage of completion: 100%
  
Preparing upstream project for initial launch: https://fedorahosted.org/gdb-heap/
+
TODO:
 +
This is "feature-complete", but some issues remain:
 +
* I need to blog about this and write better docs
 +
* Fix [https://bugzilla.redhat.com/buglist.cgi?component=gdb-heap&product=Fedora the bugs]
 +
* Testing
 +
 
 +
You will need [https://admin.fedoraproject.org/updates/gdb-7.2-7.fc14 gdb-7.2-7.fc14] or later when attaching to a process (rather than starting the process within gdb), due to [https://bugzilla.redhat.com/show_bug.cgi?id=634660 bug 634660]
 +
 
 +
DONE:
 +
* Preparing upstream project for initial launch: https://fedorahosted.org/gdb-heap/
 +
* I've disabled C++ support for now, as the current implementation slows down other operations.
  
 
Initial version of code uploaded
 
Initial version of code uploaded
 +
 +
Packaged and [https://bugzilla.redhat.com/show_bug.cgi?id=618451 reviewed]; available as [https://admin.fedoraproject.org/updates/gdb-heap-0.4-1.fc14 an update to F14].
  
 
Upstream tickets:
 
Upstream tickets:
 
* [https://fedorahosted.org/gdb-heap/milestone/Initial%20Release Initial Release]
 
* [https://fedorahosted.org/gdb-heap/milestone/Initial%20Release Initial Release]
 
* [https://fedorahosted.org/gdb-heap/milestone/F14%20Goals Fedora 14 Goals]
 
* [https://fedorahosted.org/gdb-heap/milestone/F14%20Goals Fedora 14 Goals]
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
 
  
 
== Detailed Description ==
 
== Detailed Description ==
 
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 +
The new "gdb-heap" package adds a new "heap" command to /usr/bin/gdb. 
 +
 +
The command allows you to get a breakdown of how that process is using dynamic memory.
 +
 +
It allows for ''unplanned'' memory usage debugging: if a process unexpectedly starts using large amounts of memory you can attach to it with gdb, and use the heap command to figure out where the memory is going. You should also be able to use it on core dumps.
 +
 +
We believe this approach is entirely new, and is unique to Fedora 14.
  
 
== Benefit to Fedora ==
 
== Benefit to Fedora ==
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
 
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
 +
This feature could be of great use to developers and system administrators: it provides a new way of analyzing how a process uses memory, without requiring advance planning.
 +
 +
It is unique to Fedora (it makes [https://fedorahosted.org/gdb-heap/wiki#Compatibilitywithgdbversions heavy use of the gdb/python integration we have in Fedora]), and was developed by a Fedora contributor (who is a Red Hat engineer).
  
 
== Scope ==
 
== Scope ==
 
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 +
Code is isolated, as an extension to gdb, written in Python.
 +
 +
* I'm tracking development of the code in the upstream tracker here:
 +
** [https://fedorahosted.org/gdb-heap/milestone/Initial%20Release Initial Release]
 +
** [https://fedorahosted.org/gdb-heap/milestone/F14%20Goals Fedora 14 Goals]
 +
* Package the code in RPM form, add it to Fedora
 +
* Ensure that it's available without the user needing excessive configuration; ideally, if the rpm is installed, then you get the command automagically
 +
* Add it to comps so that it's suggested for installed by default if gdb is installed.
 +
* Testing
  
 
== How To Test ==
 
== How To Test ==
Line 56: Line 77:
 
3. What are the expected results of those actions?
 
3. What are the expected results of those actions?
 
-->
 
-->
 +
No special hardware is needed.
 +
 +
You will need to install the gdb-heap package (not yet packaged)
 +
 +
=== Exploratory testing ===
 +
* Pick a process on your system (either as root, or one of your own processes)
 +
* Use "gdb attach PID" to connect to it
 +
* Use <code>python import heap</code> to register the "heap" command
 +
* Use the "heap" command and its various subcommands (as described on [https://fedorahosted.org/gdb-heap/wiki the upstream website])
 +
* Ensure that all results look correct, and that there are no Python tracebacks within gdb.
 +
 +
Ideally the amount of "uncategorized" data should not be a substantial proportion of the overall size of the dynamically-allocated memory; if it is, then that may be a bug.
 +
 +
Ideally the command should not take too long to run.  The more blocks of memory that are "live" within a process, the longer it will take to analyze the usage.  Crude timings suggest it can analyze about 5000 allocations per second, so if you have a process with 300,000 allocations, it could take a minute to analyze them.
  
 
== User Experience ==
 
== User Experience ==
 
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
 
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
 +
=== Basic Operation ===
 +
Having attached to a process with gdb
 +
<pre>
 +
[david@fedora-14] $ gdb attach $(pidof -x name-of-program)
 +
</pre>
 +
you should be able to use the "heap" command to get a breakdown of how that process is using memory.
 +
 +
You can also do this with core dumps:
 +
<pre>
 +
[david@fedora-14] $ gdb -c core.1976
 +
</pre>
 +
 +
In this example, I've attached gdb to a python process:
 +
<pre>
 +
(gdb) heap
 +
      Domain                        Kind              Detail  Count  Allocated size
 +
-------------  --------------------------  ------------------  ------  --------------
 +
      python                        str                      6,689        477,840
 +
      cpython          PyDictEntry table                        167        456,944
 +
      cpython          PyDictEntry table            interned      1        200,704
 +
      python                        str            bytecode    648          92,024
 +
uncategorized                                        32 bytes  2,866          91,712
 +
      python                        code                        648          82,944
 +
uncategorized                                      4128 bytes      19          78,432
 +
      python                    function                        609          73,080
 +
      python          wrapper_descriptor                        905          72,400
 +
      python                        dict                        247          71,200
 +
uncategorized                                        72 bytes    852          61,344
 +
(snipped)
 +
</pre>
 +
 +
As you can see, gdb-heap will attempt to categorize the chunks of dynamically-allocated memory that it finds.  It shows you how many blocks of memory of each category it found, with the categories sorted by the number of bytes of RAM that they're using.
 +
 +
The categorization is divided into three parts:
 +
* ''domain'': high-level grouping e.g. "python", "C++", etc
 +
* ''kind'': type information, appropriate to the domain e.g. a class/type
 +
* ''detail'': additional detail (e.g. the size of a buffer, or a note that this python string is actually bytecode)
 +
 +
Some domains:
 +
{|
 +
!Domain                !! Meaning of 'kind'
 +
|-
 +
| <code>C</code>      || ''string data'' signifies a NUL-terminated string
 +
|-
 +
| <code>python</code>  ||the python class
 +
|-
 +
| <code>cpython</code> ||C structure/type (implementation detail within Python)
 +
|-
 +
| <code>pyarena</code> ||Python's optimized memory allocator
 +
|-
 +
| <code>uncategorized</code> || (none; gdb-heap wasn't able to identify what this is used for)
 +
|-
 +
| <code>C++</code>    ||the C++ class (disabled for now in Fedora 14's gdb-heap; the heuristic needs to be optimized)
 +
|-
 +
| <code>GType</code>  ||the GLib type/GObject class, within the GTK+ stack (not yet present Fedora 14's gdb-heap; in upstream git, but needs to be made robust and optimized)
 +
|}
 +
 +
You can see in the above example that much of the memory is taken up by python strings (the "str" type), but a considerable amount is also occupied by implementation details of python dictionaries (the "PyDictEntry tables").
 +
 +
 +
There are numerous subcommands. heap is integrated into gdb's tab-completion, so that you can see the available commands with the TAB key:
 +
<pre>
 +
(gdb) heap
 +
[TAB pressed]
 +
all    diff  label  log    sizes  used 
 +
</pre>
 +
 +
Here's a tour of what's available.  Refer to [https://fedorahosted.org/gdb-heap/wiki the upstream documentation] for more information.
 +
 +
=== Finding blocks of RAM (query language) ===
 +
gdb-heap has a <code>heap select</code> subcommand, which provides a simple language for querying for blocks matching criteria.
 +
 +
For example, here's how to find all dynamically-allocated block of a given size:
 +
<pre>
 +
(gdb) heap select size == 1778224
 +
            Start                End        Domain  Kind        Detail                                                                            Hexdump
 +
------------------  ------------------  -------------  ----  -------------  ----------------------------------------------------------------------------------
 +
0x000000000360a810  0x00000000037bca3f  uncategorized        1778224 bytes  00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc |...C...`...?........|
 +
0x00000000068596c0  0x0000000006a0b8ef  uncategorized        1778224 bytes  00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc |...C...`...?........|
 +
</pre>
 +
 +
You can query on any of 'domain', 'kind', 'detail', 'addr', 'start', 'size', and use equalities, inequalities and booleans.
 +
 +
Here's a query for all NUL-terminated C strings above a particular size (scroll the page right to see the hexdump)
 +
<pre>
 +
(gdb) heap select kind="string data" and size > 512
 +
Blocks retrieved 10000
 +
            Start                End  Domain        Kind  Detail                                                                            Hexdump
 +
------------------  ------------------  ------  -----------  ------  ----------------------------------------------------------------------------------
 +
0x0000000000624070  0x000000000062430f      C  string data          41 20 63 6f 6e 74 65 78 74 20 6d 61 6e 61 67 65 72 20 74 68 |A context manager th|
 +
0x0000000000627b50  0x0000000000627e8f      C  string data          41 20 64 65 63 6f 72 61 74 6f 72 20 69 6e 64 69 63 61 74 69 |A decorator indicati|
 +
0x0000000000628b90  0x0000000000628e0f      C  string data          4d 65 74 61 63 6c 61 73 73 20 66 6f 72 20 64 65 66 69 6e 69 |Metaclass for defini|
 +
0x0000000000661320  0x000000000066170f      C  string data          20 10 65 00 00 00 00 00 01 00 00 00 00 00 00 00 20 2e 78 05 | .e............. .x.|
 +
0x00000000006a2410  0x00000000006a27ff      C  string data          20 13 66 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | .f.................|
 +
</pre>
 +
 +
=== History ===
 +
gdb-heap provides a "history" feature, somewhat analogous to a revision-control system such as git, although the data is merely stored within the gdb process (it does not persist to disk, and is lost when gdb exits)
 +
 +
<code>heap label</code> allows you to take a named snapshot of the current state of the heap.
 +
<pre>
 +
(gdb) heap label "about to 'import sys'"
 +
Blocks retrieved 10000
 +
2,626,416 allocated, in 18716 blocks
 +
(gdb) cont
 +
>>> import sys
 +
Ctrl+C
 +
(gdb) heap label "after 'import sys'"
 +
Blocks retrieved 10000
 +
2,626,688 allocated, in 18720 blocks
 +
</pre>
 +
 +
<code>heap log</code> shows you all such named snapshots:
 +
<pre>
 +
(gdb) heap log
 +
Label 2 ""after 'import sys'"" at 2010-09-16 18:01:17.879123
 +
    2,626,688 allocated, in 18720 blocks
 +
 +
    +272 bytes, +4 blocks
 +
 +
Label 1 ""about to 'import sys'"" at 2010-09-16 17:55:17.030709
 +
    2,626,416 allocated, in 18716 blocks
 +
</pre>
 +
 +
<code>heap diff</code> allows you to compare dynamic memory allocations between two different states: either those in the log, or with the current state.
 +
<pre>
 +
(gdb) heap diff
 +
Blocks retrieved 10000
 +
Changes from "about to 'import sys'" to current
 +
  +272 bytes, +4 blocks
 +
 +
  Free-d blocks:
 +
    0x0000000000602010 -> 0x000000000060206f      96 bytes Category(domain='uncategorized', kind='', detail='96 bytes') |00 00 00 00 00 00 00 00 a0 20 60 00 00 00 00 00 50 01 62 00 |......... `.....P.b.|
 +
    0x0000000000602070 -> 0x000000000060209f      48 bytes Category(domain='uncategorized', kind='', detail='48 bytes') |01 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |....................|
 +
(snipped)
 +
 +
</pre>
 +
 +
You can use this in conjunction with breakpoints and stepping through the code to get a sense of how different parts of the program affect memory usage.
 +
 +
=== Hexdumps ===
 +
In addition to <code>heap</code>, gdb-heap also provides a <code>hexdump</code> command, to help you figure out what a block of data is being used for.
 +
<pre>
 +
(gdb) hexdump 0x000000000360a810
 +
0x000000000360a810 -> 0x000000000360a82f 00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc 00 00 00 10 00 00 00 64 00 00 00 08 |...C...`...?...............d....|
 +
0x000000000360a830 -> 0x000000000360a84f 00 00 00 00 00 00 00 01 00 00 03 e8 00 00 00 06 00 00 00 02 00 00 00 01 00 00 03 e9 00 00 00 06 |................................|
 +
0x000000000360a850 -> 0x000000000360a86f 00 00 00 10 00 00 00 01 00 00 03 ea 00 00 00 06 00 00 00 16 00 00 00 01 00 00 03 ec 00 00 00 09 |................................|
 +
(snip)
 +
</pre>
 +
 +
The output shows a split view, showing address ranges, hexadecimal values, and ASCII values, where printable ("." elsewhere).
 +
 +
=== Showing all dynamic memory ===
 +
"heap all" shows a detailed, low-level report on all dynamically-allocated chunks of memory.  This is a simple loop through memory, typically showing you the large allocations first (implemented via "mmap"), then the smaller ones (implemented within the "sbrk" region).
 +
 +
It reports the start/end of each region, along with book-keeping information about the block.
 +
 +
This is likely to only be of use for debugging low-level problems.
 +
<pre>
 +
(gdb) heap all
 +
All chunks of memory on heap (both used and free)
 +
-------------------------------------------------
 +
0: 0x00007ffff08cd000 -> 0x00007ffff090dfff  inuse: 266240 bytes (<MChunkPtr chunk=0x7ffff08cd000 mem=0x7ffff08cd010 prev_size=0 IS_MMAPPED chunksize=266240 memsize=266224>)
 +
1: 0x00007ffff7ea7000 -> 0x00007ffff7ee7fff  inuse: 266240 bytes (<MChunkPtr chunk=0x7ffff7ea7000 mem=0x7ffff7ea7010 prev_size=0 IS_MMAPPED chunksize=266240 memsize=266224>)
 +
(copious output snipped)
 +
</pre>
  
 
== Dependencies ==
 
== Dependencies ==
 
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
 
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
 
There's a baseline of functionality that I'm developing on top of Fedora 13's gdb.
 
There's a baseline of functionality that I'm developing on top of Fedora 13's gdb.
 +
 +
The gdb-heap code peeks around inside the internals of the glibc heap implementation, violating encapsulation (rather by definition for a debugger), so if that changes, corresponding changes will need to be made to gdb-heap.
  
 
Some features require additional work in gdb, which I've filed RFE bugs for.  Naturally this will require coordination with gdb to ensure that they land in Fedora 14:
 
Some features require additional work in gdb, which I've filed RFE bugs for.  Naturally this will require coordination with gdb to ensure that they land in Fedora 14:
 
* {{bz|610241}}: RFE: please expose "info symbol ADDRESS" in the python API  
 
* {{bz|610241}}: RFE: please expose "info symbol ADDRESS" in the python API  
 
* {{bz|610249}}: RFE: notification about changes in the inferior process
 
* {{bz|610249}}: RFE: notification about changes in the inferior process
 
  
 
== Contingency Plan ==
 
== Contingency Plan ==
Line 75: Line 277:
 
== Documentation ==
 
== Documentation ==
 
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
 
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
*
+
* See above, and at [https://fedorahosted.org/gdb-heap/wiki the project's website].
  
 
== Release Notes ==
 
== Release Notes ==
 
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
*
+
* The gdb debugger has been extended with new commands that make it easier to track down and fix excessive memory usage within programs and libraries.  This functionality was created by Fedora contributor David Malcolm, and we believe it is unique to Fedora 14.
  
 
== Comments and Discussion ==
 
== Comments and Discussion ==
* See [[Talk:Features/YourFeatureName]]  <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
+
* See [[Talk:Features/MemoryDebuggingTools]]  <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
  
  
[[Category:FeaturePageIncomplete]]
+
[[Category:FeatureAcceptedF14]]
 
<!-- When your feature page is completed and ready for review -->
 
<!-- When your feature page is completed and ready for review -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
 
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
 
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
 
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 19:00, 22 September 2010

Contents

[edit] Memory Debugging Tools

[edit] Summary

The gdb debugger has been extended with new commands that make it easier to track down and fix excessive memory usage within programs and libraries.

This functionality was created by Fedora contributor David Malcolm, and we believe it is unique to Fedora.

[edit] Owner

[edit] Current status

  • Targeted release: Fedora 14
  • Last updated: 2010-09-16
  • Percentage of completion: 100%

TODO: This is "feature-complete", but some issues remain:

  • I need to blog about this and write better docs
  • Fix the bugs
  • Testing

You will need gdb-7.2-7.fc14 or later when attaching to a process (rather than starting the process within gdb), due to bug 634660

DONE:

  • Preparing upstream project for initial launch: https://fedorahosted.org/gdb-heap/
  • I've disabled C++ support for now, as the current implementation slows down other operations.

Initial version of code uploaded

Packaged and reviewed; available as an update to F14.

Upstream tickets:

[edit] Detailed Description

The new "gdb-heap" package adds a new "heap" command to /usr/bin/gdb.

The command allows you to get a breakdown of how that process is using dynamic memory.

It allows for unplanned memory usage debugging: if a process unexpectedly starts using large amounts of memory you can attach to it with gdb, and use the heap command to figure out where the memory is going. You should also be able to use it on core dumps.

We believe this approach is entirely new, and is unique to Fedora 14.

[edit] Benefit to Fedora

This feature could be of great use to developers and system administrators: it provides a new way of analyzing how a process uses memory, without requiring advance planning.

It is unique to Fedora (it makes heavy use of the gdb/python integration we have in Fedora), and was developed by a Fedora contributor (who is a Red Hat engineer).

[edit] Scope

Code is isolated, as an extension to gdb, written in Python.

  • I'm tracking development of the code in the upstream tracker here:
  • Package the code in RPM form, add it to Fedora
  • Ensure that it's available without the user needing excessive configuration; ideally, if the rpm is installed, then you get the command automagically
  • Add it to comps so that it's suggested for installed by default if gdb is installed.
  • Testing

[edit] How To Test

No special hardware is needed.

You will need to install the gdb-heap package (not yet packaged)

[edit] Exploratory testing

  • Pick a process on your system (either as root, or one of your own processes)
  • Use "gdb attach PID" to connect to it
  • Use python import heap to register the "heap" command
  • Use the "heap" command and its various subcommands (as described on the upstream website)
  • Ensure that all results look correct, and that there are no Python tracebacks within gdb.

Ideally the amount of "uncategorized" data should not be a substantial proportion of the overall size of the dynamically-allocated memory; if it is, then that may be a bug.

Ideally the command should not take too long to run. The more blocks of memory that are "live" within a process, the longer it will take to analyze the usage. Crude timings suggest it can analyze about 5000 allocations per second, so if you have a process with 300,000 allocations, it could take a minute to analyze them.

[edit] User Experience

[edit] Basic Operation

Having attached to a process with gdb

[david@fedora-14] $ gdb attach $(pidof -x name-of-program)

you should be able to use the "heap" command to get a breakdown of how that process is using memory.

You can also do this with core dumps:

[david@fedora-14] $ gdb -c core.1976

In this example, I've attached gdb to a python process:

(gdb) heap
       Domain                        Kind              Detail   Count  Allocated size
-------------  --------------------------  ------------------  ------  --------------
       python                         str                       6,689         477,840
      cpython           PyDictEntry table                         167         456,944
      cpython           PyDictEntry table            interned       1         200,704
       python                         str            bytecode     648          92,024
uncategorized                                        32 bytes   2,866          91,712
       python                        code                         648          82,944
uncategorized                                      4128 bytes      19          78,432
       python                    function                         609          73,080
       python          wrapper_descriptor                         905          72,400
       python                        dict                         247          71,200
uncategorized                                        72 bytes     852          61,344
(snipped)

As you can see, gdb-heap will attempt to categorize the chunks of dynamically-allocated memory that it finds. It shows you how many blocks of memory of each category it found, with the categories sorted by the number of bytes of RAM that they're using.

The categorization is divided into three parts:

  • domain: high-level grouping e.g. "python", "C++", etc
  • kind: type information, appropriate to the domain e.g. a class/type
  • detail: additional detail (e.g. the size of a buffer, or a note that this python string is actually bytecode)

Some domains:

Domain Meaning of 'kind'
C string data signifies a NUL-terminated string
python the python class
cpython C structure/type (implementation detail within Python)
pyarena Python's optimized memory allocator
uncategorized (none; gdb-heap wasn't able to identify what this is used for)
C++ the C++ class (disabled for now in Fedora 14's gdb-heap; the heuristic needs to be optimized)
GType the GLib type/GObject class, within the GTK+ stack (not yet present Fedora 14's gdb-heap; in upstream git, but needs to be made robust and optimized)

You can see in the above example that much of the memory is taken up by python strings (the "str" type), but a considerable amount is also occupied by implementation details of python dictionaries (the "PyDictEntry tables").


There are numerous subcommands. heap is integrated into gdb's tab-completion, so that you can see the available commands with the TAB key:

(gdb) heap
[TAB pressed]
all    diff   label  log    sizes  used   

Here's a tour of what's available. Refer to the upstream documentation for more information.

[edit] Finding blocks of RAM (query language)

gdb-heap has a heap select subcommand, which provides a simple language for querying for blocks matching criteria.

For example, here's how to find all dynamically-allocated block of a given size:

(gdb) heap select size == 1778224
             Start                 End         Domain  Kind         Detail                                                                             Hexdump
------------------  ------------------  -------------  ----  -------------  ----------------------------------------------------------------------------------
0x000000000360a810  0x00000000037bca3f  uncategorized        1778224 bytes  00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc |...C...`...?........|
0x00000000068596c0  0x0000000006a0b8ef  uncategorized        1778224 bytes  00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc |...C...`...?........|

You can query on any of 'domain', 'kind', 'detail', 'addr', 'start', 'size', and use equalities, inequalities and booleans.

Here's a query for all NUL-terminated C strings above a particular size (scroll the page right to see the hexdump)

(gdb) heap select kind="string data" and size > 512
Blocks retrieved 10000
             Start                 End  Domain         Kind  Detail                                                                             Hexdump
------------------  ------------------  ------  -----------  ------  ----------------------------------------------------------------------------------
0x0000000000624070  0x000000000062430f       C  string data          41 20 63 6f 6e 74 65 78 74 20 6d 61 6e 61 67 65 72 20 74 68 |A context manager th|
0x0000000000627b50  0x0000000000627e8f       C  string data          41 20 64 65 63 6f 72 61 74 6f 72 20 69 6e 64 69 63 61 74 69 |A decorator indicati|
0x0000000000628b90  0x0000000000628e0f       C  string data          4d 65 74 61 63 6c 61 73 73 20 66 6f 72 20 64 65 66 69 6e 69 |Metaclass for defini|
0x0000000000661320  0x000000000066170f       C  string data          20 10 65 00 00 00 00 00 01 00 00 00 00 00 00 00 20 2e 78 05 | .e............. .x.|
0x00000000006a2410  0x00000000006a27ff       C  string data          20 13 66 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | .f.................|

[edit] History

gdb-heap provides a "history" feature, somewhat analogous to a revision-control system such as git, although the data is merely stored within the gdb process (it does not persist to disk, and is lost when gdb exits)

heap label allows you to take a named snapshot of the current state of the heap.

(gdb) heap label "about to 'import sys'"
Blocks retrieved 10000
2,626,416 allocated, in 18716 blocks
(gdb) cont
>>> import sys
Ctrl+C
(gdb) heap label "after 'import sys'"
Blocks retrieved 10000
2,626,688 allocated, in 18720 blocks

heap log shows you all such named snapshots:

(gdb) heap log
Label 2 ""after 'import sys'"" at 2010-09-16 18:01:17.879123
     2,626,688 allocated, in 18720 blocks

     +272 bytes, +4 blocks

Label 1 ""about to 'import sys'"" at 2010-09-16 17:55:17.030709
     2,626,416 allocated, in 18716 blocks

heap diff allows you to compare dynamic memory allocations between two different states: either those in the log, or with the current state.

(gdb) heap diff
Blocks retrieved 10000
Changes from "about to 'import sys'" to current
   +272 bytes, +4 blocks

  Free-d blocks:
    0x0000000000602010 -> 0x000000000060206f       96 bytes Category(domain='uncategorized', kind='', detail='96 bytes') |00 00 00 00 00 00 00 00 a0 20 60 00 00 00 00 00 50 01 62 00 |......... `.....P.b.|
    0x0000000000602070 -> 0x000000000060209f       48 bytes Category(domain='uncategorized', kind='', detail='48 bytes') |01 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |....................|
(snipped)

You can use this in conjunction with breakpoints and stepping through the code to get a sense of how different parts of the program affect memory usage.

[edit] Hexdumps

In addition to heap, gdb-heap also provides a hexdump command, to help you figure out what a block of data is being used for.

(gdb) hexdump 0x000000000360a810
0x000000000360a810 -> 0x000000000360a82f 00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc 00 00 00 10 00 00 00 64 00 00 00 08 |...C...`...?...............d....|
0x000000000360a830 -> 0x000000000360a84f 00 00 00 00 00 00 00 01 00 00 03 e8 00 00 00 06 00 00 00 02 00 00 00 01 00 00 03 e9 00 00 00 06 |................................|
0x000000000360a850 -> 0x000000000360a86f 00 00 00 10 00 00 00 01 00 00 03 ea 00 00 00 06 00 00 00 16 00 00 00 01 00 00 03 ec 00 00 00 09 |................................|
(snip)

The output shows a split view, showing address ranges, hexadecimal values, and ASCII values, where printable ("." elsewhere).

[edit] Showing all dynamic memory

"heap all" shows a detailed, low-level report on all dynamically-allocated chunks of memory. This is a simple loop through memory, typically showing you the large allocations first (implemented via "mmap"), then the smaller ones (implemented within the "sbrk" region).

It reports the start/end of each region, along with book-keeping information about the block.

This is likely to only be of use for debugging low-level problems.

(gdb) heap all
All chunks of memory on heap (both used and free)
-------------------------------------------------
0: 0x00007ffff08cd000 -> 0x00007ffff090dfff  inuse: 266240 bytes (<MChunkPtr chunk=0x7ffff08cd000 mem=0x7ffff08cd010 prev_size=0 IS_MMAPPED chunksize=266240 memsize=266224>)
1: 0x00007ffff7ea7000 -> 0x00007ffff7ee7fff  inuse: 266240 bytes (<MChunkPtr chunk=0x7ffff7ea7000 mem=0x7ffff7ea7010 prev_size=0 IS_MMAPPED chunksize=266240 memsize=266224>)
(copious output snipped)

[edit] Dependencies

There's a baseline of functionality that I'm developing on top of Fedora 13's gdb.

The gdb-heap code peeks around inside the internals of the glibc heap implementation, violating encapsulation (rather by definition for a debugger), so if that changes, corresponding changes will need to be made to gdb-heap.

Some features require additional work in gdb, which I've filed RFE bugs for. Naturally this will require coordination with gdb to ensure that they land in Fedora 14:

  • RHBZ #610241: RFE: please expose "info symbol ADDRESS" in the python API
  • RHBZ #610249: RFE: notification about changes in the inferior process

[edit] Contingency Plan

None necessary, simply remove the package

[edit] Documentation

[edit] Release Notes

  • The gdb debugger has been extended with new commands that make it easier to track down and fix excessive memory usage within programs and libraries. This functionality was created by Fedora contributor David Malcolm, and we believe it is unique to Fedora 14.

[edit] Comments and Discussion