RolandMcGrath/BuildID

Quick debuginfo validation
The current  ties a stripped object (executable, DSO, ) to its   file with a name (in practice always  in a file originally called  ), and a (weak) CRC32 checksum of the  file. The only way to find the  file is by name based on knowing the name of the stripped object, the two having come in parallel from rpm or whatever. The only way to verify it's really the right one is to read the entire contents of the  file from disk to checksum it. This is obviously poor for memory and I/O performance, especially as the DWARF consumers keep improving to avoid reading in as much debuginfo at once.

A while ago, a few of us spoke about improving this. What we discussed was switching to a strong checksum (sha1sum) so that comparing two stored checksums is sufficient for confidence in validation. The important change is to have the checksum stored in both the  file and the stripped file, so that there is a small amount of data to read from the file that is sufficient to do the validation. We had in mind simply a new kind of debuglink section (still not allocated) that /  would insert into both output files. We didn't really change the checksum plan aside from the algorithm, i.e. using sha1sum of the entire unstripped file or of the  file with an all-zeros debuglink section.

In later contemplation, I became dissatisfied with using a checksum on the entire contents of the  file. I would like to be able to perform transformations on the DWARF data after the fact (fancy compression et al) and still say "this is the DWARF data for your binary". It feels wrong to have to edit the stripped binary to make it match transformed debug data. (In the abstract, I also think one should always be able to do spurious ELF file layout juggling that doesn't change the semantics of the data.) So I thought about using sha1sum of the loaded segments and phdrs or something. But that's not right. Your  and my   are going to produce the same stripped binaries, but my binaries should get me the source code with my comments in it, and not yours just because it's identical modulo pontification. (It's all about the pontification!) But, the real plan behind using a strong checksum was never actually to compute a checksum from the data ever again after build time, because you really just rely on the comparison of a strongly unique embedded identifier.

Finding binaries for dumps
People dealing with core files often confront the problem of not knowing exactly which executable and DSO binaries were in use when the program was running. It may have been hours or months after the crash happened when a person is attempting ernest post-mortem analysis, and one really needs to have the right binaries to get anywhere. Normally the text segments are elided from the dump, so you don't have the original code to examine or to compare to a file on disk you think was the one in use.

For post-mortem debugging now, the only way to find the DSOs involved is to examine the dynamic linker's data in the dump's memory image where quasi-standard data structures give the file names the dynamic linker used to open the DSOs. That process requires you know the right address in the executable's data to look at, or know the right dynamic linker file to consult and grok its internal symbols. So even to get bootstrapped, you need to have the right executable file (or perhaps the right dynamic linker). Then, all you have are names, with no clear way to know whether the files by those names now match what existed there when this dump was made.

The case of kernel-mode/whole-machine dumps is not fundamentally different from user core dump files, except that usually all the memory is dumped so you have the text to look at.

File storage space being what it is, perhaps many people would be happy to have core dumps include the full text segments ( see below ); the data being dumped is often far larger than the text already. Full text segments are very handy to make e.g. disassembly and unwind info (if in ) available immediately without consulting any other files at all. But, often you really want to know the detailed provenance of the binary, or at least quickly find the debuginfo that goes with it. What you need from the dump is an unambiguous identifier that you can associate with each DSO/executable that went into the crash. Having the whole text is not so important, because you really just rely on the lookup and comparison of a strongly unique embedded identifier.

Unique build ID
What we really want is a unique build ID. At first I liked the canonical UUID generation for this (128 bits of random or of something time and host-based). But that has the very undesireable property of making for unreproducible builds, where it's between difficult and impossible to start from the same conditions and repeat the procedure of making binaries from all the same constituents to get binaries without gratuitous differences from the original build. Perhaps something like  of the unstripped file is what we want to use as the basis of a reproducible identifier unique to completely identical builds. But I'd like to specify it explicitly as being a unique identifier good only for matching, not any kind of checksum that can be verified against the contents. (There are external general means for content verification, and I don't think debuginfo association needs to do that.)

To embed an ID into both the stripped object and its  file, I've chosen to use an ELF note section. et al can keep the section intact in both files when its type is. The new section is canonically called, but the name is not normative, and the section can be merged with other  sections. The ELF note headers give name  and type 3  for a build ID note, of which there can be only one in a linked object (or an   file of the   style). The  and the following data can be any nonzero number of bytes chosen by the producer. It should be long enough to be plausibly truly unique while not being unreasonably long to use as a shorthand. The bits should be chosen in a fashion that makes it a useful approximation of true uniqueness across all binaries that might be used by overlapping sets of people. Likely common sizes are 16 (UUID or MD5) and 20 (SHA1). The section canonically has  in , meaning it appears in the loaded memory image, and the normal link order puts allocated note sections very early in the memory image.

Use build ID to validate debuginfo
DWARF consumers that look at separate debuginfo files to go with a stripped file now validate their association by doing CRC32 on the whole debuginfo file (sometimes huge) to compare with the  checksum. They can change to check for a build ID note in the stripped file. When a build ID is present, check that the debuginfo file contains an identical corresponding  section. This requires reading only a small part of each file, probably the first and last pages of each.

Include build IDs in core dumps
In the kernel's core dump code, it's easy to detect the mappings likely to be the first mapping for an executable or DSO:  to a file at offset 0, with   in the first word of the mapped contents. For each of those, include the first page of the mapping in the core dump ( = ,   = larger total). The build ID note is normally near the beginning of the image and will be in the first page unless there are an unreasonable number of phdrs or other notes. (The kernel should not deal with anything more complex than a simple rule like the first page, so the odd binary with its notes in the wrong places will just lose.) If it was some innocent mmap'ing and not really a loaded DSO or the executable, then no harm done. Noone minds several extra pages in a core dump (it just shouldn't have whole huge text segments).

It's possible that existing consumers are confused by a core file segment with  <   but not   == 0. If need be, we can write a simple tool to extract ELF ident notes and list ID:address associations while removing the extra pages from the core file. GDB loaded the remaining part of the code segments zeroed, fix is accepted upstream and present since Rawhide gdb-6.6-21.fc8.

The Work
Work I have done so far that is not already upstream is in http://people.redhat.com/roland/build-id/.

Put a build ID into every binary

 * Compiler toolchain


 * : new option  			DONE

This adds an option to  to synthesize a  section with type   and flags   (read-only data), that contains an ELF note header and the build ID bits. This then goes into the link as if it were part of the first object file (so it may be placed or merged by the linker script). The build ID bits are determined as the very last thing  does before writing out the linked file. You can give  chose ,   (128 random bits), or   (your chosen bytes in hex). Just defaults to, which computes an 128-bit MD5 signature based all the ELF header bits and section contents in the file--i.e., an ID that is unique among the set of meaningful contents for ELF files and identical when the output file would otherwise have been identical.

The Linux binutils-2.17.50.0.17 release includes this, in f8test1.

binutils-2.17.50.0.17-3 and later in Rawhide have the feature.


 * : keep allocated notes in debuginfo	DONE

The  option used for separate debuginfo generation has been changed to preserve all  sections intact in the debuginfo file. The version released in  does this.


 * : keep allocated notes in debuginfo	DONE

The  option is used for separate debuginfo generation in regimes not using. I have committed a change upstream to keep all  sections intact in the debuginfo file. The Linux binutils-2.17.50.0.17 release includes this, in f8test1.


 * compiler passes option by default			POSTED, RAWHIDE

So this can be useful all the time, the default compiler settings should pass  for every normal final link of a DSO or executable. Probably  or something. We can try this first in Fedora 8, but I think it is a reasonable default for upstream gcc once ld supports.

f8test1 gcc has this. I've sent a patch upstream, but it has not been integrated yet.


 * Packaging


 * DONE

The program  is used in  's separate debuginfo creation. It modifies DWARF data to replace the build-time absolute directory names with consistent names. This makes the installed debuginfo's source references usable, and it makes for reproducible rpm builds from identical constituents to produce identical binaries. The build ID computed by  was affected by the name of the rpm build directory, so it will differ between two otherwise identical builds that used a different.

My patch adds the  option to make   recompute the build ID based on the contents of the file after the transformation. It also prints out the build ID bits in hex.

This is in rpm-4.4.2.1, in f8-test1.


 * DONE

The  script is what runs. I've modified the script to pass the new option to. Since my goal for Fedora8 is to have a build ID in every binary in the distribution, I've made it more sensitive to errors from  and it will fail if there was no build ID. It also adds to the debuginfo package a symlink to the binary and to the debuginfo file from ; [#symlinks see below].

The new script is part of rpm-build >= 4.4.2.1-4.fc8, in Rawhide after test1.

Linux kernel changes

 * core dump code					SUBMITTED,RAWHIDE

The upstream 2.6.23 kernel includes new controls via /proc/pid/coredump_filter. My patch adds a new bit there, which makes core dumping include the first page of an ELF file mapping that would otherwise have been elided. This is turned on by default in F8, but can be disabled for a process and its children via /proc/pid/coredump_filter.

kernel-2.6.23-0.30.rc0.git6.fc8 and later in Rawhide have the feature.


 * use  on the kernel	  		DONE

My patch series changes the kernel linker script on several machines to handle allocated note sections, and uses  when   supports it. Any other oddball programs that have a really good reason to use the linker directly instead of letting the compiler chose the options also need to be changed.

This is incorporated upstream now.


 * xen							NEEDED

Use  when linking the hypervisor.


 * assembly debuginfo					POSTED,RAWHIDE

My patch series turns on  for assembly files in the kernel, so the debuginfo is useful for even more of the code (to read comments in the source and so forth).

This is in -mm, and is enabled in Rawhide kernels now.


 * vDSO debuginfo					POSTED,RAWHIDE

My patch series also has the kernel's  copy unstripped versions of the vDSO images into  alongside the kernel's modules. These are linked as normal DSOs via, so they will get build IDs implicitly. Having these on disk means that when a build ID is found in the vDSO image embedded in a core dump, the same means used for normal DSOs can find the kernel debuginfo package and lead to seeing the vDSO's assembly in original source form.

This is in -mm, and is enabled in Rawhide kernels now.


 * /sys/kernel/notes					DONE

The bonus feature in my patch series adds the magic file. Reading this gives you the binary contents of the ELF notes section built into the kernel. Here you can find the build ID of the running kernel. This gives a solution to a problem that has arisen for  users, where nothing prevents them from using the  data to drive Systemtap's probe details, but are actually running the kernel from the rpm. This is a failure on many levels, but some simple sanity-checking at the bottom always helps. Now it is easy to verify you have the right debuginfo file for the kernel you are running.

This is incorporated upstream now.

Replace debuginfo CRC32 check
The consumers of debuginfo files that now use the CRC32 value in the section can look for build IDs and compare those instead.


 * RAWHIDE
 * UPSTREAM

Find files by build ID
There needs to be some standard way to take a build ID gleaned from a memory dump and look it up to learn its proper name and find its debuginfo file. A large organization producing lots of builds might want to incorporate build ID info into their own database tracking the details of all their builds. Such complex things are beyond the scope of my what I've designed.


 * Packager/consumer convention.	now de facto standard

My current thinking is to use some simple filesystem conventions for looking up build IDs. This uses everybody's favorite backend database for file-sized objects keyed by fixed-size bitstrings, the humble Unix directory. At least for system installed binaries, this seems appropriately simple and adequate for the number of unique IDs in use on one system.

My plan is that the debuginfo directory contains a subdirectory of symlinks named by the build ID bits rendered in ASCII hex, one symlink to the stripped file and one to the debuginfo file. For example,  for   (real ones are 32 or more chars long, not 10), so   might yield: /usr/lib/debug/.build-id/ab/cdef1234 -> ../../../../bin/foo /usr/lib/debug/.build-id/ab/cdef1234.debug -> ../../usr/bin/foo.debug

The advantages of this are: Consumers can just start looking for  in the directories where they now look for debuginfo files by name, no new user configuration is required. Go from build ID bits to open debuginfo file in one system call. For rpm, the changes in  are straightforward, so build-ID symlinks are included in the debuginfo rpm automatically along with the debuginfo files and source files. e.g. finding file names included in two different packages in the same repository can flag it for the debuginfo repository.
 * Use existing configuration for debuginfo directory or path
 * Simple convention for all users, packagers, consumers to follow
 * Optimal for consumers
 * Simple addition to existing packaging
 * Leverage existing packaging support for files
 * works with no special support
 * If something goes awry in the toolchain, existing mechanisms for


 * Fedora 8 rebuild with build IDs in binaries, symlinks in debuginfo DONE


 * build ID-aware consumer
 * build ID-driven find_debuginfo			UPSTREAM
 * build ID recovery from core files			WORK IN PROGRESS

This is in upstream elfutils and may still make F8 release. If not, it will be in an elfutils update shortly after F8.


 * build ID-aware consumer		RAWHIDE

Jan Kratochvil is working on this for F8.

Compatibility testing
DONE: old gdb no good

Existing versions of gdb are confused by a core file with segments that are only partially present in the dump but not elided completely. With my patch the kernel produces these, which have not been seen before. The meaning of the ELF header fields is perfectly clear and valid for this case, but it is a new wrinkle compared to tradition. This is why the new format option won't be enabled by default in upstream kernels for some time to come.