Rasmus Thomsen/Introducing Corecollector, a coredump collector written in D

Created Tue, 18 Feb 2020 23:13:10 +0100
1066 Words

I’ve recently wanted to debug gnome-shell, that was crashing at seemingly random times for me (especially after hibernating), but attaching a debugger to gnome-shell isn’t as easy as it appears. I’ve noticed that it tended to dump a core.$pid file in my $HOME, which is a coredump file one can load into a debugger to print the backtrace among other things to after the fact. It annoyed me a little that it cluttered my $HOME and I figured it’d be nice having a manager for these coredumps so one can see what programs crashed, why and how and to easily rotate them. I’ve remembered using systemd-coredumpd at some point and kind of liked that, but since I’m using Alpine Linux that wasn’t an option, so I wrote corecollector.

Installation

If corecollector is packaged in your distribution installing corecollector should be as simple as installing it via your package manager. From then on corecollector will collect coredumps for you (see further down for the usage).

If corecollector hasn’t been packaged yet, you can install it manually:

Simply download a release from https://github.com/Cogitri/corecollector/releases , unpack the tarball and cd into the resulting dir. Then run:

ninja --buildtype=release build
ninja -C build install

You’ll need a functional D compiler for this to work. You’ll also have to create a corecollector user and group, as which corehelper, which is invoked by the kernel when a program crashes and a core is dumped, will save the coredumps to avoid running as root for longer than required. You can add users to the corecollector group to allow them to access coredumps without root, but be mindful that coredumps are snapshots of the memory of a program and can contain highly sensitive data.

Using (the CLI)

The main binary to interact with is corectl, here’s it’s --help which explains most things:

Usage:
  corectl <subcommand> [OPTION...]

List and interact with coredumps

Subcommands:
  backtrace [ID]    - Print the backtrace of the coredump identified by ID.
  debug [ID]        - Open the coredump identified by ID in a debugger.
  dump  [ID] [FILE] - Dump the coredump identified by ID to file denoted by FILE.
                      Defaults to 'stdout'.
  info [ID]         - Get more detailed information for the coredump identified by ID.
  list              - List all available coredumps.

Help Options:
  -h, --help         - Show help options.

Application Options:
  -v, --version      - Print program version.
  -d, --debug [0-3]  - Specify the debug level.

If you want to know what programs have crashed you can simply run corectl list:

$ corectl list

ID    SIGNAL   UID      GID      PID      TIMESTAMP             EXE
1     6        1000     1000     4837     2020-Feb-18 18:32:05  /usr/bin/Xwayland
2     5        105      109      30052    2020-Feb-18 18:32:06  /usr/bin/gnome-shell
3     11       1000     1000     5296     2020-Feb-18 21:17:49  /usr/bin/gnome-software
4     31       1000     1000     6830     2020-Feb-18 21:19:21  /usr/libexec/tracker-extract
5     11       1000     1000     7770     2020-Feb-18 21:24:41  /usr/local/bin/gnome-software

You can then run corectl debug $ID to attach a debugger to one of the coredumps:

$ corectl debug 3
GNU gdb (GDB) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/gnome-software...
[New LWP 28750]
[New LWP 28749]
[New LWP 28755]
[New LWP 28753]
[New LWP 28747]
[New LWP 28748]
[New LWP 28757]
[New LWP 28751]
[New LWP 28752]
[New LWP 28759]
[New LWP 28756]
[New LWP 28754]

Core was generated by `gnome-software'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ffbdf330a2c in composite_boxes (compositor=compositor@entry=0x7ffbdf3cc920 <spans>, extents=extents@entry=0x7ffbdd9c76f0, boxes=boxes@entry=0x7ffbdd9c72d0) at cairo-spans-compositor.c:720
720	cairo-spans-compositor.c: No such file or directory.
[Current thread is 1 (LWP 28750)]
(gdb) bt
#0  0x00007ffbdf330a2c in composite_boxes (compositor=compositor@entry=0x7ffbdf3cc920 <spans>, extents=extents@entry=0x7ffbdd9c76f0, boxes=boxes@entry=0x7ffbdd9c72d0) at cairo-spans-compositor.c:720
...
..
.

You can also print the backtrace directly via corectl backtrace $ID which tends to be the most frequently used feature:

$corectl backtrace 3
#0  0x00007ffbdf330a2c in composite_boxes (compositor=compositor@entry=0x7ffbdf3cc920 <spans>, extents=extents@entry=0x7ffbdd9c76f0, boxes=boxes@entry=0x7ffbdd9c72d0) at cairo-spans-compositor.c:720
...
..
.

Configuration

Corecollector also has a few configuration options, see /etc/corecollector/corecollector.conf:

# What compression to use for coredumps.
# Available choices: none zlib
compression = zlib
# The maximum size of coredumps.
# Unset or set to 0 to set no size limit.
# Measured in KByte, must be whole numbers.
maxSize = 0
# The path to save coredumps to.
targetPath = /var/lib/corecollector
# Where to log to. Pass /dev/null to only log to syslog
logPath = /var/log/corecollector.log
# Enable debugging
enableDebug = false
# The maximum size of the coredumpdir.
# Unset or set to 0 to set no size limit.
# Measured in KByte, must be whole numbers.
maxDirSize = 0

It uses zlib for compression by default, since coredumps tend to be very easy to compress. As such a coredump might be some 400KiB big when uncompressed but be as small as 5 KiB when compressed. Corecollector also offers options to limit the maximum size of coredumps, rotate the coredump directory once it reaches a certain size or to log to a certain path (additionally to syslog).

Conclusion

Corecollector is made to make debugging programs easy and to make users aware of crashing programs, since it can be easy to miss them in the logs or dmesg. I hope it proves useful to you :)

Corecollector was my first project in DLang and it was a very pleasant experience. Since DLang has a pretty small community (and as such doesn’t have a too broad library landscape) it can seem a little frightening to get started with it, but I’ve found it to be an incredibly productive language: I don’t feel held back because it’s somewhat “low-level-ish”, but due to it’s GC (which can easily be controlled or even turned off completely for performance critical areas) it’s easy to use without introducing security vulnerabilities due to use-after-frees or double-frees and without having to worry about lifetimes or similar. It’s also easy to bind to C libraries in D (one only needs the function declaration and link against the right library and it’s as if you’re calling it from C), so the library landscape doesn’t look so bad after all :)