|
When crash is run on a dumpfile, at least two arguments are
always required:
-
The kernel object filename, often referred to as the kernel namelist.
When initially built from the kernel sources, its name is vmlinux;
typically it is then copied to the /boot directory, where the
operating system release number is appended, as in
vmlinux-2.4.21-4.ELsmp.
-
The dumpfile name. For example, Netdump-generated dumpfiles are named vmcore.
For example:
When crash is run on a live system, /dev/mem is used as the
memory image. Therefore, only the kernel object filename is required:
Furthermore, when crash is run on a live system,
the vmlinux argument is not required when the kernel
object file is located in any of the following locations:
- / (root directory)
- /boot
- /boot/efi/EFI/redhat
- any subdirectory of /usr/src
- /usr/src/redhat/BUILD/kernel-x.x.x/linux
- /usr/lib/debug/lib/modules/<release><flavor>
When the vmlinux file is not entered on the command line, a search
will be made in all of the directories above until a kernel object file
is found that contains a version string that matches the running
kernel, as indicated by /proc/version.
If a matching kernel is found, then crash may be invoked on
a live system simply by entering:
In the examples above, it is presumed that the vmlinux kernel
has been built with the -g C flag, which traditionally has not
been done by default. To address this requirement, Red Hat Enterprise Linux 3
(RHEL 3) and Red Hat Enterprise Linux 4 (RHEL 4) kernels are
now built with -g C flag. The manner of accessing the
debug data for
RHEL 3
and
RHEL 4
kernels is described in the following two sections.
Unfortunately, since Red Hat Advanced Server 2.1 kernels are not built with -g, the
kernel must be rebuilt;
directions for rebuilding
Red Hat Advanced Server 2.1
kernels can
be found
here.
In RHEL 3, the vmlinux kernel debug information is stripped
and stored in a separate debuginfo file.
The stripped vmlinux file in /boot has
an embedded link to its associated debuginfo file in
/usr/lib/debug/boot, so that the crash utility (and the
built-in gdb module) knows
where to find it:
| RHEL 3 kernel namelist in /boot |
RHEL 3 kernel debuginfo file in /usr/lib/debug/boot |
vmlinux-<release>.EL
vmlinux-<release>.ELsmp
vmlinux-<release>.ELhugemem
|
vmlinux-<release>.EL.debug
vmlinux-<release>.ELsmp.debug
vmlinux-<release>.ELhugemem.debug
|
The debuginfo files for a specific kernel <release> come from a
separate RPM that must be installed for the crash
utility to work. For example, the i686 RPM for the examples above would
be named kernel-debuginfo-<release>.i686.rpm,
and would install the debuginfo file for all three of the kernel flavors.
For example, to run crash on a live system, the associated
debuginfo package must be installed:
# uname -r
2.4.21-4.ELsmp
# rpm -ivh kernel-debuginfo-2.4.21-4.EL.i686.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo ########################################### [100%]
# ls /usr/lib/debug/boot
vmlinux-2.4.21-4.EL.debug
vmlinux-2.4.21-4.ELhugemem.debug
vmlinux-2.4.21-4.ELsmp.debug
#
|
Accordingly, if the running kernel's vmlinux
file is in one the search locations above, and its associated
debuginfo file is located in
the /usr/lib/debug/boot directory or in the
current directory from which crash is invoked, no arguments
are required to run on a live system:
However, if the linked debuginfo file is not in either of those locations,
it can be added to the crash command line along with the
vmlinux filename. So, for example, if the debuginfo file was
located in /tmp:
# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug
|
For analyzing dumpfiles however, the vmlinux file name must be on the command
line along with the dumpfile name, as in the following examples:
# crash /boot/vmlinux-2.4.21-4.ELsmp vmcore
|
or if the debuginfo file is not in the standard location:
# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug vmcore
|
The procedure has been made much simpler for RHEL-4 kernels. The kernel
is built with the -g flag, and the resultant vmlinux
file is stored in the associated debuginfo package.
After installing the debuginfo package, the vmlinux file for each kernel
flavor of a given RHEL 4 release will be installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for i686 kernels,
<flavor> can be either hugemem, smp, or nothing (for uniprocessor
kernels). For example:
# uname -r
2.6.9-6.39.ELsmp
# rpm -ivh kernel-debuginfo-2.6.9-6.39.EL.i686.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo ########################################### [100%]
#
# find /usr/lib/debug/lib/modules/2.6.9-6.39.EL* -name vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.ELhugemem/vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux
/usr/lib/debug/lib/modules/2.6.9-6.39.EL/vmlinux
#
|
Once the debuginfo package is installed, crash can be invoked on the live system
with no arguments, because the vmlinux file will be found automatically:
To run crash on a dumpfile, however, the appropriate vmlinux
file and the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux vmcore
|
If the running kernel was not built with the -g C flag,
then it is necessary to rebuild a kernel of the same configuration with
the -g C flag.
The essential change done by this kernel rebuild task is a modification
of top-level Makefile of the kernel source tree,
such that the CFLAGS definition contains the -g flag.
For example, this is the line that must be changed:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common
|
by adding the -g flag:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common -g
|
For example, since Red Hat Advanced
Server 2.1 kernels are not built with -g, a kernel
rebuild is required.
For a detailed example of how to rebuild a Red Hat Advanced Server 2.1 kernel
with the -g flag, please refer to these
directions.
Upon rebuilding the kernel, a new
vmlinux file will be created that contains the debug data
required by crash. However, the symbol values will not match
those of the running or dumped kernel. To deal with this inequity, the
actual symbol values can be gathered from
either the original non-debug vmlinux file or from its
associated System.map file. That being the case, two arguments must
be supplied to crash to fully describe the running/dumped kernel,
the newly-created vmlinux file compiled with -g, as well as a source of the
real symbol values.
So, for example, if the vmlinux file
built with -g were renamed to vmlinux.dbg, the invocation
line would look like this on a live system:
# crash vmlinux vmlinux.dbg
(or)
# crash /boot/System.map vmlinux.dbg
(or)
# crash -S vmlinux.debug
|
The -S argument above is simply an alternative to
entering the default /boot/System.map string.
Similarly, when looking at a dumpfile, two arguments are required
to describe the dumped kernel, along with the vmcore image:
# crash vmlinux vmlinux.dbg vmcore
(or)
# crash /boot/System.map vmlinux.dbg vmcore
(or)
# crash -S vmlinux.dbg vmcore
|
Again, for a detailed example of how to rebuild a Red Hat Advanced Server 2.1 kernel
with the -g flag, refer to these
directions.
Invocation output
The arguments may be entered in any order. If the file arguments are not
in the current directory, absolute pathnames must be used.
When in doubt, simply enter crash -h to get an explanation of
the command line arguments:
# crash -h
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
[namelist]
The [namelist] argument is a pathname to an uncompressed kernel image
(a vmlinux file) that has been compiled with the "-g" switch, or
that has an accessible, associated, debuginfo file. If the [dumpfile]
argument is entered, then the [namelist] argument must be entered
If the [namelist] argument is not entered when running on a live
system, a search will be made in several typical directories for
for a kernel namelist file that matches the live system.
[dumpfile]
The [dumpfile] argument is a pathname to a kernel memory core dump
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional
[mapfile] argument is required. The [mapfile] argument may consist
of either the associated System.map file, or the non-debug kernel
namelist. However, if the [mapfile] argument is used, then the
[namelist] argument must be a kernel namelist of a similar kernel
version that was built with the -g switch.
[-S]
Use "/boot/System.map" as the [mapfile].
Examples when running on a live system:
$ crash
$ crash /usr/tmp/vmlinux
$ crash /boot/System.map vmlinux.dbg
$ crash -S vmlinux.dbg
$ crash vmlinux vmlinux.dbg
Examples when running on a dumpfile:
$ crash vmlinux vmcore
$ crash /boot/System.map vmlinux.dbg vmcore
$ crash -S vmlinux.dbg vmcore
$ crash vmlinux vmlinux.dbg vmcore
[-h [opt]]
The -h option alone displays this message. If the [opt] argument is
a crash command name, the help page for that command is displayed. If
the string "input" is entered, a page describing the various crash
command line input options is displayed. If the string "output" is
entered, a page describing command line output options is displayed.
[-v]
Display the versions of crash and gdb making up this executable.
[-s]
Do not display any version, GPL, or crash initialization data; proceed
directly to the "crash>" prompt.
[-i file]
Execute the crash command(s) in [file] prior to accepting any user
input from the "crash>" prompt.
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
|
Given that all invocation arguments are in order, here is an example of
a successful invocation on a dumpfile, running a kernel that was built
with -g, along with
a vmcore dump file was
created by the Red Hat Netdump facility:
# crash vmlinux-2.4.20-2.1.15.entsmp vmcore
crash 3.7-1
Copyright (C) 2002, 2003 Red Hat, Inc.
Copyright (C) 1998-2003 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux-2.4.20-2.1.15.entsmp
DUMPFILE: vmcore
CPUS: 1
DATE: Wed Mar 12 10:12:56 2003
UPTIME: 00:38:25
LOAD AVERAGE: 1.16, 0.74, 0.30
TASKS: 60
NODENAME: dhcp64-220.boston.redhat.com
RELEASE: 2.4.20-2.1.15.entsmp
VERSION: #1 SMP Tue Mar 11 16:12:22 EST 2003
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 0
COMMAND: "swapper"
TASK: c038e000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
This next example shows the output when the panicking kernel was not
built with -g. In this case, a similar kernel type was built
with -g, and the resultant kernel object file was
renamed as vmlinux.dbg. Note that there will be a message
concerning the patching of gdb data; this indicates that
the non-matching symbol values from the vmlinux.dbg are being
over-written by the correct symbol values found in the original vmlinux file:
# crash vmlinux vmlinux.dbg vmcore
crash 3.7-1
Copyright (C) 2002, 2003 Red Hat, Inc.
Copyright (C) 1998-2003 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
please wait... (patching 16053 gdb minimal_symbol values)
DEBUG KERNEL: vmlinux.dbg
DUMPFILE: vmcore
CPUS: 1
DATE: Wed Mar 27 11:02:31 2002
UPTIME: 00:07:24
LOAD AVERAGE: 0.43, 0.42, 0.19
TASKS: 68
NODENAME: anderson.boston.redhat.com
RELEASE: 2.4.9-26beta.48enterprise
VERSION: #1 SMP Thu Mar 21 12:33:05 EST 2002
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
Invocation on a live system looks essentially the same, except that
the DUMPFILE will be indicated as /dev/mem. In the
following example, no arguments were entered, because the running
RHEL 3 kernel was found in the /boot directory, and its
associated debuginfo file in the /usr/lib/debug/boot directory.
The debuginfo file is listed next to the DEBUGINFO tag:
# crash
crash 3.7-1
Copyright (C) 2002, 2003 Red Hat, Inc.
Copyright (C) 1998-2003 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb Red Hat Linux (5.3post-0.20021129.36rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /boot/vmlinux-2.4.21-4.ELhugemem
DEBUGINFO: /usr/lib/debug/boot/vmlinux-2.4.21-4.ELhugemem.debug
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Aug 21 11:24:38 2003
UPTIME: 1 days, 23:14:11
LOAD AVERAGE: 0.14, 0.10, 0.08
TASKS: 106
NODENAME: crash.boston.redhat.com
RELEASE: 2.4.21-4.ELhugemem
VERSION: #1 SMP Wed Aug 13 21:33:31 EDT 2003
MACHINE: i686 (1993 Mhz)
MEMORY: 511.5 MB
PID: 4757
COMMAND: "crash"
TASK: 19b82000
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash>
|
Invocation Errors
Invocation errors will cause the crash session to abort upon
initialization. Typically they occur as the result of one of the following
reasons:
- The vmlinux file contains no debug data (i.e., was built
without the -g flag), and no additional debug kernel object file
name was entered on the command line. The error message will be of
the form:
crash: /boot/vmlinux-2.4.18-14: no debugging data available
|
- The vmlinux file does not match the dumpfile. The error
message will be of the form:
crash: vmlinux and tmp/vmcore do not match!
|
- The vmlinux file could not be found on a live system.
The error message will be of the form:
crash: cannot find booted kernel -- please enter namelist argument
|
-
The associated debuginfo file cannot be found.
The error message will be of the form:
crash: /boot/vmlinux-2.4.21-4.ELsmp: no debugging data available
crash: vmlinux-2.4.21-4.ELsmp.debug: debuginfo file not found
|
|
