Each crash command generally falls into one of the
following categories:
The remainder
of this section breaks the command set into categories, and gives a short
description of each command in that category.
However, for complete details and examples,
recall that the crash utility has a
self-contained help page for each command; to view the full help page,
click on the command name next to its description below.
Symbolic Display of Kernel Text or Data
The following commands typically take full advantage of the power
of gdb to display kernel data structures symbolically.
Command |
|
Description
|
struct |
|
Displays a formatted kernel data structure type located at a
given address, or at an address referred to by a symbol; if no address
is specified, the structure definition is displayed. The output can
be narrowed down to a singular member of the structure, or
to display the offset of every member from the beginning
of the structure. A count may be appended to display an array of structures.
Its usage is so common that two short-cuts exist such that the user
need not enter the "struct" command name:
- The "pointer-to" * command below can be substituted.
- If a structure name is entered as the first token on a command line,
the "struct" command is actually not necessary.
|
union |
|
Same as struct command, but used for kernel data types
defined as unions instead of structures..
|
* |
|
"Pointer-to" command wich can be used in lieu of entering struct
or union; the gdb module first determines whether the
argument is a structure or a union, and then calls the appropriate function.
|
p |
|
Displays the contents of a kernel variable; the arguments
are passed on to gdb's print command for proper formatting.
Two builtin aliases, px and pd, set the numerical
output radix to hexadecimal or decimal for the print operation, temporarily
overriding the current default.
|
whatis |
|
Displays all available symbol table information concerning a data type
or a data symbol.
|
sym |
|
Translates a kernel symbol name to its kernel virtual address and section,
or a kernel virtual address to its symbol name and section. It can also
be used to dump the complete list of kernel symbols, or to query the symbol
list for all symbols containing a given sub-string.
|
dis |
|
Disassembles the text of complete kernel function, or from a specified
address for a given number of instructions, or from the beginning of a
function up to a specified address.
|
System State
The majority of crash commands come from the following set of
"kernel-aware" commands, which delve into various kernel subsystems on
a system-wide or per-task basis. The task-specific commands are
context-sensitive, meaning that they act upon the current context unless
a PID or task address is specified as an argument.
Command |
|
Description
|
bt |
|
Arguably the most useful crash command, bt displays a
task's kernel stack
back-trace, including full exception frame dumps. It is context-sensitive,
although the -a option will display the stack
traces of the active task on each CPU. This command is often used within
the foreach wrapper command
in order to display the back traces of all tasks with one command.
|
dev |
|
Displays data concerning the character and block device assignments,
I/O port usage, I/O memory usage, and PCI device data.
|
files |
|
This context-sensitive command displays the task's current root directory
and working directories, and then for each open file descriptor, shows:
- its file struct address
- its dentry struct address
- its inode struct address
- the file type
- the file's full pathname
Another option acts upon a specified dentry address, showing:
- its inode struct address
- its superblock struct address
- the file type
- the file's full pathname
It can be called from the
foreach wrapper command.
|
fuser |
|
Displays a list of tasks that reference a specified filename
or inode address as the current root or working directory,
an open file descriptor, or which mmap the file.
|
irq |
|
Display data concerning interrupt request numbers and bottom-half handling.
|
kmem |
|
This command has numerous options that delve into the state of
several kernel memory subsystems:
- general memory usage, similar in scope to /proc/meminfo
- kmalloc slab memory allocator, including an option
that lists each slab object and its state, verifying the slab chain
- display and verification of free page lists
- vmalloc memory allocator vmlist contents
- display and verification of the page cache
- the mem_map page list
- display NUMA data, if applicable
Also, given an address, this command searches the symbol table,
the slab subsystem, the free list, the page_hash_table,
the vmlist, and the mem_map array,
displaying where it was found.
|
log |
|
Dumps the kernel message buffer chronologically, accounting for
any wrap-around.
|
mach |
|
Displays machine and/or processor specific data.
|
mod |
|
Displays the list of currently-loaded kernel modules. More importantly,
it loads the debug data from the module object files if they are
available, allowing symbolic debugging capability
of kernel modules.
|
mount |
|
For each mounted filesystem, or for just a specified filesystem, displays:
- its vfsmount struct address
- its super_block struct address
- its type
- its device name
- its mount point
Options exist to dump a list of a specified filesystem's open files or dirty inodes. Filesystems may be specified by
vfsmount, super_block, or inode addresses, or
by device name or mount point names.
|
net |
|
Displays various network-related data:
- displays each configured network
device's net_device address, its name, and IP address
- displays the ARP cache
- context-sensitive display of information concerning the open sockets
of a task
- translates an IP address expressed as a decimal or
hexadecimal
value into a standard numbers-and-dots notation
It can be called from the
foreach wrapper command.
|
ps |
|
Useful process status command, in typical Linux ps command
type output, containing:
- PID number
- PPID number
- CPU number
- task address
- process state
- percent of physical memory consumed
- virtual address size
- resident set size
- command name
Also has an option to show a task's parental hierarchy back to the
init process, and another to show all children of a task.
|
pte |
|
This command translates the contents of a PTE into its physical
page address and page bit settings, or if it references a swap location,
the swap device and offset.
|
runq |
|
Displays list of tasks on the run queue.
|
sig |
|
A context-sensitive command which displays a task's signal
information, including:
-
whether an unblocked signal is pending
- the pending and blocked signals
- the handler data for each signal
- queued signals, if any
Other options list the signal number/names combination for a processor type,
and translate the contents of a sigset_t into the signal names
whose bits are set.
It can be called from the
foreach wrapper command.
|
swap |
|
For each configured swap device, this command displays
the same data that is shown by the Linux command swapon -s.
|
sys |
|
Re-displays the same system-related data that is seen during
crash initialization:
- the kernel object filename
- the dumpfile name
- the number of CPUS
- the date
- system uptime
- system load average
- the number of tasks
- the nodename
- the kernel release and version data
- the processor type and speed
- the amount of memory
- the panic string
Other options display information concerning the system call table,
and one allows the root user to panic a live system.
|
task |
|
This context-sensitive command displays a task's complete task_struct
contents, or one or more members of the structure.
This command is often used within the
foreach wrapper command
in order to display task_struct data for all tasks with one
command.
|
timer |
|
Displays the timer queue entries in chronological order,
listing the target function names, the current value of jiffies,
and the expiration time of each entry.
|
vm |
|
This powerful, context-sensitive command displays a wealth of information
concerning a task's virtual memory data, including:
- its mm_struct address
- its page directory address
- its resident set size
- its total virtual memory size
- each vm_area_struct address, along with
its start and ending virtual
address, flags, and source file if applicable.
- optionally, every virtual page referenced by a
vm_area_struct can be translated into its
physical address, or if not resident, its file and
offset.
Other options tranlate the flags of a vm_area_struct,
or display the full contents of a task's mm_struct
or of each vm_area_struct.
It can be called from the
foreach wrapper command.
|
vtop |
|
This context-sensitive command translates a user or kernel virtual address
to its physical address. Also displayed are:
- the full PTE translation from page directory through to
the page table
- the vm_area_struct data for user virtual addresses
- the mem_map page data associated with the physical page
- the swap location or file location if a user virtual page is not
currently mapped
It can be called from the
foreach wrapper command.
|
waitq |
|
Lists the tasks linked on a specified kernel wait queue.
|
Utility Functions
The following commands are a set of useful helper commands
serving various purposes, some simple, others quite powerful.
Command |
|
Description
|
ascii |
|
Translates a numerical value into its ASCII components; with no arguments,
displays an ASCII chart.
|
btop |
|
Translates a byte value (physical address) to its page number.
|
eval |
|
A simple calculator, evaluates an expression and displays the result
in hexadecimal, decimal, octal and binary, and optionally showing
the bit numbers set in the result.
|
list |
|
Dumps the entries of a linked list of structures.
It can handle lists of structures that are
singly-linked with simple "next" pointers, or those with
embedded list_head
structures. The output may be constrained to simply display the address of each
structure in the list, or if directed, also dump each complete structure,
or just one member of each structure. The gathered list entries are hashed,
so a corrupted list that loops back upon itself will be recognized.
|
ptob |
|
translates a page frame number to its byte value (physical address).
|
ptov |
|
Translates a physical address into a kernel virtual address by
adding the appropriate PAGE_OFFSET value.
|
search |
|
Searches a range of user or kernel memory space for given value, with
an optional "don't care" bit-mask argument.
|
rd |
|
Displays a specified amount of user virtual, kernel virtual, or physical memory
in several formats, such as 8, 16, 32 or 64 byte values, hexadecimal or
decimal, symbolically, and with ASCII translations. When reading user
virtual addresses, the command is context-sensitive.
|
wr |
|
Modifies the contents of memory on a live system. Write permission
on /dev/mem is required; this command should obviously be used
with great care. The write operation is constrained to one 8, 16, 32
or 64 byte location.
|
Session Control Commands
The following commands typcally aid in the efficient
running of a crash session.
Command |
|
Description
|
alias |
|
Creates a single-word alias for a command string.
Several aliases are built into
crash; user-defined aliases may also be defined in
a .crashrc file, or during a crash session
by entering it on the command line or reading it from an input file.
|
exit |
|
Shuts down the crash session (same as q).
|
extend |
|
Extend the crash command set by
dynamically loading a shared object library containing one or more
user-written commands.
|
foreach |
|
Quite often it is helpful, or even necessary, to run the same crash
context-sensitive command on a number of tasks by just entering one command.
This wrapper command sets off the execution
of a given crash command on each of a defined set of tasks,
temporarily changing the current context to that of the targeted task
before running the command.
The set of tasks that are issued the given command can be defined by:
- one or more PID numbers
- one or more task numbers
- one or more command name
- all user tasks
- all kernel tasks
- the active task on each CPU
The identifiers above may be mixed if it makes sense, such as using a
combination of PIDs, task addresses, and command names.
The context-sensitive commands that can be issued to the selected tasks are:
A header containing the PID, task address, CPU and command name will be
pre-pended before the command output for each selected task.
|
gdb |
|
This command passes its arguments directly to gdb for processing.
This is typically not necessary, but where ambiguities between crash
and gdb command names exist, this will force the command to be
executed by gdb.
|
repeat |
|
This wrapper command repeats a crash command indefinitely, optionally
delaying a given number of seconds between each command execution.
Obviously this command is only useful when running on a live system.
|
set |
|
This primary purpose for this command is to set the crash context to
a new task, or to display the current context. It can also be used to view or
change one of a set of internal crash variables that modify program
behavior, such as the default output radix or scrolling behavior.
It can be called from the
foreach wrapper command for
viewing the context data of each task.
|
q |
|
Shuts down the crash session (same as
exit).
|
|