# crash

## NAME



           crash - Analyze Linux crash dump data or a live system
    


## SYNOPSIS



           **crash**[_OPTION_]... _NAMELIST MEMORY-IMAGE[@ADDRESS]_    _(dumpfile_
           _form)_
           **crash**[_OPTION_]... _[NAMELIST]_                         _(live system_
           _form)_
    


## DESCRIPTION



           **Crash** is a tool for interactively analyzing the state of the Linux
           system while it is running, or after a kernel crash has occurred
           and a core dump has been created by the _netdump, diskdump, LKCD,_
           _kdump, xendump kvmdump_ or _VMware_ facilities.  It is loosely based
           on the SVR4 UNIX crash command, but has been significantly
           enhanced by completely merging it with the gdb(1) debugger. The
           marriage of the two effectively combines the kernel-specific
           nature of the traditional UNIX crash utility with the source code
           level debugging capabilities of **gdb(1).**
    
           In the _dumpfile form,_ both a NAMELIST and a MEMORY-IMAGE argument
           must be entered.  In the _live system form,_ the NAMELIST argument
           must be entered if the kernel's _vmlinux_ file is not located in a
           known location, such as the _/usr/lib/debug/lib/modules/ <kernel-_
           _version >_ directory.
    
           The **crash** utility has also been extended to support the analysis
           of dumpfiles generated by a crash of the Xen hypervisor.  In that
           case, the NAMELIST argument must be that of the _xen-syms_ binary.
           Live system analysis is not supported for the Xen hypervisor.
    
           The **crash** utility command set consists of common kernel core
           analysis tools such as kernel stack back traces of all processes,
           source code disassembly, formatted kernel structure and variable
           displays, virtual memory data, dumps of linked-lists, etc., along
           with several commands that delve deeper into specific kernel
           subsystems.  Appropriate **gdb** commands may also be entered, which
           in turn are passed on to the **gdb** module for execution.  If
           desired, commands may be placed in either a _$HOME/.crashrc_ file
           and/or in a _.crashrc_ file in the current directory.  During
           initialization, the commands in _$HOME/.crashrc_ are executed first,
           followed by those in the _./.crashrc_ file.
    
           The **crash** utility is designed to be independent of Linux version
           dependencies. When new kernel source code impacts the correct
           functionality of **crash** and its command set, the utility will be
           updated to recognize new kernel code changes, while maintaining
           backwards compatibility with earlier releases.
    


## OPTIONS



           **NAMELIST**
                  This is a pathname to an uncompressed kernel image (a
                  _vmlinux_ file), or a Xen hypervisor image (a _xen-syms_ file)
                  which has been compiled with the "-g" option.  If using the
                  _dumpfile form,_ a _vmlinux_ file may be compressed in either
                  gzip or bzip2 formats.
    
           **MEMORY-IMAGE[@ADDRESS]**
                  A kernel core dump file created by the _netdump, diskdump,_
                  _LKCD kdump, xendump kvmdump_ or _VMware_ facilities.
    
                  If a MEMORY-IMAGE argument is not entered, the session will
                  be invoked on the live system, which typically requires
                  root privileges because of the device file used to access
                  system RAM.  By default, _/dev/crash_ will be used if it
                  exists.  If it does not exist, then _/dev/mem_ will be used;
                  but if the kernel has been configured with
                  **CONFIG_STRICT_DEVMEM,** then _/proc/kcore_ will be used.  It is
                  permissible to explicitly enter _/dev/crash, /dev/mem_ or
                  _/proc/kcore._
    
                  An @ADDRESS value must be appended to the MEMORY-IMAGE if
                  the dumpfile is a raw RAM dumpfile that has no header
                  information describing the file contents.  Multiple MEMORY-
                  IMAGE@ADDRESS ordered pairs may be entered, with each
                  dumpfile containing a contiguous block of RAM, where the
                  ADDRESS value is the physical start address of the block
                  expressed in hexadecimal.  The physical address value(s)
                  will be used to create a temporary ELF header in /var/tmp,
                  which will only exist during the crash session.  If a raw
                  RAM dumpile represents a live memory source, such as that
                  specified by the QEMU mem-path argument of a memory-
                  backend-file object, then "live:" must be prepended to the
                  MEMORY-IMAGE name.
    
                  As VMware facility, the **crash** utility is able to process
                  VMware VM memory dump generated by VM suspend or guest core
                  dump. In that case, .vmss or .guest file should be used as
                  a MEMORY-IMAGE and .vmem file must be located in the same
                  folder.
    
           **mapfile**
                  If the NAMELIST file is not the same kernel that is running
                  (live system form), or the kernel that was running when the
                  system crashed (dumpfile form), then the _System.map_ file of
                  the original kernel should be entered on the command line.
    
           **-h** _[option]_
           **--help** _[option]_
                  Without an _option_ argument, display a **crash** usage help
                  message.  If the _option_ argument is a **crash** command name,
                  the help page for that command is displayed.  If it is the
                  string "input", a page describing the various **crash** command
                  line input options is displayed.  If it is the string
                  "output", a page describing command line output options is
                  displayed.  If it is the string "all", then all of the
                  possible help messages are displayed.  After the help
                  message is displayed, **crash** exits.
    
           **-s** Silently proceed directly to the "crash>" prompt without
                  displaying any version, GPL, or **crash** initialization data
                  during startup, and by default, runtime command output is
                  not passed to any scrolling command.
    
           **-i** _file_
                  Execute the command(s) contained in _file_ prior to
                  displaying the "crash>" prompt for interactive user input.
    
           **-d** _num_ Set the internal debug level.  The higher the number, the
                  more debugging data will be printed when **crash** initializes
                  and runs.
    
           **-S** Use _/boot/System.map_ as the _mapfile_.
    
           **-e** _vi | emacs_
                  Set the **readline(3)** command line editing mode to "vi" or
                  "emacs".  The default editing mode is "vi".
    
           **-f** Force the usage of a compressed _vmlinux_ file if its
                  original name does not start with "vmlinux".
    
           **-k** Indicate that the NAMELIST file is an LKCD "Kerntypes"
                  debuginfo file.
    
           **-g** _[namelist]_
                  Determine if a _vmlinux_ or _xen-syms_ namelist file contains
                  debugging data.
    
           **-t** Display the system-crash timestamp and exit.
    
           **-L** Attempt to lock all of its virtual address space into
                  memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) during
                  initialization.  If the system call fails, an error message
                  will be displayed, but the session continues.
    
           **-c** _tty-device_
                  Open the _tty-device_ as the console used for debug messages.
    
           **-p** _page-size_
                  If a processor's page size cannot be determined by the
                  dumpfile, and the processor default cannot be used, use
                  _page-size._
    
           **-o** _filename_
                  Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM
                  dumpfiles, specifies a filename of a new ELF vmcore that
                  will be created and used as the dumpfile.  It will be saved
                  to allow future use as a standalone vmcore, replacing the
                  original raw RAM dumpfile.
    
           **-m** _option=value_
           **--machdep** _option=value_
                  Pass an option and value pair to machine-dependent code.
                  These architecture-specific option/pairs should only be
                  required in very rare circumstances:
    
                  X86_64:
                    phys_base=<physical-address>
                    irq_eframe_link=<value>
                    irq_stack_gap=<value>
                    max_physmem_bits=<value>
                    kernel_image_size=<value>
                    vm=orig       (pre-2.6.11 virtual memory address ranges)
                    vm=2.6.11     (2.6.11 and later virtual memory address ranges)
                    vm=xen        (Xen kernel virtual memory address ranges)
                    vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
                    vm=5level     (5-level page tables)
                    page_offset=<PAGE_OFFSET-value>
                  PPC64:
                    vm=orig
                    vm=2.6.14     (4-level page tables)
                  IA64:
                    phys_start=<physical-address>
                    init_stack_size=<size>
                    vm=4l         (4-level page tables)
                  ARM:
                    phys_base=<physical-address>
                  ARM64:
                    phys_offset=<physical-address>
                    kimage_voffset=<kimage_voffset-value>
                    max_physmem_bits=<value>
                    vabits_actual=<value>
                  X86:
                    page_offset=<CONFIG_PAGE_OFFSET-value>
    
           **-x** Automatically load extension modules from a particular
                  directory.  If a directory is specified in the
                  **CRASH_EXTENSIONS** shell environment variable, then that
                  directory will be used.  Otherwise
                  _/usr/lib64/crash/extensions_ (64-bit architectures) or
                  _/usr/lib/crash/extensions_ (32-bit architectures) will be
                  used; if they do not exist, then the _./extensions directory_
                  _will be used._
    
           **--active**
                  Track only the active task on each cpu.
    
           **--buildinfo**
                  Display the crash binary's build date, the user ID of the
                  builder, the hostname of the machine where the build was
                  done, the target architecture, the version number, and the
                  compiler version.
    
           **--memory_module** _modname_
                  Use the _modname_ as an alternative kernel module to the
                  _crash.ko_ module that creates the _/dev/crash_ device.
    
           **--memory_device** _device_
                  Use _device_ as an alternative device to the _/dev/crash,_
                  _/dev/mem_ or _/proc/kcore_ devices.
    
           **--log** _dumpfile_
                  Dump the contents of the kernel log buffer.  A kernel
                  namelist argument is not necessary, but the dumpfile must
                  contain the VMCOREINFO data taken from the original
                  /proc/vmcore ELF header.  Note: this option is deprecated
                  and will no longer work for kernel(>=v5.10).
    
           **--no_kallsyms**
                  Do not use kallsyms-generated symbol information contained
                  within kernel module object files.
    
           **--no_modules**
                  Do not access or display any kernel module related
                  information.
    
           **--no_ikconf**
                  Do not attempt to read configuration data that was built
                  into kernels configured with **CONFIG_IKCONFIG.**
    
           **--no_data_debug**
                  Do not verify the validity of all structure member offsets
                  and structure sizes that it uses.
    
           **--no_kmem_cache**
                  Do not initialize the kernel's slab cache infrastructure,
                  and commands that use kmem_cache-related data will not
                  work.
    
           **--no_elf_notes**
                  Do not use the registers from the ELF NT_PRSTATUS notes
                  saved in a compressed kdump header for backtraces.
    
           **--kmem_cache_delay**
                  Delay the initialization of the kernel's slab cache
                  infrastructure until it is required by a run-time command.
    
           **--readnow**
                  Pass this flag to the embedded **gdb** module, which will
                  override its two-stage strategy that it uses for reading
                  symbol tables from the NAMELIST.
    
           **--smp** Specify that the system being analyzed is an SMP kernel.
    
           **-v**
           **--version**
                  Display the version of the **crash** utility, the version of
                  the embedded **gdb** module, GPL information, and copyright
                  notices.
    
           **--cpus** _number_
                  Specify the _number_ of cpus in the SMP system being
                  analyzed.
    
           **--osrelease** _dumpfile_
                  Display the OSRELEASE vmcoreinfo string from a kdump
                  _dumpfile_ header.
    
           **--hyper**
                  Force the session to be that of a Xen hypervisor.
    
           **--p2m_mfn** _pfn_
                  When a Xen Hypervisor or its dom0 kernel crashes, the
                  dumpfile is typically analyzed with either the Xen
                  hypervisor or the dom0 kernel.  It is also possible to
                  analyze any of the guest domU kernels if the
                  pfn_to_mfn_list_list _pfn_ value of the guest kernel is
                  passed on the command line along with its NAMELIST and the
                  dumpfile.
    
           **--xen_phys_start** _physical-address_
                  Supply the base physical address of the Xen hypervisor's
                  text and static data for older xendump dumpfiles that did
                  not pass that information in the dumpfile header.
    
           **--zero_excluded**
                  If the makedumpfile(8) facility has filtered a compressed
                  kdump dumpfile to exclude various types of non-essential
                  pages, or has marked a compressed or ELF kdump dumpfile as
                  incomplete due to an ENOSPC or other error during its
                  creation, any attempt to read missing pages will fail.
                  With this flag, reads from any of those pages will return
                  zero-filled memory.
    
           **--no_panic**
                  Do not attempt to find the task that was running when the
                  kernel crashed.  Set the initial context to that of the
                  "swapper" task on cpu 0.
    
           **--more** Use _/bin/more_ as the command output scroller, overriding
                  the default of _/usr/bin/less_ and any settings in either
                  _./.crashrc_ or _$HOME/.crashrc._
    
           **--less** Use _/usr/bin/less_ as the command output scroller,
                  overriding any settings in either _./.crashrc_ or
                  _$HOME/.crashrc._
    
           **--hex** Set the default command output radix to 16, overriding the
                  default radix of 10, and any radix settings in either
                  _./.crashrc_ or _$HOME/.crashrc._
    
           **--dec** Set the default command output radix to 10, overriding any
                  radix settings in either _./.crashrc_ or _$HOME/.crashrc. This_
                  _is the default radix setting._
    
           **--CRASHPAGER**
                  Use the output paging command defined in the **CRASHPAGER**
                  shell environment variable, overriding any settings in
                  either _./.crashrc_ or _$HOME/.crashrc._
    
           **--no_scroll**
                  Do not pass run-time command output to any scrolling
                  command.
    
           **--no_strip**
                  Do not strip cloned kernel text symbol names.
    
           **--no_crashrc**
                  Do not execute the commands in either _$HOME/.crashrc_ or
                  _./.crashrc._
    
           **--mod** _directory_
                  When loading the debuginfo data of kernel modules with the
                  _mod -S_ command, search for their object files in _directory_
                  instead of in the standard location.
    
           **--src** _directory_
                  Search for the kernel source code in directory instead of
                  in the standard location that is compiled into the
                  debuginfo data.
    
           **--kaslr** _offset_**|**_auto_
                  If an x86, x86_64, s390x or loongarch64 kernel was
                  configured with **CONFIG_RANDOMIZE_BASE,** the offset value is
                  equal to the difference between the symbol values compiled
                  into the vmlinux file and their relocated KASLR values.  If
                  set to auto, the KASLR offset value will be automatically
                  calculated.
    
           **--reloc** _size_
                  When analyzing live x86 kernels that were configured with a
                  **CONFIG_PHYSICAL_START** value that is larger than its
                  **CONFIG_PHYSICAL_ALIGN** value, then it will be necessary to
                  enter a relocation size equal to the difference between the
                  two values.
    
           **--hash** _count_
                  Set the number of internal hash queue heads used for list
                  gathering and verification.  The default count is 32768.
    
           **--minimal**
                  Bring up a session that is restricted to the _log, dis, rd,_
                  _sym, eval, set_ and _exit_ commands.  This option may provide
                  a way to extract some minimal/quick information from a
                  corrupted or truncated dumpfile, or in situations where one
                  of the several kernel subsystem initialization routines
                  would abort the **crash** session.
    
           **--kvmhost** _[32|64]_
                  When examining an x86 KVM guest dumpfile, this option
                  specifies that the KVM host that created the dumpfile was
                  an x86 (32-bit) or an x86_64 (64-bit) machine, overriding
                  the automatically determined value.
    
           **--kvmio** _< size>_
                  override the automatically-calculated KVM guest I/O hole
                  size.
    
           **--offline** _[show|hide]_
                  Show or hide command output that is related to offline
                  cpus.  The default setting is show.
    


## COMMANDS



           Each **crash** command generally falls into one of the following
           categories:
    
           _Symbolic display_
                  Displays of kernel text/data, which take full advantage of
                  the power of **gdb** to format and display data structures
                  symbolically.
    
           _System state_
                  The majority of **crash** commands consist of a set of "kernel-
                  aware" commands, which delve into various kernel subsystems
                  on a system-wide or per-task basis.
    
           _Utility functions_
                  A set of useful helper commands serving various purposes,
                  some simple, others quite powerful.
    
           _Session control_
                  Commands that control the **crash** session itself.
    
           The following alphabetical list consists of a very simple overview
           of each **crash** command.  However, since individual commands often
           have several options resulting in significantly different output,
           it is suggested that the full description of each command be
           viewed by executing _crash -h <command>,_ or during a **crash** session
           by simply entering _help command_**.**
    
           _*_      "pointer to" is shorthand for either the _struct_ or _union_
                  commands.  It displays the contents of a kernel structure
                  or union.
    
           _alias_  creates a single-word alias for a command.
    
           _ascii_  displays an ascii chart or translates a numeric value into
                  its ascii components.
    
           _bpf_    provides information on currently-loaded eBPF programs and
                  maps.
    
           _bt_     displays a task's kernel-stack backtrace.  If it is given
                  the _-a_ option, it displays the stack traces of the active
                  tasks on all CPUs.  It is often used with the _foreach_
                  command to display the backtraces of all tasks with one
                  command.
    
           _btop_   translates a byte value (physical offset) to its page
                  number.
    
           _dev_    displays data concerning the character and block device
                  assignments, I/O port usage, I/O memory usage, and PCI
                  device data.
    
           _dis_    disassembles memory, either entire kernel functions, from a
                  location for a specified number of instructions, or from
                  the start of a function up to a specified memory location.
    
           _eval_   evaluates an expression or numeric type and displays the
                  result in hexadecimal, decimal, octal and binary.
    
           _exit_   causes **crash** to exit.
    
           _extend_ dynamically loads or unloads **crash** shared object extension
                  modules.
    
           _files_  displays information about open files in a context.
    
           _foreach_
                  repeats a specified command for the specified (or all)
                  tasks in the system.
    
           _fuser_  displays the tasks using the specified file or socket.
    
           _gdb_    passes its argument to the embedded **gdb** module.  It is
                  useful for executing **gdb** commands that have the same name
                  as **crash** commands.
    
           _help_   alone displays the command menu; if followed by a command
                  name, a full description of a command, its options, and
                  examples are displayed.  Its output is far more complete
                  and useful than this man page.
    
           _ipcs_   displays data about the System V IPC facilities.
    
           _irq_    displays data concerning interrupt request numbers and
                  bottom-half interrupt handling.
    
           _kmem_   displays information about the use of kernel memory.
    
           _list_   displays the contents of a linked list.
    
           _log_    displays the kernel log_buf contents in chronological
                  order.
    
           _mach_   displays data specific to the machine type.
    
           _mod_    displays information about the currently installed kernel
                  modules, or adds or deletes symbolic or debugging
                  information about specified kernel modules.
    
           _mount_  displays information about the currently-mounted
                  filesystems.
    
           _net_    display various network related data.
    
           _p_      passes its arguments to the **gdb** "print" command for
                  evaluation and display.
    
           _ps_     displays process status for specified, or all, processes in
                  the system.
    
           _pte_    translates the hexadecimal contents of a PTE into its
                  physical page address and page bit settings.
    
           _ptob_   translates a page frame number to its byte value.
    
           _ptov_   translates a hexadecimal physical address into a kernel
                  virtual address.
    
           _q_      is an alias for the "exit" command.
    
           _rd_     displays the contents of memory, with the output formatted
                  in several different manners.
    
           _repeat_ repeats a command indefinitely, optionally delaying a given
                  number of seconds between each command execution.
    
           _runq_   displays the tasks on the run queue.
    
           _sbitmapq_
                  dumps the contents of the sbitmap_queue structure and the
                  used bits in the bitmap. Also, it shows the dump of a
                  structure array associated with the sbitmap_queue.
    
           _search_ searches a range of user or kernel memory space for given
                  value.
    
           _set_    either sets a new context, or gets the current context for
                  display.
    
           _sig_    displays signal-handling data of one or more tasks.
    
           _struct_ displays either a structure definition or the contents of a
                  kernel structure at a specified address.
    
           _swap_   displays information about each configured swap device.
    
           _sym_    translates a symbol to its virtual address, or a static
                  kernel virtual address to its symbol -- or to a symbol-
                  plus-offset value, if appropriate.
    
           _sys_    displays system-specific data.
    
           _task_   displays the contents of a task_struct.
    
           _tree_   displays the contents of a red-black tree or a radix tree.
    
           _timer_  displays the timer queue entries, both old- and new-style,
                  in chronological order.
    
           _union_  is similar to the _struct_ command, except that it works on
                  kernel unions.
    
           _vm_     displays basic virtual memory information of a context.
    
           _vtop_   translates a user or kernel virtual address to its physical
                  address.
    
           _waitq_  walks the wait queue list displaying the tasks which are
                  blocked on the specified wait queue.
    
           _whatis_ displays the definition of structures, unions, typedefs or
                  text/data symbols.
    
           _wr_     modifies the contents of memory on a live system.  It can
                  only be used if _/dev/mem_ is the device file being used to
                  access system RAM, and should obviously be used with great
                  care.
    
           When **crash** is invoked with a Xen hypervisor binary as the
           NAMELIST, the command set is slightly modified.  The _*, alias,_
           _ascii, bt, dis, eval, exit, extend, gdb, help, list, log, p, pte,_
           _rd, repeat, search, set, struct, sym, sys, union, whatis, wr_ and _q_
           commands are the same as above.  The following commands are
           specific to the Xen hypervisor:
    
           _domain_ displays the contents of the domain structure for selected,
                  or all, domains.
    
           _doms_   displays domain status for selected, or all, domains.
    
           _dumpinfo_
                  displays Xen dump information for selected, or all, cpus.
    
           _pcpus_  displays physical cpu information for selected, or all,
                  cpus.
    
           _vcpus_  displays vcpu status for selected, or all, vcpus.
    


## FILES



           _.crashrc_
                  Initialization commands.  The file can be located in the
                  user's **HOME** directory and/or the current directory.
                  Commands found in the _.crashrc_ file in the **HOME** directory
                  are executed before those in the current directory's
                  _.crashrc_ file.
    


## ENVIRONMENT



           **EDITOR** Command input is read using readline(3).  If **EDITOR** is set
                  to _emacs_ or _vi_ then suitable keybindings are used.  If
                  **EDITOR** is not set, then _vi_ is used.  This can be overridden
                  by **set vi** or **set emacs** commands located in a _.crashrc_ file,
                  or by entering **-e emacs** on the **crash** command line.
    
           **CRASHPAGER**
                  If **CRASHPAGER** is set, its value is used as the name of the
                  program to which command output will be sent.  If not, then
                  command output is sent to **/usr/bin/less -E -X** by default.
    
           **CRASH_MODULE_PATH**
                  Specifies an alternative directory tree to search for
                  kernel module object files.
    
           **CRASH_EXTENSIONS**
                  Specifies a directory containing extension modules that
                  will be loaded automatically if the **-x** command line option
                  is used.
    


## NOTES



           If **crash** does not work, look for a newer version: kernel evolution
           frequently makes **crash** updates necessary.
    
           The command **set scroll off** will cause output to be sent directly
           to the terminal rather than through a paging program.  This is
           useful, for example, if you are running **crash** in a window of
           **emacs**.
    


* * *