| By Adam Zhang, Sun Microsystems, April 2007, Updated June 2007 |
|
 |
Abstract: Abnormal termination of a process will trigger a
core dump file. A core dump file is very helpful to programmers or
support engineers for determining the root cause of abnormal
termination, because it provides invaluable information about the
runtime status at crash time. This article provides information about
core dumps, as well as features and analysis tools in the Solaris
Operating System that can be used to manage core dumps.
Note: The information provided in this article is mainly for the Solaris 10 OS.
Contents:
- Types of Core Dumps: Process and System
- Cause of Process Core Dumps
- How to Manage a Process Core Dump
- How to Create a Process Core Dump Manually
- How to Analyze a Process Core Dump File
- Cause of System Core Dumps
- How a System Core Dump Is Created
- How to Manage a System Core Dump
- How to Create a System Core Dump Manually
- How to Analyze a System Core Dump File
- References
|
A core dump is a file that records the contents of a process along with other useful information,
such as the processor register's value. There are two types of core dumps: system core dumps and process core dumps.
They differ in many aspects, such as the manner in which they are created and the method used to analyze them.
|
When an application process receives a specific signal and terminates, the system generates a core dump
and stops the process. In most cases, the signal leading to the application crash is SIGSEGV
or SIGBUS.
SIGSEGV indicates that the application is accessing an invalid memory address. This situation
often occurs in C/C++ programs if there are code errors in pointer manipulation.
On the Solaris OS, you can use the libumem(3LIB) library as the user-mode memory allocator
instead of libc. The libumem library can help find memory leaks, buffer overflows,
attempts to use freed data, and many other memory allocation errors. Also, its memory allocator is very fast
and scalable with multithreaded applications.
SIGBUS indicates that the application is accessing a memory address that does not conform to
CPU memory alignment rules. This usually happens to a system with an UltraSPARC processor. Systems with
x86/x64 CPUs can handle unaligned memory addresses, but there is a performance impact.
The Sun Studio C/C++ compiler has the -xmemalign option, which can be used to adjust
the behavior of the UltraSPARC CPU when there are unaligned memory addresses that can be determined at
compile time. The -xmemalign option causes the compiler to generate additional load/store
instructions for unaligned memory access. However, the -xmemalign option
cannot handle unaligned memory access during runtime. If unaligned memory access happens during runtime,
the developer needs to change the source code.
There are other signals whose default disposition is to create a core dump, for example, SIGFPE,
which indicates a floating point exception. The Signal(3HEAD) man page provides more details.
|
The Solaris OS attempts to create up to three core dump files for each abnormally terminated process.
One of the core dump files, which is called the per-process core file, is located in the current directory.
Another core dump file, which is called the global core file, is created in the system-wide location. If
the process is running in a local zone, a third core file is created in the global zone's location.
You can use the coreadm(1M) command to manage the core dumps. All the settings are saved
in the /etc/coreadm.conf configuration file.
Below is a typical scenario, which shows the current system configuration for core dumps:
-bash-3.00# coreadm |
In the previous output:
- The
global core dumps: disabledline indicates no global core dump will be generated. - The
per-process core dumps: enabledline indicates a per-process core dump will be generated
for each abnormal process. - The
init core file patternline indicates the contents will be gathered from the live process
to the per-process core dump.
You can also use the coreadm command to control the core dump file name:
-bash-3.00# coreadm -i core.%f.%p |
This command causes the per-process core file name to be appended with the program file name (%f)
and the runtime process ID (%p). A core dump file will be generated in the current working
directory of the process.
-bash-3.00# coreadm -g /globalcore/core.%f.%p -e global |
By default, the global core dump is disabled. You need to use the coreadm command with the
-e global option to enable it. The -g option causes the command to append
the program name (%f) and the runtime process ID (%p) to the core file name.
As indicated previously, coreadm can specify the parts of the process that will be saved
to the core file. Previously, when you performed a post-mortem analysis, you needed to obtain all
the specific versions of the dependent libraries and runtime modules, because the core dump file does not
contain this text information. It is quite a headache for programmers to recreate the environment from
the original machine.
With the default configuration, the Solaris OS applies the "default"
pattern to each process core dump, which means the process core dump
contains stack, heap, text, shared memory (SHM), intimate shared memory
(ISM), and dynamic intimate shared memory (DISM) information, plus
other information. The text part of the process core dump also contains
a partial symbol table (dynsm), which will help you get a readable
stack trace directly from one core file without any other boring
dependent libraries. If the dynsm is insufficient, you can use coreadm to include all symbol tables, as follows:
-bash-3.00# coreadm -G all -I all |
This previous command makes both the global core file (-G) and the per-process
core file (-I) contain all the parts of the process.
Here's how to use coreadm to verify the changes:
-bash-3.00# coreadm |
The coreadm command is used to edit the configuration file of the coreadm service, which is managed by the Service Management Facility (SMF) with this service identifier: svc:/system/coreadm:default.
|
The Solaris OS provides the gcore(1) command in case you need to create a core dump manually
for a live process for analysis purposes:
-bash-3.00# echo $$ |
The live process ID is appended automatically to the name of the generated core dump. In the previous example, the process
of the current shell is dumped and its process ID is 2770.
Note: There are other constraints you need take into account while generating the core dump,
for example, the write permissions on the destination directory, the existence of the destination directory,
the file system mount option, and process resource limitation. For resource limitation information, refer to the man pages for setrlimit(2) and ulimit(1).
Another useful tool called AppCrash
is available. It automatically collects diagnostic and debugging
information when any application crashes under the Solaris OS. This
article does not address its usage. For more information on using
AppCrash, refer to Greg Nakhimovsky's blog.
|
There are lots of tools in the Solaris OS for analyzing core dump files: dbx(1), mdb(1),
and pstack(1). The most convenient method is to use the pstack tool to determine the
process stack. This tool helps show multithreaded programs as well:
-bash-3.00# pstack core.2580 | more |
In general, if the program's symbol table is not stripped and its runtime stack trace is available, you can
expect almost 50 percent of the problems to be resolved.
dbx is a free source-level debug tool provided by Sun
Studio software. Sun Studio software includes free, optimizing C, C++,
and Fortran compilers that can be used on both the Solaris OS and
Linux. dbx not only helps you inspect the state of your
program, but it also collects the program performance data. Here is a
typical scenario for analyzing the core file using dbx. For more details on dbx, please refer to the document called Sun Studio 11: Debugging a Program With dbx.
-bash-3.00# /opt/SUNWspro/bin/dbx tServer core |
From the previous example, you can use dbx to determine the abnormal thread, which is marked
with "o," and its root cause by showing the source code. Of course, this will not happen unless you
provide the application source code and add debug information during the compile phase.
If you are familiar with assembly language and hardware specifications, you can use mdb to
debug the core file, because mdb is a low-level debugging utility for both programs and the Solaris OS.
|
There are lots of reasons why the Solaris OS might crash and produce a core dump. Not only software problems,
such as like drivers and programs, but also hardware errors can induce a system core dump.
|
When detecting whether the integrity of data was corrupted or whether a fatal error in hardware occurred, the
Solaris OS invokes panic(). The panic() routine interrupts all processes
as if the OS is suspended. Then it generates a system core dump, which is a copy of OS in the memory, and saves
it to the dump device. After a crash, the OS use savecore(1) to retrieve the core dump from
the dump device to the savecore directory during the next boot. The savecore routine
generates two files. One file is unix.<X>, which is an OS symbol table list, and the other is
vmcore.<X>, which is the core dump data file. By default, the dump device is a swap disk
partition and the savcore directory is set to /var/crash/<hostname>.
The trailing <X> in the file names is an integer that grows every time savecore runs.
|
You can use dumpadm(1M) to manage dump devices and the savecore directory:
-bash-3.00# dumpadm -d /dump -s /savecore |
To verify this or see the current configuration, you can run only dumpadm:
-bash-3.00# dumpadm |
You can also use dumpadm to set the dump content and enable savecore(1)
operation during the boot.
All the configuration information is saved in the /etc/dumpadm.conf configure file. The
system crash dump service is also managed by SMF with this service
identifier: svc:/system/dumpadm:default.
|
In some cases, you need to save a core dump manually to take a snapshot of the live system. In the
Solaris OS, there are several means you can use. For example, you can use reboot -d to force
the generation of a core dump with reboot. Or you can use savecore -L to create a
live OS core dump. If you want to use savecore(1M) to create a live core dump,
you must use dumpadm to set a non-swap device as the dump device, because live core dumps
take a swap device as a part of virtual memory, which is to be dumped.
Sometimes, the system will hang without crashing. If you are using a Sun UltraSPARC processor-based machine, you can
press Stop-A to run in OpenBoot PROM (OBP) mode, and then use the sync OBP command to
force a crash core dump.
On x86 platforms, there is no corresponding OBP unit. However, you can use kmdb(1M). To
use kmdb to create a core dump, you need load its module during system booting.
Here are the steps for the Solaris 10 1/06 OS or later.
- Edit the
/boot/grub/menu.lstfile and append the-kstring to the
initrdline, as follows: - Then restart the machine manually.
title Solaris 10 11/06 s10x_u3wos_10 X86 |
This will make the OS boot with kmdb.
Alternatively, if you are using the Solaris 10 GA OS, just enter b -k when you see the
Select (B)oot or (I)nterpreter: system prompt during the system boot stage.
After performing these steps, press F1-A to break the system to kmdb. This action must be performed in console mode,
because kmdb suspends the system and GUI applications. If you are using a desktop system, the Solaris OS
will fail to switch to console mode and your desktop will appear to hang. However kmdb is running
and you can still type commands.
$<systemdump |
The systemdump command generates the core dump file for you. The dump device and savecore
directory for this operation are still constrained by dumpadm.
Sometimes, the system will hang without any response even when you use kmdb or OBP. In this case,
use the "deadman timer." The deadman timer allows the OS to force a kernel panic in the event of a system hang.
This feature is available on x86 and SPARC systems. Add the following line to /etc/system and reboot
so the deadman timer will be enabled.
set snooping=1 |
The enabled deadman timer will perform a level 15 interrupt once a second. It will check whether the kernel
lbolt variable is updated. If the deadman timer notices that the lbolt variable has
not been incremented for a period of time (the default is 50 seconds), it will cause a panic. The period of time
can be configured in /etc/system. The following example makes the deadman timer wait 120 seconds
for the lbolt variable update:
set snoop_interval=120000000 |
Solaris Dynamic Tracing (DTrace) was introduced with the release of Solaris 10
OS. DTrace allows you to understand and explore applications or the operating
system. DTrace contains a feature called Anonymous Tracing. It provides device
driver developers with a way to debug and trace system activity that occurs
during the system boot. If the Solaris OS hangs, you can use this feature to
generate a core dump and catch other information you want. For the information
on using DTrace, refer to the DTrace HowTo Guide and the Solaris Dynamic Tracing
Guide.
|
This article cannot provide solutions for fixing a system core dump, because such an analysis requires
much low-level computing knowledge of the OS kernel and also of the hardware. However here are some basic guidelines
for your reference:
- Check the system console and the
/var/adm/messagesfile, because they contain valuable
information for identifying the problem that the system encountered. - Use the
strings(1)command to process the core dump file. This command prints out the
ASCII strings in any binary file, including a core dump file. You need to look at these ASCII strings. - Check the error you encounter on the SunSolve web site, or use the
free Solaris Crash Analysis Tool (CAT) to help you investigate further.
|
- Man pages for
signal(3HEAD) (base signals),coredmp(1M),dumpadm(1M), and
proc(1) - Sun Studio 12: Debugging a Program with dbx, on docs.sun.com
- Sun Info Doc number 13258 and Sun Info Doc number 15553 (Note: This content is available to registered SunSolve users with a valid Sun Service Plan.)
- Sun Studio web site
- Solaris Crash Analysis Tool download
- Greg Nakhimovsky's AppCrash blog
- AppCrash
- Solaris Dynamic Tracing Guide
- DTrace How To Guide
|
Thanks to Xinfeng Liu and Oliver Yang, Professional Engineers from the
Sun China Engineering & Research Institute, for their invaluable
comments.
|
The author can also be reached at adam.zhang.cn@google.com.