参考《Linux® Debugging and Performance Tuning: Tips and Techniques》 chapter 6
1. The wchan option is the most interesting option for finding the location of the hang in this case. If this is a kernel address, ps uses /proc/kallsyms to find the nearest symbolic location.
2. The next ps command shows every process with the PID number, % of CPU, memory size, name, and what syscall the process is currently executing. The output is similar to this:
/home/a/j/nomad2:ps -eo pid,%cpu,vsz,args,wchan
PID %CPU VSZ COMMAND WCHAN
1 0.0 4020 /sbin/init -
2 0.0 0 [kthreadd] kthreadd
3 0.0 0 [migration/0] migration_thread
4 0.0 0 [ksoftirqd/0] ksoftirqd
5 0.0 0 [watchdog/0] watchdog
3. pgrep Lists the Process IDs That Match the Input Criteria
/home/a/j/nomad2:pgrep bash 1305 3494 3931 4704 5652
4. pstree Displays a Tree of Processes
/home/a/j/nomad2:pstree
init-+-atd
|-console-kit-dae---61*[{console-kit-dae}]
|-cron-+-2*[cron---sh---python]
| `-cron---sh
|-5*[dbus-daemon]
|-dd
|-gdm---gdm---kill
|-6*[getty]
|-4*[gvfsd]
|-4*[gvfsd-burn]
|-4*[gvfsd-trash]
|-hald---hald-runner-+-hald-addon-acpi
| |-hald-addon-inpu
| `-hald-addon-stor
|-klogd
|-loop.sh---sleep
|-nscd---8*[{nscd}]
|-portmap
|-python
|-rpc.statd
|-sh---sh---wget
|-sshd-+-13*[sshd---sshd---bash]
| |-sshd---sshd---bash---pstree
| |-14*[sshd---sshd]
| |-sshd---sshd---bash-+-man---pager
| | `-vim
| |-2*[sshd---sshd---sftp-server]
| `-sshd---sshd---bash---vi
|-svnserve
|-syslogd
|-system-tools-ba
|-tinyproxy---10*[tinyproxy]
|-udevd
|-vsftpd
`-xinetd
5. strace can be a very effective way to debug an application if you think a system call is failing.
/home/a/j/nomad2:strace -o /tmp/1 ./a.out
6. back trace
A back trace can be an effective way to identify which process is hung on a system.
If your Linux system is hanging but your keyboard is still functioning, use the following method to help resolve the source of the hang. These steps perform a back trace of the current running process and all processes using the magic key sequence:
1) The kernel that is running on the system must be built with CONFIG_MAGIC_SYS-REQ enabled. The system must also be in text mode. Pressing Ctrl-Alt-F1 places the system in text mode. Pressing Ctrl-Alt-F7 places the system back in X Window.
2) While in text mode, press Alt-ScrollLock followed by Ctrl-ScrollLock. These magic keystrokes give you a stack trace of the currently running processes and all processes, respectively.
3) Look in the system's /var/log/messages file for the back trace. If everything is set up correctly, the system should have converted the symbolic kernel addresses.
7. lsof Lists Open Files
/home/a/j/nomad2:lsof -p 1305 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME bash 1305 nomad2 cwd DIR 0,21 4096 29609555 /home/a/j/nomad2/linux/ch6 (192.168.2.5:/store/home) bash 1305 nomad2 rtd DIR 8,1 4096 2 / bash 1305 nomad2 txt REG 8,1 813912 17694722 /bin/bash bash 1305 nomad2 mem REG 8,1 1436976 17825953 /lib/libc-2.7.so bash 1305 nomad2 mem REG 8,1 14624 17825970 /lib/libdl-2.7.so bash 1305 nomad2 mem REG 8,1 241408 17825802 /lib/libncurses.so.5.6 bash 1305 nomad2 mem REG 8,1 127480 17825944 /lib/ld-2.7.so bash 1305 nomad2 mem REG 8,1 702576 8323249 /var/cache/nscd/passwd bash 1305 nomad2 0u CHR 136,19 21 /dev/pts/19 bash 1305 nomad2 1u CHR 136,19 21 /dev/pts/19 bash 1305 nomad2 2u CHR 136,19 21 /dev/pts/19 bash 1305 nomad2 255u CHR 136,19 21 /dev/pts/19
8. the network debugging tools ifconfig, arp, tcpdump, ethereal, and netstat.
The option for netstat -tap, which is a good way to determine what programs are serving from your system. It can be used to look for rogue connections to your server.