Mike Taulty's Weblog
Ones & Zeros
I read...
Blog Roll (Opml)
Microsoft
MS UK Post Events Site
News
BBC News
The Guardian
The Observer
A word for WinDbg
Before I came to work at Microsoft I worked as a professional developer on the Microsoft platform and I used to work with the Visual Studio (6 latterly) debugger which I thought was a pretty cool debugger.
When I joined Microsoft I found that a lot of people worked with WinDbg as their primary debugger and, for a while, I just didn't get it as it seemed like an unnecessarily grungy tool for achieving things that Visual Studio could already do.
However, over a period of time I came to see the light and I'm now of the view that WinDbg is about the best debugger I've ever used (including those ones I used to work with on Unix platforms). WinDbg is just the most powerful debugger out there for user-mode debugging and I don’t do any other kind of debugging J
So, there is a lot of information out there already on WinDbg but I just wanted to try and highlight it here and provide a single-pager on it as it seems to be grossly overlooked as a debugger for managed code and, yet, it’s capable of a whole tonne of stuff that no single other debugger can do for you right now.
People are often put off WinDbg because of its interface and the vast array of commands so in this little series of postings I’ll try and demystify a little bit and show how you can debug a few common scenarios in managed code with it.
In this post, let’s just get to the point where we can use WinDbg to debug a regular process and do some work with it then in the next post(s) I’ll look at more stuff that you can do.
Installation
So, first things first, get hold of a copy of WinDbg from here and install it. If you want to follow along with this post then you can go and download it right now - the install takes about 2 minutes once you've got the 10MB download down from the site.
The first thing you'll realise is how lightweight an installation WinDbg is and that (sometimes) means that you can install it into places that you'd never install a copy of Visual Studio or the Visual Studio remote debug set up. Indeed, often if you raise a support call with Microsoft you'll get asked to install WinDbg and it's automated helper (the "autodumper") in order to get a crash dump of your application to send back to Microsoft.
Before we get into debugging something I need to have a quick word or two about Symbols and Commands.
Symbols
The first issue that we need to address in order to progress is that of symbols. If you want to get decent stack traces and dumps of variable values and so on out of the debugger then you need symbols for the modules that you’re debugging whether those modules are yours or someone else’s and whether those modules are managed or unmanaged (although there’s quite a lot more that you can do with managed code without symbols).
Symbols are typically either private symbols (include variable information), retail symbols (function information but not variable information) or export symbols (generally not so useful for debugging purposes). If you’re debugging with “export” symbols it’s not usually enough to actually work out what’s going on.
For your own code you’re responsible for building the symbol files (usually .PDB program databases) by setting the right flags on the compiler or choosing the right configuration in a project (this is true for VC++, VB6 and all the .NET languages). This will give you full private symbols.
When the debugger (and lots of other tools) want to find symbols they typically check the contents of an environment variable named _NT_SYMBOL_PATH which is used (just like PATH – i.e. a list of folder separated by semi-colons) to find symbol files. You can set _NT_SYMBOL_PATH prior to running WinDbg or you can use the debugger’s own symbol path settings to find symbol files. These settings are controlled via the File->Symbol File Path menu option or using the .sympath command.
For other people’s code you need to get symbols from them. For Microsoft code you’re in luck because the symbols are available from a public symbol server on the internet. In order to make use of the symbol server you should set your symbol path to be something like;
SRV*C:/MyLocalSymbols*http://msdl.microsoft.com/download/symbols
What this says is that the debugger should first check a local cache named C:/MyLocalSymbols and if symbols are not found there then go out to the Microsoft symbol server for symbols and, if found, download them and cache them in that folder. Note that symbols are downloaded based upon name, version and checksum information from the modules being debugged so the debugger usually knows if the symbols match the code properly.
If you wanted to combine this with a local location for your own code then you’d use something like;
C:/MyCodesSymbols; SRV*C:/MyLocalSymbols*http://msdl.microsoft.com/download/symbols
so that the debugger would check your folder first and would then proceed to look in the cache and finally to the symbol server for symbols.
Commands
A core set of commands would be as below. I rarely venture beyond these as I’m not an advanced user of WinDbg by any means and this gets me by.
.hh - bring up the help file 
The help file is fantastic so don't skip on it - everything's documented in there.
g - "GO!". That is, continue running.
bp (Address) - set a break point. Note that (address) can take many formats here including just a memory location but the most common format specifier here is to use syntax such as;
bp KERNEL32!CreateFileA
(remember that most Windows functions have an ASCII and a Unicode variant so we have KERNEL32!CreateFileA and KERNEL32!CreateFileW).
bl - list all breakpoints. Each breakpoint listed has a number in the list which you need for...
bc (number), be (number), bd (number) - these respectively clear, enable and disable breakpoints from the list.
kb, kp, kd etc. The commands beginning with "k" show the stack for the thread that the debugger is looking at. There are a few variants so check out the help for those. The simplest variant that I use is;
kb 200
(200 is the maximum depth of stack trace that I'm looking for)
dw, db, ds, etc. The commands beginning with "d" show the contents of memory. One of the most common is "dt" which shows you the contents of memory laid out according to the specification of a type as long as the debugger can see the definition of a type.
sxe, sxd, sxi, sxn. The commands beginning with "sx" set the behaviour for what the debugger should do when an exception occurs. A good example here would be;
sxe 0xc0000005
which is saying "I want to break into the debugger if there's an access violation".
lm - lists the modules loaded by the program and what kind of symbols are loaded for the modules. Note that export symbols are not really symbols at all but are really the debugger guessing which function you're in based upon the export table of the DLL. For COM servers in particular (which have only 4 exported functions or so) you'll often find yourself appearing in DllRegisterServer if you only have export symbols. Good examples here would be;
lm v (verbose mode)
lm v mUSER* (verbose mode, matc any modules that begin with USER*)
x - Examine symbols. This is very, very useful as it shows you the set of symbols loaded for particular modules. A good example here would be;
x USER32!* (show me all the symbols loaded from the USER32 module)
x USER32!Create* (show me all the symbols loaded from the USER32 module that begin with "Create")
.cls – clear the screen. You’ll be needing this one!
.reload – this causes the symbol information to be reloaded for a particular module. The most common form of this would be to do something like;
.reload /f user32.dll
Where the /f overrides the debugger’s naturally lazy mode of working whereby it wouldn’t actually do the reload right there and then. Normally, this is a useful parameter to use.
~ - this is the tilde character and it lists all the threads in the process and shows their status. This can be combined with wildcards in order to execute particular commands on all threads in the process. The single most common one is probably to combine it with a “stack” command to do something like;
~* kb 200
which will show you the stack frames for all the threads in the process.
~N s – this changes the debugger’s focus to another thread. For example;
~7 s
Will switch the debugger so that it’s focused on thread 7. Note that 7 here is the debugger’s thread number rather than the real thread ID and it comes from the list given by the ~ command.
Along with the core set of commands, WinDbg is capable of loading up debugger extensions to extend the core functionality of the debugger. These are “bang commands” in that they begin with an exclamation mark. You can manipulate the extensions that the debugger has loaded using the following commands;
.chain – this shows the extensions that the debugger has loaded
.load – this loads an extension DLL (e.g. .load SOS.DLL)
.unload – unloads an extension DLL
.unloadall
.setdll – this sets the “default” extension DLL.
A quick word about .setdll and how command processing works here. If you have an extension named UEXT.DLL which contains a command named help (and most extensions will have a help command) then you can run that command using;
!UEXT.help
And that works fine. If you find yourself using UEXT more than any other extension then you can use;
.setdll UEXT.DLL
To make UEXT.DLL the default extension and then you can just use;
!help
And that will now default to mean !UEXT.help because of the default extension setting.
A First Debugging Session
So, let’s finish up this post with a quick debugging session and next time around I’ll talk about debugging managed code rather than just any old process.
Run up WinDbg and also run up a copy of NotePad.
Within WinDbg attach to the NotePad process that you just ran by either using the File->Attach to Process menu or the F6 shortcut. This will give you a dialog where you can select a process. Note that the “Non Invasive” option would allow you to detach from the debugged process without killing it (if you’re on Windows XP or 2003) but it limits the commands that you can use so leave it unchecked.
Don’t worry about the workspace information, let it go.
The debugger should present some diagnostics as it loads information and will then stop with the command prompt. The debuggee (NotePad) is now halted.
Let’s take a look around. Firstly, let’s look at the modules that are loaded by notepad. Execute a “lm v” to list all the modules and follow it up with a simple “lm”. On my system I get this;
0:001> lm
start end module name
01000000 01014000 notepad (no symbols)
4ce00000 4cf02000 COMCTL32 (export symbols) C:/WINDOWS/WinSxS/x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.2149_x-ww_a84b1f06/COMCTL32.dll
5ad70000 5ada7000 uxtheme (export symbols) C:/WINDOWS/system32/uxtheme.dll
605d0000 605d9000 mslbui (export symbols) C:/WINDOWS/system32/mslbui.dll
61220000 61232000 MSH_ZWF (export symbols) C:/Program Files/Microsoft Hardware/Mouse/MSH_ZWF.dll
73000000 73026000 WINSPOOL (export symbols) C:/WINDOWS/system32/WINSPOOL.DRV
74720000 7476b000 MSCTF (export symbols) C:/WINDOWS/system32/MSCTF.dll
763b0000 763f9000 comdlg32 (export symbols) C:/WINDOWS/system32/comdlg32.dll
77120000 771ac000 OLEAUT32 (export symbols) C:/WINDOWS/system32/OLEAUT32.DLL
771b0000 779c4000 SHELL32 (export symbols) C:/WINDOWS/system32/SHELL32.dll
779d0000 77a46000 SHLWAPI (export symbols) C:/WINDOWS/system32/SHLWAPI.dll
77a50000 77b8d000 ole32 (export symbols) C:/WINDOWS/system32/ole32.dll
77c10000 77c68000 msvcrt (export symbols) C:/WINDOWS/system32/msvcrt.dll
77cc0000 77d5b000 ADVAPI32 (export symbols) C:/WINDOWS/system32/ADVAPI32.dll
77d60000 77df1000 USER32 (export symbols) C:/WINDOWS/system32/USER32.dll
77e00000 77e93000 RPCRT4 (export symbols) C:/WINDOWS/system32/RPCRT4.dll
77ea0000 77ee5000 GDI32 (export symbols) C:/WINDOWS/system32/GDI32.dll
7c800000 7c8f2000 kernel32 (export symbols) C:/WINDOWS/system32/kernel32.dll
7c900000 7c9ae000 ntdll (export symbols) C:/WINDOWS/system32/ntdll.dll
Let’s get some symbols because we can’t really work with export symbols. Set up your symbol path (either through the File->Symbol Path menu or through the .sympath command) so look something like this;
SRV*e:/LocalSymbolCache*http://msdl.microsoft.com/download/symbols
Reload symbols for the loaded modules by using the “.reload /f” command – this may well take a little time as the symbols for all those modules that NotePad loaded trickle down from the internet to your local cache. This gets better with time as you pull down common symbols from the net.
After the reload completes do another lm and make sure you got some symbols. My session now gives me;
0:001> lm
start end module name
01000000 01014000 notepad (pdb symbols) e:/LocalSymbols/notepad.pdb/15800B8231AF4FDE85232D42B267D3E51/notepad.pdb
4ce00000 4cf02000 COMCTL32 (pdb symbols) e:/LocalSymbols/MicrosoftWindowsCommon-Controls-6.0.2600.2149-comctl32.pdb/4133208CAF9A4B93982A1F873FBC79261/MicrosoftWindowsCommon-Controls-6.0.2600.2149-comctl32.pdb
5ad70000 5ada7000 uxtheme (pdb symbols) e:/LocalSymbols/uxtheme.pdb/3A4D37B97BA34837AC589BDB15CC265D2/uxtheme.pdb
605d0000 605d9000 mslbui (pdb symbols) e:/LocalSymbols/MSLBUI.pdb/ED5CCEFBD83C4F929AAB9A1029C54DDD1/MSLBUI.pdb
61220000 61232000 MSH_ZWF (export symbols) C:/Program Files/Microsoft Hardware/Mouse/MSH_ZWF.dll
73000000 73026000 WINSPOOL (pdb symbols) e:/LocalSymbols/winspool.pdb/20EE1501116B4C4180ABE40B209DEC962/winspool.pdb
74720000 7476b000 MSCTF (pdb symbols) e:/LocalSymbols/msctf.pdb/F8F1C6E0E1CC4D1CB4936C12A6A6D4C92/msctf.pdb
763b0000 763f9000 comdlg32 (pdb symbols) e:/LocalSymbols/comdlg32.pdb/2BC57F8BE63A429FB4B439EA2854387A2/comdlg32.pdb
77120000 771ac000 OLEAUT32 (pdb symbols) e:/LocalSymbols/oleaut32.pdb/96D8A9A6CB704201A365F345B2A9F72D2/oleaut32.pdb
771b0000 779c4000 SHELL32 (pdb symbols) e:/LocalSymbols/shell32.pdb/FE0DC899DD78422AB29561FE8255E1C92/shell32.pdb
779d0000 77a46000 SHLWAPI (pdb symbols) e:/LocalSymbols/shlwapi.pdb/231AB6F0A19F4DBB9837440BCC167BEC2/shlwapi.pdb
77a50000 77b8d000 ole32 (pdb symbols) e:/LocalSymbols/ole32.pdb/656B303C367B4E679CA249E32CE572D92/ole32.pdb
77c10000 77c68000 msvcrt (pdb symbols) e:/LocalSymbols/msvcrt.pdb/8E17C9B9A55C4118A7D35025C5BE51871/msvcrt.pdb
77cc0000 77d5b000 ADVAPI32 (pdb symbols) e:/LocalSymbols/advapi32.pdb/57E2A801C610451AA66B7CD94F0910F72/advapi32.pdb
77d60000 77df1000 USER32 (pdb symbols) e:/LocalSymbols/user32.pdb/35948255FAC04CA99F5CF61959B45DC42/user32.pdb
77e00000 77e93000 RPCRT4 (pdb symbols) e:/LocalSymbols/rpcrt4.pdb/79C7047CB77B435DB01EB365C3AF17152/rpcrt4.pdb
77ea0000 77ee5000 GDI32 (pdb symbols) e:/LocalSymbols/gdi32.pdb/E7D5B033EFE5466E97DB048BE4C88FA62/gdi32.pdb
7c800000 7c8f2000 kernel32 (pdb symbols) e:/LocalSymbols/kernel32.pdb/5091FA1742544F2EA3D425642634F78E2/kernel32.pdb
7c900000 7c9ae000 ntdll (pdb symbols) e:/LocalSymbols/ntdll.pdb/7B2A2EF0E63F4848A8D309E84E47322C2/ntdll.pdb
So, that’s modules and symbols. Let’s see if we can set some breakpoints. Suppose that I want to hit a breakpoint anytime that NotePad opens up a file. Let’s do that. Firstly, take a look at symbols in KERNEL32 to see if we can find the CreateFile function that NotePad uses.
Issue a “x KERNEL32!CreateF*” and see what results we get back you should find the ASCII CreateFileA and the Unicode CreateFileW functions in there.
Set a breakpoint on a function. Issue a “bp kernel32!CreateFileW” to set a breakpoint on that function.
Issue a “GO” command “g” to continue the debuggee running.
Note that when the debuggee is running you can always get back to the debugger by issuing a CTRL+BREAK to the debugger window when it will try and halt the debuggee for you and drop you back to the command prompt.
Open a file with notepad. You’ll spot NotePad loading some more modules in order to do this operation and get that dialog onto the screen. Your breakpoint will more than likely hit before the file dialog comes up.
Check-out where you are with your breakpoint – i.e. what the stack frame looks like. Issue a “kb 200” command. You should get results as below;
0:000> kb
ChildEBP RetAddr Args to Child
0006be18 76671b2a 76671b74 00000020 00000003 kernel32!CreateFileW
0006be94 766724e2 000a6b40 0006bf4c 00000000 cscui!IsCSCEnabled+0x38
0006bea8 77a68b49 000a7084 77a51a60 0006bf44 cscui!DllGetClassObject+0x72
0006bec4 77a80f5e 000a7084 77a51a60 0006bf44 ole32!CClassCache::CDllPathEntry::DllGetClassObject+0x2d
0006bedc 77a80e9a 0006bef0 77a51a60 0006bf44 ole32!CClassCache::CDllFnPtrMoniker::BindToObjectNoSwitch+0x1f
0006bf08 77a81cc6 0006bf4c 00000000 0006c540 ole32!CClassCache::GetClassObject+0x38
0006bf84 77a806aa 77b76ca4 00000000 0006c540 ole32!CServerContextActivator::CreateInstance+0x106
0006bfc4 77a81e19 0006c540 00000000 0006ca8c ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
0006c018 77a81d90 77b76ca8 00000000 0006c540 ole32!CApartmentActivator::CreateInstance+0x110
0006c038 77a8101e 77b76ca8 00000001 00000000 ole32!CProcessActivator::CCICallback+0x6d
0006c058 77a80fd5 77b76ca0 0006c39c 00000000 ole32!CProcessActivator::AttemptActivation+0x2c
0006c090 77a81e7a 77b76ca0 0006c39c 00000000 ole32!CProcessActivator::ActivateByContext+0x42
0006c0b8 77a806aa 77b76ca0 00000000 0006c540 ole32!CProcessActivator::CreateInstance+0x49
0006c0f8 77a81bc4 0006c540 00000000 0006ca8c ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
0006c348 77a806aa 77b765d4 00000000 0006c540 ole32!CClientContextActivator::CreateInstance+0x8f
0006c388 77a805dc 0006c540 00000000 0006ca8c ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7
0006cb38 77a64eb1 000a2f08 00000000 00000001 ole32!ICoCreateInstanceEx+0x3c9
0006cb60 77a64e80 000a2f08 00000000 00000001 ole32!CComActivator::DoCreateInstance+0x28
0006cb84 77a65102 000a2f08 00000000 00000001 ole32!CoCreateInstanceEx+0x1e
0006cbb4 779d69a5 000a2f08 00000000 00000001 ole32!CoCreateInstance+0x37
So, we can see the direct call-stack that led to our call to CreateFileW. If we had full symbols for these modules we could use a KP rather than a KB command and we’d get a list of parameters to the functions. As it is with what we have here we’d need to resort to disassembly to get the parameter information here and it’s beyond this posting.
Continue the debuggee by issuing a “g” command and continue issuing “g” commands until NotePad finally puts its dialog on the screen – note how many times we hit this function here J If you’re interested, issue a few KB’s on some of these call chains and see what exactly it is that Notepad is doing.
Break the debuggee by hitting CTRL+BREAK on the WinDbg window.
Issue a “~” command to get a picture of the threads in the process. In my session I get 3 as below;
0:002> ~
0 Id: 9bc.b58 Suspend: 1 Teb: 7ffdf000 Unfrozen
1 Id: 9bc.8a4 Suspend: 1 Teb: 7ffde000 Unfrozen
. 2 Id: 9bc.d4 Suspend: 1 Teb: 7ffdd000 Unfrozen
Note that this display indicates that the debugger is focused on thread “2” which is really OS thread 8A4 in process 9BC (all hex I’m afraid).
Check out where all the threads are by issuing a “~* kb 200”. You might not get anything too exciting here but it’s illustrative of how to use the technique.
Finally, close off this session by having a look at some extension commands.
List the set of extensions that you’ve got by issuing the “.chain” command.
Have a look at the commands available in these extensions by issuing: !exts.help, !uext.help, !ntdsexts.help, !ext.help and see what commands are available to you.
Try an extension command. For example, “!exts.cs” will display all the Critical Sections that the process has and their status as to whether they are locked or not. This can be a useful command in many server-side scenarios.
End the debugging session with a “q” command.
That’s it. I’ll post again in the near future around how you can build on what I’ve just walked through here in order to debug managed code with WinDbg and take advantage of features within the debugger that don’t (yet) appear elsewhere.
posted on Tuesday, August 03, 2004 3:54 AM
Mike Taulty's Weblog
Posted @ 8/3/2004 5:28 AM
# WinDbg review
Robert Hurlbut's .Net Blog
Posted @ 8/3/2004 8:05 AM
# WinDbg is your friend
Jed's Blog
Posted @ 8/3/2004 9:09 AM
# re: A word for WinDbg
Hakan Berg hbe@master.nu
Posted @ 10/11/2004 4:21 AM
Excellent article. I love such clear step-by-step how-to-do instructions. This will really help me in my progam developments.
Thank you!
# re: A word for WinDbg
david
Posted @ 12/31/2004 1:14 AM
It is great! thank you very much
*_*
# re: A word for WinDbg
Yi
Posted @ 1/14/2005 7:09 PM
Very good article!
# re: A word for WinDbg
Deepu
Posted @ 2/21/2005 6:10 AM
A crisp article, with a smart example. Thank you very much. Good Job Done!!!
# A word for Windbg
SmartWombat Blog
Posted @ 2/22/2005 8:40 AM
# John Robbins on .NET V2.0 and SOS.DLL
Mike Taulty's Weblog
Posted @ 3/1/2005 2:39 PM
# re: A word for WinDbg
Yuji
Posted @ 3/3/2005 3:43 AM
This is very good article. Thanks!!
# re: A word for WinDbg
Rene Pally
Posted @ 3/9/2005 3:11 PM
I have some problems trying to run Windbg on Windows 2003 specially configuring aedebug. Ok, the Windbg catch the crashed process but, when windbg starts with crashed process windbg console freezes. Any Ideas? happens on Windows 2003 SP1
# Excellent .NET debugging tool
Blog: Martin Parry
Posted @ 3/21/2005 3:34 AM
# A word for Windbg
Smartwombat Blog
Posted @ 3/28/2005 2:07 AM
Introduction to windbg for both managed and unmanaged debugging A word for WinDbg - short introduction...
# Would it work with VB6
Samion
Posted @ 4/4/2005 11:02 AM
assuming I have a symbols file for an application.
# re: A word for WinDbg
Mike
Posted @ 4/4/2005 11:54 AM
You can debug VB6 code with WinDbg, the SOS stuff is just for managed (.NET) code but regular VB6 code with symbols and without optimizations debugs ok in WinDBg.
# re: A word for WinDbg
jason
Posted @ 4/10/2005 11:51 PM
Any Ideas for the following output?
"POISONED_TB: nt!MmPoisonedTb is non-zero"
2. "mrxsmb!BowserGetBackupListResponse+7a"
3. "The machine has been manipulated using the kernel debugger."
Thanks
jason
# re: A word for WinDbg
Rob
Posted @ 6/16/2005 5:48 AM
Good article.
I am interested in the dump analysis. What do you do if the item causing a server crash is not a Microsoft file and does not have an associated symbol file? is it possible to make your own symbol file?
# re: A word for WinDbg
Mike
Posted @ 6/16/2005 5:55 AM
Hi,
I'm not quite sure what you're looking for here. If you've got code in a crashing process that's not from MS then you can still have symbols (usually .pdb files) for that code but those symbol files are part of the build process so you'd normally get them from the vendor of that software or as part of building the software yourself.
# re: A word for WinDbg
Rob
Posted @ 6/16/2005 12:25 PM
I have gone through this page ten times already. You did a great job.
My problem deals with a Novell client file (SRVLOC.SYS) which causes my Terminal servers to randomly crash. When I run WinDbg to analyze the dump file, it can't find the proper symbol file for srvloc.sys, but it does indicate this is indeed the problem. The srvloc.pdb file is nowhere to be found.
I am going to keep running through your notepad example above in an attempt to better learn WinDbg. You have definitely opened my eyes to the power of WinDbg. Thank you
# re: A word for WinDbg
Tim
Posted @ 3/3/2006 11:17 AM
Personally, I love LiveKD from System Internals. Uses WinDBG for live kernel debugging.