Have you ever met that the BSOD can't be reproduced easily? Sometimes, it crashed when you run IE, sometimes when you login msn. There is totally no patten to reproduce.
And the worse thing is that when you try to analyze the dumps, sometimes it points to Ntoskrnl.exe, sometimes to Win32k.sys. When this happened, don't reach a conclusion that windows is the culprit so early. According to Microsoft Online Crash Analysis, more than 70% crashes are caused by Third-party device driver. So your recent changes to the system should have high probability to crash the system.
Now, you should try to use Driver Verifier. With verifer, you can try to turn the un-analyzable crashes into easy to analyze ones. Started windows 2000, verifier is included in windows to promote
stability and reliability.This tool enables additional system checks, that will make it possible for the system to crash immediately, when a driver does something invalid, without allowing it to corrupt more data. For some intermittent BSOD, this tool is very useful.
Now let's see how to use it.
Verifier supports GUI and command-line. We will take GUI as example first.
1. Run Verifier.exe, select 'Create custom Settings (for code developers)', click Next.
2.
3. Don't select "Low resources simulation" since we are not investigating driver's behavior under low resources condition.
4. It is not recommeded to select all drivers which will cause the system become very slow.
5. Select your newly added drivers or some suspicious ones.
6. Reboot your machine. And try to reproduce your BSOD which might crash immediately.
You can also use command-line based verifier.exe.
Open a command line window, type "verifier.exe /?" which will show the details of its usuage.
For example, XP and prior OS will show the usuage like below.
C:/Documents and Settings/Administrator>verifier /?
Driver Verifier Manager - version 5.1.2600.0
Copyright (c) Microsoft Corporation. All rights reserved.
verifier /standard /driver NAME [NAME ...]
verifier /standard /all
verifier [ /flags FLAGS ] /driver NAME [NAME ...]
verifier [ /flags FLAGS ] /all
verifier /querysettings
verifier /volatile /flags FLAGS
verifier /volatile /adddriver NAME [NAME ...]
verifier /volatile /removedriver NAME [NAME ...]
verifier /reset
verifier /query
verifier /log LOG_FILE_NAME [/interval SECONDS]
FLAGS must be a number in decimal or hex, combination of bits:
bit 0 - special pool checking
bit 1 - force irql checking
bit 2 - low resources simulation
bit 3 - pool tracking
bit 4 - I/O verification
bit 5 - deadlock detection
bit 6 - enhanced I/O verification
bit 7 - DMA verification
E.g. /flags 27 is equivalent with /flags 0x1B
The /volatile option can be used to change the verifier settings
dynamically without restarting the system. Any new settings will be lost
when the system is rebooted.
Vista and later OS will have some like below.
C:/Windows/system32>verifier /?
Driver Verifier Manager - version 6.0.6001.18000
Copyright (c) Microsoft Corporation. All rights reserved.
verifier /standard /driver NAME [NAME ...]
verifier /standard /all
verifier [/disk] [ /flags FLAGS ] [/faults [PROBABILITY [TAGS [APPLICATIONS [MINUTES]]]] /driver NAME [NAME ...]
verifier [/disk] [ /flags FLAGS ] [/faults [PROBABILITY [TAGS [APPLICATIONS [MINUTES]]]] /all
verifier /querysettings
verifier /volatile /flags FLAGS
verifier /volatile /adddriver NAME [NAME ...]
verifier /volatile /removedriver NAME [NAME ...]
verifier /volatile /faults [PROBABILITY [TAGS [APPLICATIONS]]
verifier /reset
verifier /query
verifier /log LOG_FILE_NAME [/interval SECONDS]
FLAGS must be a number in decimal or hex, combination of bits:
bit 0 - special pool checking
bit 1 - force irql checking
bit 2 - low resources simulation
bit 3 - pool tracking
bit 4 - I/O verification
bit 5 - deadlock detection
bit 6 - enhanced I/O verification
bit 7 - DMA verification
bit 8 - security checks
bit 9 - force pending I/O requests
bit 10 - IRP logging
bit 11 - miscellaneous checks
For example, /flags 27 is equivalent with /flags 0x1B
The /volatile option can be used to change the verifier settings
dynamically without restarting the system. Any new settings will be lost
when the system is restarted.
...
Please note compared with XP-, vista+ has 4 more bits for flags. Below is a simple command to set the driver verify settings.
For XP-, set all bits except bit 2 low resources simulation:
verifier /flags 0xfb /driver xxx.sys
For Vista+, set all bits except bit 2 low resources simulation:
verifier /flags 0xffb /driver xxx.sys
Additional information about verifier flags, for whole and more detailed description, please refer to http://www.osronline.com/article.cfm?article=325
.
Special Pool
By enabling the Special Pool option, you enable two safeguards for one of the most insidious types of driver error: memory corruption.
The first set of potentially memory corrupting errors that this option will catch is buffer overruns - accessing memory after a valid address range. This is important, because it is very common for a driver that is writing to a random location off the end of its buffer to corrupt another driver in the system. When a situation like that happens, the system will bugcheck and typically blame the wrong driver. These types of blue screens are extremely hard to debug and even harder to explain to your customers.
The other set of potentially memory corrupting errors that Special Pool will catch are accesses to memory after it has been freed.
Pool Tracking
The Pool Tracking option enables one check that is similar to the Special Pool overrun check and another to track resource cleanup.
Force IRQL Checking
Force IRQL Checking enforces the number one IRQL rule: you must not touch any pageable memory at IRQL DISPATCH_LEVEL or above. The reason for this, of course, is that if the pageable memory happens to not be resident, a DISPATCH_LEVEL software interrupt must be executed to bring the page into memory. If the code that is currently running is already at DISPATCH_LEVEL or above, the DISPATCH_LEVEL interrupt cannot run and the page fault cannot be satisfied. Because the Memory Manager aggressively caches pages, it is entirely possible that this bug will go unnoticed during your testing because the pages have already been faulted in at an earlier time by a thread running at a proper IRQL.
The way that Driver Verifier enforces the pageable memory and IRQL rule is by paging out all pageable memory after every IRQL raised to DISPATCH_LEVEL or above. This ensures that all accesses to memory regions marked as pageable at an elevated IRQL generate a DRIVER_IRQL_NOT_LESS_OR_EQUAL bugcheck.
Deadlock Detection
Deadlock Detection is another Driver Verifier option that was added to Windows XP. Enabling it causes Verifier to track all of your driver’s acquires and releases of spinlocks, mutexes and fast mutexes and ensures that a locking hierarchy is in place and is followed.
Low Resources Simulation
Low Resources Simulation is the one test that we generally recommend that you do not enable until your final rounds of testing. Enabling this option will result in random failures for memory requests. What your driver does in these situations and how gracefully it must handle them is entirely device and environment specific, but at the very least the system should not bug check because of a NULL pointer dereference.