For myriad reasons outside of un-kept engineering curiosity one might find value and interest in reviewing the list of device drivers installed and engaged on a system.
With a list one might also want to get hard numbers on how much resource (memory) each driver is taken.
This analysis allows you one to do certain things:
- Respond to Microsoft and other Vendors explicit\implicit requests for supporting data as a trouble ticket is being worked
- Account for memory usage. Each machine has a finite amount of memory and a DBA needs to work with OS Administrators and plan for how much memory to allocate to the OS, each Application, and how much to leave available to 3rd party applications (Backup, Anti-Virus). The rest of the memory is thus left available to device drivers.
- Address intermittent system crashes. Compared to Software Applications, Device drivers run a bit closer to the kernel. Errant Device Driver calls or them not properly managing memory are thus more problematic to overall system stability.There is quite a bit of information on the Internet that covers how to ensure that memory dump files will be created and how to read the generated dump files; i.e. “How to read the small memory dump file that is created by Windows if a crash occurs” (http://support.microsoft.com/kb/315263) and thus will not cover it here.
Depending on your OS, you can acquire poolmon through a few install media and location.
Windows 2003 Support Tools Install on Windows 2008
Install Media location
Upon kicking off installation of the downloaded “Windows 2003 Support Tools” on my target computer, a Windows 2008-R2 Server, received a warning right away.
This program has known compatibility issues. Check online to see if solutions are available from the Microsoft website. If solutions are found, Windows will automatically display a website that lists steps you can take.
Program: Windows Support Tools
Location: Not Available
If you click on the “Check for Solutions online” button, you get a message stating that “No solutions found for Windows Support Tools”
Start poolmon.exe by launching a command shell, changing your current directory to the folder you chose as your install folder, and entering poolmon.exe
Poolmon: Query perf Failed (returned: c0000004)
So no go!
Windows Driver Kit (WDK) 8.1
What are Windows Driver Kit?
They are the tools to build, test, debug, and deploy drivers.
Install Media Location
There are a couple of items to note as you install Windows Driver Kit (for Windows 8.1):
- The Install path can no longer be changed. It has to be C:\Program Files (x86)\Windows Kits\8.1\
- The driver kit does not contain actual Application development software, but rather assisting tools
The install succeeded.
Remember that the targeted folder is C:\Program Files (x86)\Windows Kits\8.1.
As my targeted system is a 64-bit OS, we need to sojourn to C:\Program Files (x86)\Windows Kits\8.1\Tools\x64.
To try out poolmon launch a Command Shell and change your current folder to that folder.
-b order by bytes accumulated
Here is the output (Development box):
Tabulated Output (Development box):
-a order by # of Allocations
Here is the output (Development box):
Tabulated Output (Development box):
Explanation (Development box):
- Our busiest Tags are sourced by Self, Ndre, Even, and NSpg
- Self is such a generic word and I could not affirmatively link it to a specific Vendor
- NDre is C:\Windows\System32\drivers\ndis.sys (Microsoft Network Sub-system)
- Even is Microsoft’s Events Object (http://blogs.technet.com/b/yongrhee/archive/2009/06/24/pool-tag-list.aspx)
- NSpg is C:\Windows\System32\drivers\nsiproxy.sys (Microsoft Network\Proxy Sub-system)
- Though busy, memory is being allocated and freed at the same rate for all the listed sources, outside of Even
Review Poolmon output
Once you have Poolmon running, you want to look for a few things:
- The Tags using the most Bytes. Anything over 40,00,000 bytes ( 40 MB) deserves your acute look
- Tags that are steadily Allocating bytes (Allocs column growing), but not de-allocating memory (Frees staying as is) might point at memory management problems
Review Specific Tags
Map Tags to Specific Device Driver – Using SysInternals Strings & Findstr
In some cases, the corresponding device drivers for each Tag entries are easy enough.
But, in some cases the Tags can not easily be mapped to specific Driver and thus vendor.
Here again Mark Russinovich is a true and beloved friend. His tool (Strings.exe) available @ http://technet.microsoft.com/en-us/sysinternals/bb897439.aspx searches files resource area for strings passed in.
So in this case you pass in a Tag and for well written applications that support Internalization, the messages and tags are bundled a bit differently to allow for various languages to be more easily supported.
Launch Command Shell and issue command based on the syntax below:
cd c:\windows\system32\drivers strings * | findstr /i TagID
cd c:\windows\system32\drivers strings * | findstr /i "CM31"
Map Tags to Specific Device Driver – Using Findstr
Microsoft’s Findstr is useful to find strings embedded in files underneath a folder.
ss64.com has a good and very usable documentation on findstr. The URL to per-use is http://ss64.com/nt/findstr.html
cd c:\windows\system32 findstr /m /i /s TextToSearchFor file-name-pattern
cd c:\windows\system32\drivers findstr /m /i /S "SaEe" *.sys
- From the screen shot above, we moved up a folder and initiated our search from c:\window\system32 rather than c:\windows\system32\drivers and we were still able to match our Tag (SaEe) to a file name (drivers\srtsp64.sys)
- Also, do not assume that all device drivers will be placed in c:\windows\system32\drivers folder
Map Tags to Specific Device Driver (Interpretation)
On a lot of machines that I tested poolmon.exe, I found CM31 to be the driver using the most memory. Please read this excellent write-up by Microsoft’s HarshDeep Singh.
- HarshDeep_Singh – Why the registry size can cause problems with your SQL 2012 AlwaysOn/Failover Cluster setup
In the article cited above, HarshDeep does the following:
- Gave a good background on what precipitated his research, thus seeding the reader’s interest and mind
- And, he liberally listed his diagnostics steps and sources
- Also, he introduces Microsoft’s own Tag List – The troubleshooters and problem solvers… – http://blogs.technet.com/b/yongrhee/archive/2009/06/24/pool-tag-list.aspx
System Overall Kernel Memory Usage
To back-track a bit, you might want to review your overall Kernel Memory Usage. To do so, please launch Task Manager and glance over to the “kernel memory” section:
You want to track your listed “Paged” and “Nonpaged” memory.
Microsoft’s Internal god, Mark Russinovich, does a good job describing the difference between the two.
The kernel and device drivers use nonpaged pool to store data that might be accessed when the system can’t handle page faults. The kernel enters such a state when it executes interrupt service routines (ISRs) and deferred procedure calls (DPCs), which are functions related to hardware interrupts. Page faults are also illegal when the kernel or a device driver acquires a spin lock, which, because they are the only type of lock that can be used within ISRs and DPCs, must be used to protect data structures that are accessed from within ISRs or DPCs and either other ISRs or DPCs or code executing on kernel threads. Failure by a driver to honor these rules results in the most common crash code, IRQL_NOT_LESS_OR_EQUAL.
Paged pool, on the other hand, gets its name from the fact that Windows can write the data it stores to the paging file, allowing the physical memory it occupies to be repurposed. Just as for user-mode virtual memory, when a driver or the system references paged pool memory that’s in the paging file, an operation called a page fault occurs, and the memory manager reads the data back into physical memory. The largest consumer of paged pool, at least on Windows Vista and later, is typically the Registry, since references to registry keys and other registry data structures are stored in paged pool.
Couple of takeaways:
- Non-Paged as the name now suggests means memory that will never be persisted to the paging file
- Paged file can be moved to the paging file
Conclusively, our numbers are good!
Additional Reading & Tools
In our case, we did not notice any hypersensitive device drivers. But, it was still a good exercise as re-discovered Microsoft’s YongRhee and his tireless sharing.
YongRhee does a very comprehensive job aggregating patches and he publicly avails them @ http://blogs.technet.com/b/yongrhee/.
Without over-stating it, Mark Russinovich answered my need to know why CM31 happens to take so much storage. CM means configuration manager and dummy on me, it is the Registry.
It seems that Registry data is read and kept in memory and thus processes do not have to go to disk to read them.
And, so you really want to be careful as to what you install on your machine and also consider Registry cleaning & pruning tools.
References – Windows – Kernel Memory – Paged and NonPaged Pool
- Pushing the limits of Windows – Paged and NonPaged Pool
References – Pool Corruption
- Understanding Pool corruption – Part 1 – Buffer Overflows
References – Memory Exhaustion
- Ever increasing PIDs leading to Memory Exhaustion
References – Pool Usage Metrics
- Who is using the Pool?
References – Map Tag ID to File Name ( using findstr )
- How to find pool tags that are used by third-party drivers
- SS64 – FindStr