Monitoring and Troubleshooting Using Event Logs

[LEFT][CODE]http://www.windowsnetworking.com/articles_tutorials/Monitoring-Troubleshooting-Event-Logs.html[/CODE]

The event logs on Windows systems are helpful for both troubleshooting when things go wrong and monitoring performance and behavior. An event log is a file that contains events, which are entries to the log that notify the user of some occurrence relating to the operating system or applications running on the system. An event includes information about the type of occurrence, the date and time when it occurred, the computer where it happened and the user who was logged on at the time, and other information such as event ID, the event category, and the source of the event. Events may also include further detailed information concerning the event and possibly a link to where more information can be found. Figure 1 below illustrates an example of an event from the DNS Server event log on a Windows Server 2003 domain controller:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-11094547476219.gif[/IMG]
[B]Figure 1:[/B] Example of an event.[/INDENT][B]Finding More Information About an Event[/B]

If an event contains a link and you click on it, a dialog box opens warning you that information about the event will be sent to Microsoft to see if they have more information available concerning the event:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-21094547484984.gif[/IMG]
[B]Figure 2:[/B] Sending event information to Microsoft.[/INDENT]Clicking Yes opens the Help and Support Center and checks to see if there is any more information about the event that may be helpful. Figure 3 shows a typical response:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-31094550476453.gif[/IMG]
[B]Figure 3:[/B] Additional help concerning the event.[/INDENT]How many times have you been frustrated by the lack of helpful information available this way concerning some obscure event? In the example above, the additional help provided is that “this error could be caused by either a high load on the domain controller or the failure of other domain controller services” and the suggested remedy is to “restart the DNS Server service” and check the event log for anything else that happened at the same time and could be a clue. In other words, its like the old mantra “when all else fails, try rebooting.” Where can you find more help?
[URL="http://www.altairtech.ca/"]Altair Technologies[/URL] maintains a helpful site called [URL="http://www.eventid.net/"]EventID.net[/URL] where users can search for additional information about obscure Windows events to help you interpret them. This site is community-based, meaning that users post their comments concerning events to create a community database that can then be searched by others. If you search EventID.net for information about the above event (source = DNS, event ID = 4004) the following is displayed:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-41094550484438.gif[/IMG]
[B]Figure 4:[/B] Searching EventID.net for more information about event ID 4004 for DNS. [/INDENT]The really useful feature is under Details, where you can click the link “Comments and links for event id 4004 from source DNS” to see comments posted by other users:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-51094550493203.gif[/IMG]
[B]Figure 5:[/B] Comments on event ID 4004 for DNS posted by users of EventID.net[/INDENT]The last comment is particularly useful as it indicates MS is aware of why this event occurs and suggests it can usually be safely ignored. Help and Support never told us that!
[B]Configuring Event Logs[/B]

One of the first things you should do after you install a new Windows system is configure the event logs on that system. This is particularly important for servers where event logs can provide critical information to help you troubleshoot when things go wrong. Before we look at how to configure event logs, we need some background information on the different logs available, and Table 1 provides this below:
[B]Event log[/B]
[B]Log file[/B]
[B]Function[/B]
[B]Availability[/B]
Application log
AppEvent.evt
Records events as determined by each software vendor
All Windows systems
Security log
SecEvent.evt
Records events based on how audit policy is configured
All Windows systems
System log
SysEvent.evt
Records events for Windows operating system components
All Windows systems
Directory Service log
NTDS.evt
Records events for Active Directory
Domain controllers only
DNS Server log
DnsEvent.evt
Records events for DNS servers and name resolution
DNS servers only
File Replication Service log
NtFrs.evt
Records events for domain controller replication
Domain controllers only

[B]Table 1:[/B] Summary of Windows event logs By default all event logs are:

[LIST][*]Stored in the %Windir%\system32\config folder[*]Have a maximum size of 16 MB (Windows Server 2003) or 512 KB (Windows 2000/XP)[*]Overwrite events more than 7 days old[/LIST]
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-61094547844625.gif[/IMG]
[B]Figure 6:[/B] Default configuration of DNS Server event log on a Windows Server 2003 DNS server. [/INDENT]Before you put your new Windows server into production, you should decide if these default settings are appropriate. Suggested best practices for configuring event logs on servers include the following:

[LIST][*]Increase the size of each event log to at least 50 MB. Since a typical event is about half a kilobyte in size, this means you’ll be able to store 100,000 events in each log. Note that the maximum supported size of each event log is about 300 MB. If your system drive has insufficient space for your event logs, you can move them to a separate volume by editing the subkey for each log under the HKLM\SYSTEM\CurrentControlSet\Services\Eventlog using Registry Editor, see [URL="http://support.microsoft.com/default.aspx?scid=kb;en-us;315417&sd=tech"]Microsoft Knowledge Base article 315417[/URL] for more information.[*]Change the overwrite behavior for the Security log to Do Not Overwrite Events if your enterprise is a high security environment. That way if the Security log fills up the system will shut down to ensure that no events in the Security log are lost. If you do this, make sure you also archive and then clear your Security log regularly to prevent such a shutdown from occurring unexpectedly.[*]Change the overwrite behavior for the other event logs to Overwrite Events As Needed so that no overwriting occurs until the entire log becomes full. Again, be sure to regularly archive and clear your event logs to prevent the log from filling up and losing events because of overwrites.[/LIST]
If you have a number of computers and are running Active Directory on your network, you can also use Group Policy to configure event log settings. These settings are found under Computer Configuration/Windows Settings/Security Settings/Event Log in Group Policy Object Editor:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-71094547851891.gif[/IMG]
[B]Figure 7:[/B] Group Policy settings for configuring event logs. [/INDENT][B]Searching and Filtering Events[/B]

While scrolling through the Event console lets you easily examine the most recent events that have been logged on your system, this quickly becomes impractical on busy systems where event logs are tens of megabytes in size. If you’re looking for instances of a particular kind of event however, you can use the Find and Filter options to speed things up.
Say you want to find all instances of Event ID 4004 in the DNS Server log as shown previously in Figure 1 above. To use the Find feature to accomplish this, right-click on the DNS Server log and select View --> Find, then fill in the Event ID and log name in the Find box:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-81094547858969.gif[/IMG]
[B]Figure 8:[/B] Finding instances of Event ID 4004 in the DNS Server log.[/INDENT]Click the Find Next button and the first instances of this event is displayed in Event Viewer:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-91094547866281.gif[/IMG]
[B]Figure 9:[/B] An instance of Event ID 4004 displayed in Event Viewer.[/INDENT]Then click Find Next to display the next instance of this event, and so on.
The frustrating thing about this approach is that the Find interface is not built directly into the Event Viewer window. So let’s try a different approach and use Filter instead. Right-click the DNS Server log again and select View --> Filter, then fill in the Event ID in the Filter tab of the DNS Server Properties sheet:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-101094547873969.gif[/IMG]
[B]Figure 10:[/B] Filtering the DNS Server log for Event ID 4004.[/INDENT]Click OK and Event Viewer and the only events displayed in the DNS Server log are those having Event ID 4004:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-111094547881422.gif[/IMG]
[B]Figure 11:[/B] All instances of Event ID 4004 are displayed. [/INDENT]From this information we could conclude that this was only a transient problem that happened a couple of weeks ago when we rebooted the DNS server.
[B]Viewing Events on Remote Systems[/B]

Event Viewer also lets you connect to remote systems to view their event logs. The procedure is simple: right-click on the root (top) node in the console tree of Event Viewer and select Connect To Another Computer:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-121094547889719.gif[/IMG]
[B]Figure 12:[/B] Connecting to a remote computer to view its event logs.[/INDENT]Then either type the name (NetBIOS or FQDN) of the remote computer or click Browse to find it in Active Directory. Click OK to connect:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-131094547898063.gif[/IMG]
[B]Figure 13:[/B] Can’t connect to a remote computer to view event logs.[/INDENT]Oops, can’t connect! And the error message is cryptic. What went wrong? Typically this error message either indicates one of the following:

[LIST][*]You are not logged on with an account that has local Administrator access to the remote machine (a Domain Admins account should work).[*]The Remote Registry service is not running or has been disabled on the remote machine.[/LIST]
Correct the situation and you should be able to connect to the remote machine and view its event logs.
[B]Using EventCombMT.exe[/B]

In a [URL="http://www.windowsecurity.com/pages/article.asp?id=1362"]previous article[/URL] on WindowsSecurity.com we looked at the Account Lockout and Management Tools (ALTools.exe) download from Microsoft. One of these tools is EventCombMT.exe, which can be used to consolidate event logs from multiple computers into a single location for analysis. To use this tool double-click on EventCombMT.exe in the folder where you installed it, then specify the domain, servers, and kinds of events you want to find. For example, say you want to find all W32Time events on two servers (TEST230 and TEST235) in the testtwo.local domain:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-141094547904969.gif[/IMG]
[B]Figure 14:[/B] Using EventCombMT to search for W32Time events on two servers.[/INDENT]Click Search and when the operation is finished a folder will open up displaying the results files generated:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-151094547911938.gif[/IMG]
[B]Figure 15:[/B] Results files generated by our EventCombMT search.[/INDENT]Double-clicking on one of the two server files displays a comma-delimited list of W32Time events for that server:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-161094547919719.gif[/IMG]
[B]Figure 16:[/B] Comma-delimited list of W32Time events on server TEST230.[/INDENT]You could then import these files into Excel to consolidate them for further analysis. EventCombMT also has a number of built-in queries you can use for common tasks like searching for locked-out accounts:
[INDENT] [IMG]http://www.windowsnetworking.com/img/upl/Event-Logs-Figure-171094547926594.gif[/IMG]
[B]Figure 17:[/B] Searching for locked-out accounts using EventCombMT.[/INDENT]

[/LEFT]