Internet Information Server (IIS) – Application Pool – Tracking – Day 2

Background

This is the second post on our series on tracking the status of IIS’s Application Pool.

Lineage

Here is our initial post:

  1. Internet Information Server (IIS) – Application Pool – Tracking
    Link

 

TroubleShooting

Event Viewer

Log Parser Studio

Queries

Query – Get All WAS Entries
Query
SELECT TOP 1000

         TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') as TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Message
  
from  '[LOGFILEPATH]'

where ( SourceName = 'WAS' ) 
 
ORDER BY
           ComputerName
         , TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') DESC

Output

 

Query – Get WAS Entries – Application Pool Disabled
Query


SELECT TOP 1000

         TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') as TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Message
  
from  '[LOGFILEPATH]'

where ( SourceName = 'WAS' ) 
 
and ( Message like '%disable%' ) 

ORDER BY
           ComputerName
         , TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') DESC
Output

 

Summary

There are a few entries bearing the Source WAS in Windows System Event Viewer.
Inclusive are :

  1. A process serving application pool ‘DefaultAppPool’ failed to respond to a ping. The process id was ‘6208’.
  2. A process serving application pool ‘DefaultAppPool’ suffered a fatal communication error with the Windows Process Activation Service. The process id was ‘13844’. The data field contains the error number.
  3. A worker process with process id of ‘21412’ serving application pool ‘DefaultAppPool’ has requested a recycle because the worker process reached its allowed processing time limit.
  4. Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.

 

The ones most pernicious is “Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool. “

Internet Information (IIS) / Log Parser – Queries – String Pattern Matching

Background

Looking for File I/O Exceptions in the Event Viewer.

 

Query

Sample

Sample 001

Code


SELECT TOP 100 
 
         TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Mesg
       , Strings as Strings
       , EXTRACT_TOKEN(Strings,1,'|') AS AppName
       , EXTRACT_TOKEN(Strings,2,'|') AS AppVersion
       , EXTRACT_TOKEN(Strings,3,'|') AS S3
       , EXTRACT_TOKEN(Strings,4,'|') AS Module
       , INDEX_OF(Message, 'System.IO.IOException') as indexOf
       , case INDEX_OF(Message, 'System.IO.IOException') 
            when 0 then 'N'
            when NULL then 'N'
            else 'Y'
         end as IOE
       , CASE strcnt(Message, 'System.IO.IOException')
             when 0 then 'No'
             else 'Yes'   
         end as IOException
 
from  '[LOGFILEPATH]'
 
WHERE ( EventType = 1 OR EventType = 2 )

and    INDEX_OF(Message, 'System.IO.IOException') > 0

 
ORDER BY
         TimeGenerated DESC


Output

 

Explanation

  1. INDEX_OF
    • We use INDEX_OF to find the position of the sought string in the Message column
      • When the column contains System.IO.IOException the query returns the starting position of the found pattern
      • When not found, null is returned
  2. STRCNT
    • We invoke STRCNT to count number of matches
      • When String not found, 0 return
      • When matched, number of matches

 

References

  1. StackOverflow
    • Log Parser Case Statement
      Link

 

Internet Information Server (IIS) – Application Pool – Tracking

Background

We have a website that has been going offline.

TroubleShooting

MS Windows

Event Viewer

We started looking at the Event Viewer.

Windows Logs – System

Events

Here is what we see in the Event Viewer:

 

Source – WAS, Event ID = 5011

General
Image

 

Textual

A process serving application pool ‘DefaultAppPool’ suffered a fatal communication error with the Windows Process Activation Service.
The process id was ‘1720’. The data field contains the error number.

Details
Image

 

References

  1. IIS Windows Process Activation Service (WAS) > IIS Application Pool > IIS Application Pool Availability
    • Event ID 5011 — IIS Application Pool Availability
      Link

Summary

It looks like something is knocking our Application\Application Pool down.

 

 

WannaCry  – RansomWare – Patching MS Windows 2003 / Windows XP

Background

I have for some Windows XP and Windows 2003 boxes in my Lab.

And, since they are very vulnerable to SMB vulnerability exploited by Wanna Cry, let us go patch them.

 

KB4012598

The fix for Windows XP and Windows 2003 is packaged as KB4012598.

 

Download URL

The patch is available @

  1. Windows 2003
    • Security Update for Windows Server 2003 ( KB 4012598)
      Link

Download Patch

Browser Choice

Internet Explorer

On Windows 2003 box, launched IE and tried downloading patch for Windows 2003.

Agin, here is the URL attempted.

Chrome

Chrome show contents, avails download button, and was able to successfully download.

 

Apply Patch

Outline

  1. Access Saved Folder
  2. Launch downloaded file
    • As this is a downloaded file, prompted as to whether it is OK to run file
    • On the Welcome screen, click the Next button
    • On the “License Agreement” screen, choose the “I Agree” button
    • Keep an eye on the “Updating Your System” screen

Images

Images – Open File – Security Warning

Images – Welcome

Images – License Agreement

Images – Updating Your System

 

Review Applied Patches

Let us review Applied patches.

Outline

  1. Launch Control Panel
  2. Access the Add and Remove Programs applet
  3. Stay and choose the “Change or remove Programs” group box
    • Choose to “Show updates”
    • In the “Sort By” drop-down, choose Name
    • Review entries listed under “Windows Server 2003 – Software Updates
    • Before applying patch
      • The last update was in Sept 7th, 2015
    • Post applying patch
      • Last applied Patch KB 40122598
      • Patch applied on May 19th, 2017

Images

Image – Before

Image – Before – Top

Image – Before – Bottom

 

 

Image – After

 

Additional Reading

As always there is an awful lot of commentary out there:

  1. Talos
    • Martin Lee, Warren Mercer, Paul Rascagneres, and Craig Williams
      • Player 3 Has Entered the Game: Say Hello to ‘WannaCry’
        Link
  2. Lawrence Abrams
    • Bleeping.com
      • How to remove the WannaCry & Wana Decryptor Ransomware
        Link
  3.  Comae.io
    • Matt Suiche, Hacker, Microsoft MVP, Founder of @comaeio — Co-Founder of @CloudVolumes (now @VMWare)
      • WannaCry — The largest ransom-ware infection in History
        Link
  4. United States Computer Emergency Readiness Team ( US-CERT )
    • Indicators Associated With WannaCry Ransomware – Alert (TA17-132A)
      Link

 

SQL Server – Analysis Services – Cannot Connect ( Microsoft.AnalysisServices.AdomdClient )

Background

Users are having problems connecting to the Analysis Services Server.

 

Error Message

Image

Text


TITLE: Connect to Server
------------------------------

Cannot connect to ....sql01.

------------------------------
ADDITIONAL INFORMATION:

A connection cannot be made. Ensure that the server is running. (Microsoft.AnalysisServices.AdomdClient)

------------------------------

A connection attempt failed because the connected party did not properly respond after a period of time, 
or established connection failed because connected host has failed to respond z.y.76.188:2383 (System)


Text


===================================

Cannot connect to d-isql01.

===================================

A connection cannot be made. Ensure that the server is running. (Microsoft.AnalysisServices.AdomdClient)

------------------------------
Program Location:

at Microsoft.AnalysisServices.AdomdClient.XmlaClient.GetTcpClient(ConnectionInfo connectionInfo)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.OpenTcpConnection(ConnectionInfo connectionInfo)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.OpenConnection(ConnectionInfo connectionInfo, Boolean& isSessionTokenNeeded)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.Connect(ConnectionInfo connectionInfo, Boolean beginSession)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Connect(Boolean toIXMLA)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Microsoft.AnalysisServices.AdomdClient.AdomdConnection.IXmlaClientProviderEx.ConnectXmla()
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.ConnectToXMLA(Boolean createSession, Boolean isHTTP)
at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.Open()
at Microsoft.SqlServer.Management.SqlStudio.Explorer.ObjectExplorerService.ValidateConnection(UIConnectionInfo ci, IServerType server)
at Microsoft.SqlServer.Management.UI.ConnectionDlg.Connector.ConnectionThreadUser()

===================================

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond xx.xx.xx.xx:2383 (System)

------------------------------
Program Location:

at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port)
at Microsoft.AnalysisServices.AdomdClient.XmlaClient.GetTcpClient(ConnectionInfo connectionInfo)


TroubleShooting

Server

Resource Monitor

If the server has Microsoft’s Resource Monitor built-in let us use to it to review Network Ports.

Image

Explanation

  1. Listening Ports
    • Image :-
      • msmdsrv.exe
    • Address :-
      • IPV4 unspecified
      • IPV6 unspecified
    • Port :-
      • 2383
    • Protocol
      • TCP
    • Firewall Status
      • Allowed, not restricted

 

Client

netstat

Script


netstat -an | find "SYN_SENT"

Sample Output

Explanation

  1. Request was sent to port 2383
  2. Status is SYN_SENT
    • SYN_SENT means waiting on a reply

Remediation

Open up TCP Port 2383.

 

Review

Once the Network Port, 2383, is opened up retry access with SSMS.

 

Network Flow

Also, it makes sense to review the Network Traffic to get a full list of ports to be opened.

WireShark

Output

 

Explanation

Seems Analysis Services exclusively relies on port 2383.

Windows – Event Viewer Parsing Through Log Parser Studio

Background

Need to parse MS Windows Event Logs.

One of the ways to do so is to use Log Parser Studio.

 

Event Viewer

Let us save the events unto the File System.

Outline

  1. Launch Event Viewer
  2. Select the Logs you want ( Application / System / Security )
  3. Right click on the Logs and from the drop down menu, choose “Save All Events As …
  4. Choose Folder And Filename
  5. The file is saved with an extension of “Event Files (*.evtx )

 

Images

Launch Save Event As

Choose Filename

 

Log Parser Studio

Outline

  1. Launch Log Parser Studio
  2. Choose Log Type: EVTLOG
  3. Enter Query
  4. Execute Query

 

Choose Log Type : EVTLOG

Sample Queries


/*  Find top 1000 warnings and errors in the Application Log 
    Levels: 1=Error, 2=Warning                                
*/
SELECT TOP 1000 
             TimeGenerated
           , ComputerName
           , EventCategoryName
           , EventTypeName
           , EventID
           , SourceName
           , Message
FROM 'C:\Temp\04_WindowsLogs_Applications_20170518_0403PM.evtx'
WHERE ( EventType = 1 OR EventType = 2 )
AND   (
               (SourceName like 'ASP%' )
            or (SourceName = '.NET Runtime' )
            or (SourceName = 'Application Error' )
      )
ORDER BY TimeGenerated DESC


Click Execute Button

Click on the Execute Button – The Read icon with the exclamation mark!

 

Sample Output

 

Export

Outline

  1. In Log Parser Studio, use menu File \ Export \ Output as .CSV
  2. In the “Choose Location to save CSV File” window, please specify folder and file name

 

Images

File \ Export \ “Output as .CSV”

 

Choose Location to save CSV File

Excel File

 

SQL Server – Extended Event – Data Storage – Change Folder / File Name

Background

Due to Storage Constraints having to change the folder that Extended Events are captured in.

 

GUI

I know it is not possible to do so via GUI.

As shown below, just as your author, the properties are dimmed.

 


Script

So let us script it

Outline

  1. Stop the Extended Event Session
  2. Drop the Extended Event Session target
    • In our case, we are using the adopted default target name of package0.event_file
  3. Add a new Target
    • Keeping everything from the old target
    • But, changing the targeted folder
  4. Restart the Extended Event Session

Code




	ALTER EVENT SESSION [performance]   
	ON SERVER  
	STATE = stop;  

	ALTER EVENT SESSION [performance]  
		ON SERVER 
		DROP TARGET package0.event_file;

	ALTER EVENT SESSION [performance]  
		ON SERVER 
		ADD TARGET package0.event_file
		(
			SET 
				  filename=N'C:\Microsoft\SQLServer\ExtendedEvents\TraceFiles\performance.xel'
				, max_file_size=(100)
				, max_rollover_files=(100)
		)

	ALTER EVENT SESSION [performance]   
	ON SERVER  
	STATE = start;  

GO