IIS Logs / Log Parser Studio – Aggregated Hits per Server

Background

Our monitoring team has developed and rolled out scripts for monitoring our web farm.

And, we are getting alerts through email.

Quite a lot of emails are coming across and wanted to see if they are coming from same host or a combination of hosts.

 

Emails

Looked at the emails and they happen to be coming from same host.

And, so will have to engage our Network team and see how the Load Balancer is configured.

Is there a prospect that more traffic is being directed at the failing node?

Network Load Balancer

As we prepared to go to the Network Load Balancer team took the opportunity to take gather and query the IIS Logs, as well.

 

TroubleShooting

Log Parser Studio

Query


SELECT 
            To_String(date, 'yyyy-MM-dd') as dated

          , sc-status as status

          , sum (
                    case s-ip
                        when '10.0.4.25' then 1
                        else 0
                   end
               ) as S1

          , sum (
                    case s-ip
                        when '10.0.4.26' then 1
                        else 0
                   end
               ) as S2


          , sum (
                    case s-ip
                        when '10.0.4.27' then 1
                        else 0
                   end
               ) as S3

         , sum (
                    case s-ip
                        when '10.0.4.28' then 1
                        else 0
                   end
               ) as S4

          , min(TO_TIMESTAMP(date, time)) as tsRecordedMin


          , max(TO_TIMESTAMP(date, time)) as tsRecordedMax


FROM '[LOGFILEPATH]' 


where   (


           (

             TO_TIMESTAMP(date, time) 
                     between timestamp('2017/08/02 10:30:00', 'yyyy/MM/dd hh:mm:ss')  
                          and timestamp('2017/08/02 17:20:00', 'yyyy/MM/dd hh:mm:ss')
           )

       )

/*

	and  c-ip not in ('10.0.4.141')
	
*/

group by
         date
       , sc-status


order by
           dated 
         , status



Output

Time Range – 1 ( August 2nd 10:30 AM – 5:20 PM )

Results

Explanation
  1. It is difficult to make case that traffic is exhaustively being waded into a specific host

Time Range – 2 ( August 8th 5:13 PM – 8:40 PM )

Results

Explanation
  1. In our second time slot, 4700 records bearing HTTP 200 is right around average

Summary

At this time it is likely that the sufferance we are seeing with this specific host is not due to outside pressure, but internal to the host itself.

 

SSRS – Setting up Smart Host

 

Background

A year or so ago we setup subscriptions to a couple of Reports that we are providing through SQL Server Reporting Services.

The subscriptions go out through daily email.

Ever so often things just break.

I was recently informed that Emails have not been going for over a week now.

Last time I blamed it on other processes that are using that same host.

Hoping today I can do same and go on about my business.

But, no such luck.

 

Environment

Here is our topology

  1. Reporting Services
    • Reporting Services is running on a local server in our intranet.
  2. Database Server
    • Database Server is running in our Colocation’s Data Center
  3. Email Server
    • The email server is Microsoft’s Office365.com

 

Troubleshooting

Thinking out loud

As always don’t have a clue what changed.

Could it be…

  1. Tightened Security
    • Can emails only go out from certain hosts
    • Do I need an actual username and password combination
    • Firewall
      • Local
        • Is it Windows Firewall
      • Corporate
        • Is it a Corporate Firewall
    • Is it Antivirus Configuration

 

Remediation

Proposal

Not sure what is getting in the way of SSRS getting the emails out.

But, a likely workaround is use a local functional SMTP server as a bridge.

 

Local SMTP Server

Installation

Launch “Server Manager” and we will choose to add “SMTP Server Tools” as a Feature.

 

Step

  1. Tab – Features
    • If “SMTP Server” feature is not checked, please place a check mark next to it
    • Dependencies
      • The “Add role services and features required for SMTP Server” window appear
        • The features listed are “Web Server (IIS)” and “Remote Server Administrative Tools”
  2. Tab – Web Server ( IIS )
    • Shows Web Server literature
  3. Tab – Confirmation
    • Confirmation that IIS and Remote Server Administrative Tools will be augmented
  4. Tab – Progress
    • As installation is proceeding each step is chronicled
  5. Tab – Results
    • The status of each component installed is noted

Images

Add Features Wizard – Select Features
Initial Screen

Before Adding “SMTP Server….

SelectFeatures_SMTP_201708087_0420PM

 

Post Checking “SMTP Server”

Adding “SMTP Server “….

SelectFeatures_SMTP_201708087_0421PM

Add Features Wizard – Add role services and features required for SMTP Server?

Dependencies are listed.

And, they include Web Server ( IIS ) and Remote Server Administrator Tools.

SelectFeatures_AddFeaturesWizard_201708087_0420PM

 

Web Server ( IIS)

Components :-

  1. Internet Information Services ( IIS ) 7.0
    • ASP.Net
    • Windows Communication Foundation

 SelectFeatures_SMTP_WebServer_IIS_201708087_0421PM

 

Confirm Installation Selections

Confirm Installation.

In our case:

  1. Web Server ( IIS )
    • Health and Diagnostics
      • ODBC Logging
    • Remote Server Administrator Tools
      • SMTP Server Tools

 

SelectFeatures_SMTP_WebServer_IIS_RoleServices_Confirmation_201708087_0423PM

Installation Progress

Installation is progress…

 

SelectFeatures_SMTP_InstallationProgress_201708087_0424PM

 

 

Installation Results

Installation Succeeded.

SelectFeatures_SMTP_WebServer_IIS_RoleServices_InstallationResults_201708087_0435PM

 

 

Configuration

Customization

  1. Tab – General
    • Enable Logging
      • It is most useful to turn on logging during initial setup and follow-up troubleshooting sessions
  2. Tab – Access
    • Group – Connection
      • Select which computers may access this session
        • All, except the list below
    • Group – Relay Restrictions
      • Only the list below
        • Self ( for now )
          • 127.0.0.1
  3. Tab – Messages
    • Send copy of non-delivery report to
      • Mail Administrator
        • Hopefully a monitored distribution list
    • Bad mail directory
      • Default
        • C:\Bad Mail
      • Non-system drive folder
        • Hopefully, you take the opportunity to change the folder to a non-system drive
  4. Tab – Delivery
    • Group box – Outbound Security
      • Authentication Choices
        • Anonymous
        • Basic Authentication
        • Windows Integration
      • In our case :-
        • Anonymous ( NO )
          • Are trying to get away from Anonymous as our hosting platform, Microsoft Office, requires user authentication
        • Integrated Windows Authentication ( NO )
          • We do not have cross-domain relationship between us and Microsoft’s Hosted Outlook
        • Basic Authentication ( YES )
      • TLS
        • We enabled TLS
    • Group box – Outbound Connections
      • TCP Port
        • 587
          • This is the default mail submission port. When a mail client or server is submitting an email to be routed by a proper mail server, it should always use this port.
            Unless you’re explicitly blocked by your upstream network or hosting provider.
            This port, coupled with TLS encryption, will ensure that email is submitted securely and following the guidelines set out by the IETF”

            John Carl Villanueva ( Link )
    • Group box – Advanced Delivery
      • Fully Qualified Domain Name
        • Especially for domains that have SPFs set up
      • Smart Host
        • smtp.office365.com
      • Attempt direct delivery before sending to smart host
        • Unchecked
      • Perform reverse DNS lookup on incoming messages
        • Unchecked
  5. LDAP Routing
    • Not going to need to use LDAP Routing for user authentication
  6. Grant Operator permissions to these Windows User Accounts

 

Screenshot

SMTP Virtual Server – Properties – General

General_20170808_0840AM

 

 

SMTP Virtual Server – Properties – Access

Access_20170807_0752PM

 

SMTP Virtual Server – Properties – Access – Connection
Initial

Access_ConnectionControl_20170807_0753PM

 

SMTP Virtual Server – Properties – Access – Relay Restrictions
Initial

Access_RelayRestrictions_20170807_0754PM

 

Add Computer

List

  1. Single Computer
    • IP address:- 127.0.0.1

 

Access_RelayRestrictions_AddComputer_20170807_0755PM

Completed

Access_RelayRestrictions_Computer_20170809_0114PM

 

SMTP Virtual Server – Properties – Messages
Initial

Messages_20170807_0756PM

 

Completed

Messages_20170807_0757PM (BrushedUp)

 

 

SMTP Virtual Server – Properties – Delivery
Initial

Delivery_20170807_0757PM

 

SMTP Virtual Server – Properties – Delivery – Outbound Security
Initial

Delivery_OutboundConnections_20170807_0444PM

 

 

Complete

Delivery_OutboundSecurity_20170807_0758PM (BrushedUp)

 

SMTP Virtual Server – Properties – Delivery – Outbound Connections
Initial

Delivery_OutboundConnections_20170807_0445PM

 

Completed

Delivery_OutboundConnections_20170807_0758PM

 

 

SMTP Virtual Server – Properties – Delivery – Advanced Delivery
Initial
Completed

AdvancedDelivery_20170807_0759PM [BrushedUp]

 

Conclusion

We have an SMTP Server setup.

We will come back and unit test it out and once verified, we will point Sql Server Reporting Services ( SSRS ) to route emails through it.

 

References

  1. jscape
    • John Carl Villanueva
      • Still Confused With SMTP Ports? Read This
        Link

IIS – TroubleShooting – High CPU Utilization – SysInternals/Process Explorer – Day 01

Background

In our last post we spoke on how to troubleshoot IIS using Microsoft’s own Debug Diagnostic Tool ( DebugDiag).

That tool works by first getting a dump of a wobbled IIS process and then training the tool against that up.

Using a set of rules the dump is inspected.

Familiar issues are checked for and if they occur they are grouped and cited.

 

SysInternals / Process Explorer

Introduction

In this post we will use a more rudimentary tool and point it at a running IIS process and do the inspection ourselves.

 

Download

If you don’t have a recent version of SysInternal’s Process Explorer, please download it from here.

The current version is 16.12

Install

No need to install, just run it.

Usage

Start Mode

If not started as an “Administrator”, you will miss some functionalities.

Mode :- User

Mode :- Administrator

Explanation

  1. Tabs
    • Additional Tabs are shown when running as an Administrator
      • Job
      • .Net Assemblies
      • .Net Performance

Application :- W3wp.exe

Disk and Network

Image

Explanation
  1. Get a feel of IO requirements and throughput
    • Network I/O ( Receives VS Sends )
    • Disk I/O  ( Reads VS Writes )

GPU Graph

Image

Explanation
  1. GPU Graph is empty
    • Not doing anything with Graphics Coprocessor

 

Threads

Image

Explanation

Listed are

  1. Number of threads
  2. Individual Threads
    • Thread ID ( TID )
    • CPU ( Ordered by CPU% usage )
    • Start Address
      • clr.dll!DllRegisterServerInternal+0x1f060

TCP/IP

Image

Explanation
  1. Using ephemeral ports on localhost to communicate with backend DB/SQL Server ( ms-sql-s)

Security

Image

Explanation
  1. User :- IIS APPPOOL\DefaultAppPool
  2. Groups
    1. BUILTIN\IIS_ISUSRS
    2. BUILTIN\USERS
    3. CONSOLE LOGON

 

Environment

Image

Explanation

Review:

  1. Processor Identifier
  2. Number of processors
  3. User Domain and Name

 

Job

Image

Explanation

List of Jobs and Processes.

.Net Assemblies

Image

Explanation
  1. Entity Framework
    • EntityFramework
    • EntityFramework.Extended
    • EntityFramework.SQLServer
  2.  Glimpse
    • Glimpse.Ado
    • Glimpse.AspNet
    • Glimpse.Core

.Net Performance

In the .Net Performance Tab, we are able to track different .Net Performance counters.

Inclusive :

  1. .Net CLR Exceptions
    • # of Exceps thrown
      • Number of Exceptions thrown
  2. .Net CLR Threads & Locks
    • Total # of Contentions
    • Current Queue Length
    • Queue Length Peak
  3. .Net CLR Loading
    • Total # of Load Failures

 

Summary

It goes without saying that SysInternal’s Process Explorer gives unending information about the going ons of a running process.

Careful attention to details is needed to gather pertinent data.

It is also helpful to work with actual end users to get an idea of what they will likely doing during various captured events timeline.

An, go back to Development and gauge their understanding of how various APIs stress the system.

 

References

  1. Microsoft Technet
    • Chad Schultz – MSFT
      • How to use Sysinternals Process Monitor and Process Explorer to Troubleshoot SharePoint
        Link
  2. Piers ( Cup(Of T )
    • Production Debugging for Hung ASP.Net 2 applications – a crash course
      Link

 

IIS – TroubleShooting – High CPU Utilization – Debug Diagnostic Tools v2.1 – Day 1

Background

Needing to do a deep drive into troubleshooting IIS Servers.  As part of our troubleshooting exercises we will cover one of Microsoft’s own tool, Debug Diagnostic Tools.

 

Debug Diagnostic Tools

Version

The tool’s team blog is here.

And, the current version is 2.1 and it is available here.

Details

The version was availed on 2015-Nov-13.

Requirements

Link

We are running MS Windows 2012 R2.

And, the bitness is 64-bit.

 

Install

Image

Welcome

End-User License Agreement

End-User License Agreement – Initial

End-User License Agreement – Completed

Custom Setup

 

Ready to install

Installing….

Completed

Debug File

Before we actually use the debug tool, let us go generate dump files from a working w3wp process.

Identify Process

Code


c:\windows\system32\inetsrv\appcmd list wp

 

Output

Dump file

Task Manager

On the IIS machine, launch Task Manager, access the Processes tab, select the “IIS Worker Process”.

Right click on your selection and choose “Create dump file“.

 

Use Debug Diagnostic Tool

Start Menu

Here is the Start Menu,

We will choose the menu option “Debug Diagnostics Tool 2” \ “Debug Diag2 Analysis”.

 

Data Files

We will click the “Add Data Files” button and navigate to the folder where we kept the dump file that was created earlier.

Here is the screenshot once we have added our dump file.

 

Analysis Rules

From the list of rules, we chose “Default Analysis” “CrashHangAnalysis“.

Start Analysis

Please click the “Start Analysis” button.

Reports

Analysis Report – Dashboard

Here we noted 3 Warnings.

 

Sections

Previous .Net Exceptions Reports ( Exception in all .Net Heaps )

Image

Tabulated

 

Exception Count Message Stack Trace
System.Exception 1 <none>  ;
System.OutOfMemoryException 1 <none>  ;
 

System.StackOverflowException

1 <none>  ;
System.ExecutionEngineException 1 <none>  ;
System.Threading.ThreadAbortException 2 <none>  ;
System.NullReferenceException 3 Object reference not set to an instance of an object.  garage.Models.Articulation.GeneratedPrefixAgreement+<>c__DisplayClass1e.<Generate>b__b(garage.Models.Articulation.Courses.CourseToCourseArticulation)
System.Linq.EnumerableSorter`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].ComputeKeys(System.__Canon[], Int32)
System.Linq.EnumerableSorter`1[[System.__Canon, mscorlib]].Sort(System.__Canon[], Int32)
System.Linq.OrderedEnumerable`1+<GetEnumerator>d__1[[System.__Canon, mscorlib]].MoveNext()
garage.Models.Articulation.GeneratedPrefixAgreement.Generate(CodeFirstMembershipSharp.DataContext)
LinqKit.Extensions.ForEach[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Action`1<System.__Canon>)
garage.Infrastructure.GeneratedAgreementFactory.Generate(garage.Models.Articulation.GeneratedAgreement)
 

System.Reflection.TargetInvocationException

3 Exception has been thrown by the target of an invocation.  System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])
System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(System.Reflection.MethodInfo, System.Object, System.Object[])
 

Hangfire.Server.JobPerformanceException

3  

An exception occurred during performance of the job.

  Hangfire.Server.CoreBackgroundJobPerformer.InvokeMethod(System.Reflection.MethodInfo, System.Object, System.Object[])
Hangfire.Server.CoreBackgroundJobPerformer.Perform(Hangfire.Server.PerformContext)
Hangfire.Server.BackgroundJobPerformer+<>c__DisplayClass8_0.<PerformJobWithFilters>b__0()
Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(Hangfire.Server.IServerFilter, Hangfire.Server.PerformingContext, System.Func`1<Hangfire.Server.PerformedContext>)
Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(Hangfire.Server.PerformContext, System.Collections.Generic.IEnumerable`1<Hangfire.Server.IServerFilter>)
Hangfire.Server.BackgroundJobPerformer.Perform(Hangfire.Server.PerformContext)
Hangfire.Server.Worker.PerformJob(Hangfire.Server.BackgroundProcessContext, Hangfire.Storage.IStorageConnection, System.String)

 

Thread Report

Image

Explanation
  1. 54 Threads
    • 32% of all threads have this same call stack

 

clr!Thread::intermediateThreadProc ( 54 Threads )

Image

 

Explanation

mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)+bf 
garage.Models.garageSystem.Menu.AddItemForRoles(CodeFirstMembershipSharp.User, System.String, System.String, Boolean, System.String[], Boolean, System.String)+d5 
garage.Controllers.HomeController.BuildMenu()+6c 
garage.Controllers.BaseController.Initialize(System.Web.Routing.RequestContext)+1956 
System.Web.Mvc.Controller.BeginExecute(System.Web.Routing.RequestContext, System.AsyncCallback, System.Object)+179 
System.Web.Mvc.MvcHandler.b__4(System.AsyncCallback, System.Object, ProcessRequestState)+37 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncVoid`1[[System.Web.Mvc.MvcHandler+ProcessRequestState, System.Web.Mvc]].CallBeginDelegate(System.AsyncCallback, System.Object)+41 
System.Web.Mvc.Async.AsyncResultWrapper+WrappedAsyncResultBase`1[[System.Web.Mvc.Async.AsyncVoid, System.Web.Mvc]].Begin(System.AsyncCallback, System.Object, Int32)+aa 
System.Web.Mvc.MvcHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object)+23b 
System_Web_ni!System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()+132 
System_Web_ni!System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)+9d 
System_Web_ni!System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)+5dc 
System_Web_ni!System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)+79 
System_Web_ni!System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)+e0 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+407 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+14 
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+5b 
System_Web_ni!DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)+7e 
[[InlinedCallFrame] (System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion)] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatusByRef) 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)+5e6 
System_Web_ni!System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)+14 
System_Web_ni!DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)+5b 
[[ContextTransitionFrame]] 


 

clr!Thread::intermediateThreadProc ( 16 Threads – 9% of all threads )

Image

Explanation
  1. 16 Threads
    • Thread is waiting in a waitOne

 

clr!Thread::intermediateThreadProc ( 8 Threads – 4% of all threads )

Image

Textual

Entry point   clr!Thread::intermediateThreadProc 
Create time   8/2/2017 3:30:18 PM 
Time spent in user mode   0 Days 00:00:00.093 
Time spent in kernel mode   0 Days 00:00:00.031 

This thread is waiting for .net garbage collection to finish.
The current set of scripts were not able to determine which thread induced GC.
The garbage collector thread wont start doing its work till the time the threads which have pre-emptive GC disabled have finished executing. 
The following threads have pre-emptive GC disabled 28,51,52,53,54,55,56,57,66,67,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,155,156,157,158,159,160,161,162,163,165,166,167,

Call Stack

ntdll!NtWaitForSingleObject+a 
KERNELBASE!WaitForSingleObjectEx+94 
clr!SVR::gc_heap::wait_for_gc_done+134 
clr!SVR::gc_heap::wait_for_gc_done+cb 
clr!CLREventBase::WaitEx+7c 
clr!SVR::gc_heap::bgc_thread_function+a7 
clr!Thread::intermediateThreadProc+86 
kernel32!BaseThreadInitThunk+22 
ntdll!RtlUserThreadStart+34 


Explanation

This is a very important group:

  1. This thread is waiting for .net garbage collection to finish.
  2. The garbage collector thread wont start doing its work till the time the threads which have pre-emptive GC disabled have finished executing.
  3. The following threads have pre-emptive GC disabled…..

 

Summary

Here is a quick compilation of what the tool is informing us of:

  1. We have exceptions that are not gracefully handled
    • System.NullReferenceException
      • Entity Framework calls
        • Uncompleted DB Calls that likely timed out
        • .Net code should review return code before trying to access returned dataset
    • System.Reflection.TargetInvocationException
    • Hangfire.Server.JobPerformanceException
      • Hangfire background process when accessing persistent backend
  2. Seeming contention issue accessing Dictionary Collection?
    • Stack Trace :-
      • mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)+bf
        garage.Models.garageSystem.Menu.AddItemForRoles(CodeFirstMembershipSharp.User, System.String, System.String, Boolean, System.String[], Boolean, System.String)+d5
        garage.Controllers.HomeController.BuildMenu()+6c
  3. Garbage Collection Issues
    • Stack Trace :-
      • This thread is waiting for .net garbage collection to finish.The current set of scripts were not able to determine which thread induced GC.  The garbage collector thread wont start doing its work till the time the threads which have pre-emptive GC disabled have finished executing. The following threads have pre-emptive GC disabled

 

References

  1. Developer Network
    • Mourad Lagdas
      • How to Use the Debug Diagnostic Tool v1.1 (DebugDiag) to Debug User Mode Processes
        Link
  2. Microsoft Developer
    • Michael Friis ( friis[at]microsoft.com )
      • Which w3wp.exe PID corresponds to which application pool ?
        Link
  3. Hangfire
    • Documentation \ Background processing \ Dealing with exceptions
      Link
  4. WhiteSites.com
    • Debugging Faulting Application w3wp.exe Crashes
      Link

SQL Server – Reporting Services – Connecting Locally – Day 1

Background

Hardening security via applying SSL Certs on a couple of Reporting Services Hosts and wanting to test them on same host, but “No Go“.

 

TroubleShooting

Windows Event Viewer

Checked Windows Event Viewer

Security

Security – Headers

Image

Tabulate
  1. Event ID = 4625
    • Keywords :- Audit Failure
    • Source :- Microsoft Windows Security auditing
    • Event ID :- 4625
    • Task Category :- Logon

Security – Details

Image

 

Tabulate
  1. Event ID = 4625
    • Security ID :- NULL SID
    • Logon Type :- 3
      • Logon Type 3 is Network
    • Status :- 0xC000006D
    • Event ID :- 4625
    • Task Category :- Logon

Summary

Basically, we were prompted thrice to enter our username and password. But, unable to connect.

 

 

Internet Explorer

Checked to make that we are still unable to connect when we run in Administrator Mode.

Task Manager

To verify that IE is running in Administrator mode launched Task Manager and included the “Elevated” attribute.

Select Columns

Results

Image

Explanation

For each IE Session, we are seeing two processes.
Why two processes each time we start a new IE Session?

 

Remediation

Registry

Outline

There are a couple of options and those are:

  1. BackConnectionHostNames
  2. DisableLoopbackCheck

 

BackConnectionHostNames

Worknotes

Launch regedit and access the registry key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0”.

Search for BackConnectionHostNames

Add all FQDN that the server’s resource will be self referred to.
Each entry should be entered in its own line.

  1. Type :- REG_MULTI_SZ
  2. Data :- ????

Images

Adding Entry

Entry Added

 

DisableLoopbackCheck

Worknotes

Launch regedit and access the registry key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa“.

Search for DisableLookback.

Make sure it exists as:

  1. Type :- REG_DWORD
  2. Data :- 1

Image

Script

Script – BackConnectionHostNames


@echo off
@echo on

set "_registryBranch=HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0"
set "_registryItem=BackConnectionHostNames"
set "_registryDataType=REG_MULTI_SZ"

rem ****************************************************************************************
rem please change to match your domain name
rem ****************************************************************************************
set "_domainName=labdomain.org"

set "_registryValue=%COMPUTERNAME%.%_domainName%"

echo "Value - Current"
reg query %_registryBranch% /v %_registryItem%

reg add %_registryBranch% /v %_registryItem% /t %_registryDataType% /d %_registryValue% /f

echo "Value - New"
reg query %_registryBranch% /v %_registryItem%

Script – DisableLoopbackCheck


@echo off
rem set "_registryBranch=HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa"
set "_registryBranch=HKLM\SYSTEM\CurrentControlSet\Control\Lsa"
set "_registryItem=DisableLoopbackCheck"
set "_registryDataType=REG_DWORD"
set "_registryValue=1"

echo "Value - Current"
reg query %_registryBranch% /v %_registryItem%

reg add %_registryBranch% /v %_registryItem% /t %_registryDataType% /d %_registryValue% /f

echo "Value - New"
reg query %_registryBranch% /v %_registryItem%

Summary

This problem is nothing.  It has been in the OS since Windows 2003.

And, so I suppose it is not really a problem, I just wished it was surfaced differently; than having to type my password thrice and still can’t get in.

 

References

  1. Microsoft
    • Microsoft Support
      • You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or a later version
        Link
  2. Nik Patel
    • Disable the Loopback Check for Specific Host Names on all SharePoint Web and Application Servers
      Link
  3. Michael Hanes
    • Use BackConnectionHostNames instead of DisableLoopbackCheck in production
      Link
  4. Harber.net
    • DisableLoopbackCheck & SharePoint: What every admin and developer should know
      Link

Internet Information Server (IIS) – Application Pool – Tracking – Day 2

Background

This is the second post on our series on tracking the status of IIS’s Application Pool.

Lineage

Here is our initial post:

  1. Internet Information Server (IIS) – Application Pool – Tracking
    Link

 

TroubleShooting

Event Viewer

Log Parser Studio

Queries

Query – Get All WAS Entries
Query
SELECT TOP 1000

         TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') as TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Message
  
from  '[LOGFILEPATH]'

where ( SourceName = 'WAS' ) 
 
ORDER BY
           ComputerName
         , TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') DESC

Output

 

Query – Get WAS Entries – Application Pool Disabled
Query


SELECT TOP 1000

         TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') as TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Message
  
from  '[LOGFILEPATH]'

where ( SourceName = 'WAS' ) 
 
and ( Message like '%disable%' ) 

ORDER BY
           ComputerName
         , TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') DESC
Output

 

Summary

There are a few entries bearing the Source WAS in Windows System Event Viewer.
Inclusive are :

  1. A process serving application pool ‘DefaultAppPool’ failed to respond to a ping. The process id was ‘6208’.
  2. A process serving application pool ‘DefaultAppPool’ suffered a fatal communication error with the Windows Process Activation Service. The process id was ‘13844’. The data field contains the error number.
  3. A worker process with process id of ‘21412’ serving application pool ‘DefaultAppPool’ has requested a recycle because the worker process reached its allowed processing time limit.
  4. Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.

 

The ones most pernicious is “Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool. “

Internet Information (IIS) / Log Parser – Queries – String Pattern Matching

Background

Looking for File I/O Exceptions in the Event Viewer.

 

Query

Sample

Sample 001

Code


SELECT TOP 100 
 
         TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Mesg
       , Strings as Strings
       , EXTRACT_TOKEN(Strings,1,'|') AS AppName
       , EXTRACT_TOKEN(Strings,2,'|') AS AppVersion
       , EXTRACT_TOKEN(Strings,3,'|') AS S3
       , EXTRACT_TOKEN(Strings,4,'|') AS Module
       , INDEX_OF(Message, 'System.IO.IOException') as indexOf
       , case INDEX_OF(Message, 'System.IO.IOException') 
            when 0 then 'N'
            when NULL then 'N'
            else 'Y'
         end as IOE
       , CASE strcnt(Message, 'System.IO.IOException')
             when 0 then 'No'
             else 'Yes'   
         end as IOException
 
from  '[LOGFILEPATH]'
 
WHERE ( EventType = 1 OR EventType = 2 )

and    INDEX_OF(Message, 'System.IO.IOException') > 0

 
ORDER BY
         TimeGenerated DESC


Output

 

Explanation

  1. INDEX_OF
    • We use INDEX_OF to find the position of the sought string in the Message column
      • When the column contains System.IO.IOException the query returns the starting position of the found pattern
      • When not found, null is returned
  2. STRCNT
    • We invoke STRCNT to count number of matches
      • When String not found, 0 return
      • When matched, number of matches

 

References

  1. StackOverflow
    • Log Parser Case Statement
      Link