Apache–jMeter – Test Plan – ASP.Net – Forms Authentication–Troubleshooting

Background

In a previous post we spoke about the steps we undertook to design a test plan for authenticating users connecting to an ASP.Net web site.

In this post, we will shield light on the headwinds that batted us along the way.

 

Headwinds

Outline

  1. Workflow
  2. Thread Group
    • Thread Group Configuration
      • More threads than necessary
  3. HTML Page
    • Hidden fields
    • Entry fields
    • Action or Push button
  4. Component – Cookies Manager
  5. Component – View Results Tree
  6. Component – View Results Table
  7. Web Server
    • HTTP Logs
    • Failed Request Tracing

 

Workflow

This is a very crude drawing …

Workflow_20171026_0533PM

But, it hopefully shows workflow  ..

  1. First HTTP Request Default
  2. Second HTTP Request
    • Use Get Method to request session page
    • Returns to us the session date ( viewstate, eventValidation, viewStateGenerator )
  3. Parse returned Page
    • Using CSS /JQuery parse data and retrieve session data mentioned above
  4. Third HTTP Request
      • Use Post Method to submit user credentials
      • Make sure that session data we parsed earlier is packaged, as well

 

Thread Group

Thread Group Configuration

More threads than Necessary

Images
Thread Group – Configuration @ 7:11 PM

At 7:11 PM, we were hopeful and set up for fifty users, a ramp time of 10 seconds.

And, 2 repetitions.

ThreadGroup_20171024_0714PM

 

Thread Group – Configuration @ 1:03 AM

At 1 AM of the next day, we were humbled to 1 user and a single iteration.

ThreadGroup_20171025_1158AM

 

Explanation

Once we could not successfully authenticate and started adding ViewResultsTree and viewResultsInTable, we started seeing double and some of it was due to the fact that we had more workers than was necessary.

 

HTML Page

Hidden Fields

Images

Explanation

  1. Make a note of all hidden fields
  2. Determine how they are populated
    • Static versus Dynamic
    • Vetted against replay
  3. Encoded ( Yes or No )

 

Cookies Manager

For state management, you will need cookies, server and client side, so please save yourself the headache by enabling them.

Image

Image – Before

HTTPCookieManager_20171025_1152AM

Image – After

HTTPCookieManager_20171026_0459PM

Explanation

  1. Once things are good
    • Clear cookies each iteration
      • Please mark “Clear cookies each iteration” once you are comfortable with your design

 

View Results Tree

Get

View Results Tree – Request

Image

ViewResultsTree_Request_20171025_1204PM

 

Explanation

Take a good look at the Post data

  1. Post data
    • Do you have that the user field populate
    • What about the hidden fields
      • Are the hidden fields supplied by the system and are they varied as a counter measure against replay

 

View Results Tree – Response data

Image

ViewResultsTree_ResponseData_20171025_1204PM

 

Explanation

Our response data looks perfect.

Post

View Results Tree – Request

Image

ViewResultsTree_Request_20171025_0127PM

 

Explanation
  1. Get data
    • Because our request type is not a Get, but a Post, the Get data is left vacant
  2. Cookies Data
    • We are authenticated and we have our cookies

 

Web Server

Please check IIS Logs and enable Failed Request Tracing

  • HTTP Logs
  • Failed Request Tracing

Failed Request Tracing

IIS Failed Request Tracking module offers superlative debugging tooling.

Failed Request Tracing – 001

Failed Request Tracing – 001 – Image

 

Failed Request Tracing – 001 – Textual

Validation of viewstate MAC failed. 
If this application is hosted by a Web Farm or cluster, ensure that machineKey configuration specifies the same validationKey and validation algorithm. 
AutoGenerate cannot be used in a cluster.
See http://go.microsoft.com/fwlink/?LinkID=314055 for more information.

 

 

Failed Request Tracing – 002

Failed Request Tracing – 002 – Image

 

Failed Request Tracing – 002 – Textual

The state information is invalid for this page and might be corrupted.

 

Dedication

Dedicated to Michael Stover.

Main » jmeter-user » 2003-07 » RE: using the regular expression extractor to obtain a form value
Link

MichaelStoverWorkflow_20171026_0545PM


					

IIS Logs / Log Parser Studio – Aggregated Hits per Server

Background

Our monitoring team has developed and rolled out scripts for monitoring our web farm.

And, we are getting alerts through email.

Quite a lot of emails are coming across and wanted to see if they are coming from same host or a combination of hosts.

 

Emails

Looked at the emails and they happen to be coming from same host.

And, so will have to engage our Network team and see how the Load Balancer is configured.

Is there a prospect that more traffic is being directed at the failing node?

Network Load Balancer

As we prepared to go to the Network Load Balancer team took the opportunity to take gather and query the IIS Logs, as well.

 

TroubleShooting

Log Parser Studio

Query


SELECT 
            To_String(date, 'yyyy-MM-dd') as dated

          , sc-status as status

          , sum (
                    case s-ip
                        when '10.0.4.25' then 1
                        else 0
                   end
               ) as S1

          , sum (
                    case s-ip
                        when '10.0.4.26' then 1
                        else 0
                   end
               ) as S2


          , sum (
                    case s-ip
                        when '10.0.4.27' then 1
                        else 0
                   end
               ) as S3

         , sum (
                    case s-ip
                        when '10.0.4.28' then 1
                        else 0
                   end
               ) as S4

          , min(TO_TIMESTAMP(date, time)) as tsRecordedMin


          , max(TO_TIMESTAMP(date, time)) as tsRecordedMax


FROM '[LOGFILEPATH]' 


where   (


           (

             TO_TIMESTAMP(date, time) 
                     between timestamp('2017/08/02 10:30:00', 'yyyy/MM/dd hh:mm:ss')  
                          and timestamp('2017/08/02 17:20:00', 'yyyy/MM/dd hh:mm:ss')
           )

       )

/*

	and  c-ip not in ('10.0.4.141')
	
*/

group by
         date
       , sc-status


order by
           dated 
         , status



Output

Time Range – 1 ( August 2nd 10:30 AM – 5:20 PM )

Results

Explanation
  1. It is difficult to make case that traffic is exhaustively being waded into a specific host

Time Range – 2 ( August 8th 5:13 PM – 8:40 PM )

Results

Explanation
  1. In our second time slot, 4700 records bearing HTTP 200 is right around average

Summary

At this time it is likely that the sufferance we are seeing with this specific host is not due to outside pressure, but internal to the host itself.

 

SSRS – Setting up Smart Host

 

Background

A year or so ago we setup subscriptions to a couple of Reports that we are providing through SQL Server Reporting Services.

The subscriptions go out through daily email.

Ever so often things just break.

I was recently informed that Emails have not been going for over a week now.

Last time I blamed it on other processes that are using that same host.

Hoping today I can do same and go on about my business.

But, no such luck.

 

Environment

Here is our topology

  1. Reporting Services
    • Reporting Services is running on a local server in our intranet.
  2. Database Server
    • Database Server is running in our Colocation’s Data Center
  3. Email Server
    • The email server is Microsoft’s Office365.com

 

Troubleshooting

Thinking out loud

As always don’t have a clue what changed.

Could it be…

  1. Tightened Security
    • Can emails only go out from certain hosts
    • Do I need an actual username and password combination
    • Firewall
      • Local
        • Is it Windows Firewall
      • Corporate
        • Is it a Corporate Firewall
    • Is it Antivirus Configuration

 

Remediation

Proposal

Not sure what is getting in the way of SSRS getting the emails out.

But, a likely workaround is use a local functional SMTP server as a bridge.

 

Local SMTP Server

Installation

Launch “Server Manager” and we will choose to add “SMTP Server Tools” as a Feature.

 

Step

  1. Tab – Features
    • If “SMTP Server” feature is not checked, please place a check mark next to it
    • Dependencies
      • The “Add role services and features required for SMTP Server” window appear
        • The features listed are “Web Server (IIS)” and “Remote Server Administrative Tools”
  2. Tab – Web Server ( IIS )
    • Shows Web Server literature
  3. Tab – Confirmation
    • Confirmation that IIS and Remote Server Administrative Tools will be augmented
  4. Tab – Progress
    • As installation is proceeding each step is chronicled
  5. Tab – Results
    • The status of each component installed is noted

Images

Add Features Wizard – Select Features
Initial Screen

Before Adding “SMTP Server….

SelectFeatures_SMTP_201708087_0420PM

 

Post Checking “SMTP Server”

Adding “SMTP Server “….

SelectFeatures_SMTP_201708087_0421PM

Add Features Wizard – Add role services and features required for SMTP Server?

Dependencies are listed.

And, they include Web Server ( IIS ) and Remote Server Administrator Tools.

SelectFeatures_AddFeaturesWizard_201708087_0420PM

 

Web Server ( IIS)

Components :-

  1. Internet Information Services ( IIS ) 7.0
    • ASP.Net
    • Windows Communication Foundation

 SelectFeatures_SMTP_WebServer_IIS_201708087_0421PM

 

Confirm Installation Selections

Confirm Installation.

In our case:

  1. Web Server ( IIS )
    • Health and Diagnostics
      • ODBC Logging
    • Remote Server Administrator Tools
      • SMTP Server Tools

 

SelectFeatures_SMTP_WebServer_IIS_RoleServices_Confirmation_201708087_0423PM

Installation Progress

Installation is progress…

 

SelectFeatures_SMTP_InstallationProgress_201708087_0424PM

 

 

Installation Results

Installation Succeeded.

SelectFeatures_SMTP_WebServer_IIS_RoleServices_InstallationResults_201708087_0435PM

 

 

Configuration

Customization

  1. Tab – General
    • Enable Logging
      • It is most useful to turn on logging during initial setup and follow-up troubleshooting sessions
  2. Tab – Access
    • Group – Connection
      • Select which computers may access this session
        • All, except the list below
    • Group – Relay Restrictions
      • Only the list below
        • Self ( for now )
          • 127.0.0.1
  3. Tab – Messages
    • Send copy of non-delivery report to
      • Mail Administrator
        • Hopefully a monitored distribution list
    • Bad mail directory
      • Default
        • C:\Bad Mail
      • Non-system drive folder
        • Hopefully, you take the opportunity to change the folder to a non-system drive
  4. Tab – Delivery
    • Group box – Outbound Security
      • Authentication Choices
        • Anonymous
        • Basic Authentication
        • Windows Integration
      • In our case :-
        • Anonymous ( NO )
          • Are trying to get away from Anonymous as our hosting platform, Microsoft Office, requires user authentication
        • Integrated Windows Authentication ( NO )
          • We do not have cross-domain relationship between us and Microsoft’s Hosted Outlook
        • Basic Authentication ( YES )
      • TLS
        • We enabled TLS
    • Group box – Outbound Connections
      • TCP Port
        • 587
          • This is the default mail submission port. When a mail client or server is submitting an email to be routed by a proper mail server, it should always use this port.
            Unless you’re explicitly blocked by your upstream network or hosting provider.
            This port, coupled with TLS encryption, will ensure that email is submitted securely and following the guidelines set out by the IETF”

            John Carl Villanueva ( Link )
    • Group box – Advanced Delivery
      • Fully Qualified Domain Name
        • Especially for domains that have SPFs set up
      • Smart Host
        • smtp.office365.com
      • Attempt direct delivery before sending to smart host
        • Unchecked
      • Perform reverse DNS lookup on incoming messages
        • Unchecked
  5. LDAP Routing
    • Not going to need to use LDAP Routing for user authentication
  6. Grant Operator permissions to these Windows User Accounts

 

Screenshot

SMTP Virtual Server – Properties – General

General_20170808_0840AM

 

 

SMTP Virtual Server – Properties – Access

Access_20170807_0752PM

 

SMTP Virtual Server – Properties – Access – Connection
Initial

Access_ConnectionControl_20170807_0753PM

 

SMTP Virtual Server – Properties – Access – Relay Restrictions
Initial

Access_RelayRestrictions_20170807_0754PM

 

Add Computer

List

  1. Single Computer
    • IP address:- 127.0.0.1

 

Access_RelayRestrictions_AddComputer_20170807_0755PM

Completed

Access_RelayRestrictions_Computer_20170809_0114PM

 

SMTP Virtual Server – Properties – Messages
Initial

Messages_20170807_0756PM

 

Completed

Messages_20170807_0757PM (BrushedUp)

 

 

SMTP Virtual Server – Properties – Delivery
Initial

Delivery_20170807_0757PM

 

SMTP Virtual Server – Properties – Delivery – Outbound Security
Initial

Delivery_OutboundConnections_20170807_0444PM

 

 

Complete

Delivery_OutboundSecurity_20170807_0758PM (BrushedUp)

 

SMTP Virtual Server – Properties – Delivery – Outbound Connections
Initial

Delivery_OutboundConnections_20170807_0445PM

 

Completed

Delivery_OutboundConnections_20170807_0758PM

 

 

SMTP Virtual Server – Properties – Delivery – Advanced Delivery
Initial
Completed

AdvancedDelivery_20170807_0759PM [BrushedUp]

 

Conclusion

We have an SMTP Server setup.

We will come back and unit test it out and once verified, we will point Sql Server Reporting Services ( SSRS ) to route emails through it.

 

References

  1. jscape
    • John Carl Villanueva
      • Still Confused With SMTP Ports? Read This
        Link

SQL Server – Reporting Services – Connecting Locally – Day 1

Background

Hardening security via applying SSL Certs on a couple of Reporting Services Hosts and wanting to test them on same host, but “No Go“.

 

TroubleShooting

Windows Event Viewer

Checked Windows Event Viewer

Security

Security – Headers

Image

Tabulate
  1. Event ID = 4625
    • Keywords :- Audit Failure
    • Source :- Microsoft Windows Security auditing
    • Event ID :- 4625
    • Task Category :- Logon

Security – Details

Image

 

Tabulate
  1. Event ID = 4625
    • Security ID :- NULL SID
    • Logon Type :- 3
      • Logon Type 3 is Network
    • Status :- 0xC000006D
    • Event ID :- 4625
    • Task Category :- Logon

Summary

Basically, we were prompted thrice to enter our username and password. But, unable to connect.

 

 

Internet Explorer

Checked to make that we are still unable to connect when we run in Administrator Mode.

Task Manager

To verify that IE is running in Administrator mode launched Task Manager and included the “Elevated” attribute.

Select Columns

Results

Image

Explanation

For each IE Session, we are seeing two processes.
Why two processes each time we start a new IE Session?

 

Remediation

Registry

Outline

There are a couple of options and those are:

  1. BackConnectionHostNames
  2. DisableLoopbackCheck

 

BackConnectionHostNames

Worknotes

Launch regedit and access the registry key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0”.

Search for BackConnectionHostNames

Add all FQDN that the server’s resource will be self referred to.
Each entry should be entered in its own line.

  1. Type :- REG_MULTI_SZ
  2. Data :- ????

Images

Adding Entry

Entry Added

 

DisableLoopbackCheck

Worknotes

Launch regedit and access the registry key “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa“.

Search for DisableLookback.

Make sure it exists as:

  1. Type :- REG_DWORD
  2. Data :- 1

Image

Script

Script – BackConnectionHostNames


@echo off
@echo on

set "_registryBranch=HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0"
set "_registryItem=BackConnectionHostNames"
set "_registryDataType=REG_MULTI_SZ"

rem ****************************************************************************************
rem please change to match your domain name
rem ****************************************************************************************
set "_domainName=labdomain.org"

set "_registryValue=%COMPUTERNAME%.%_domainName%"

echo "Value - Current"
reg query %_registryBranch% /v %_registryItem%

reg add %_registryBranch% /v %_registryItem% /t %_registryDataType% /d %_registryValue% /f

echo "Value - New"
reg query %_registryBranch% /v %_registryItem%

Script – DisableLoopbackCheck


@echo off
rem set "_registryBranch=HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa"
set "_registryBranch=HKLM\SYSTEM\CurrentControlSet\Control\Lsa"
set "_registryItem=DisableLoopbackCheck"
set "_registryDataType=REG_DWORD"
set "_registryValue=1"

echo "Value - Current"
reg query %_registryBranch% /v %_registryItem%

reg add %_registryBranch% /v %_registryItem% /t %_registryDataType% /d %_registryValue% /f

echo "Value - New"
reg query %_registryBranch% /v %_registryItem%

Summary

This problem is nothing.  It has been in the OS since Windows 2003.

And, so I suppose it is not really a problem, I just wished it was surfaced differently; than having to type my password thrice and still can’t get in.

 

References

  1. Microsoft
    • Microsoft Support
      • You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or a later version
        Link
  2. Nik Patel
    • Disable the Loopback Check for Specific Host Names on all SharePoint Web and Application Servers
      Link
  3. Michael Hanes
    • Use BackConnectionHostNames instead of DisableLoopbackCheck in production
      Link
  4. Harber.net
    • DisableLoopbackCheck & SharePoint: What every admin and developer should know
      Link

Internet Information Server (IIS) – Application Pool – Tracking – Day 2

Background

This is the second post on our series on tracking the status of IIS’s Application Pool.

Lineage

Here is our initial post:

  1. Internet Information Server (IIS) – Application Pool – Tracking
    Link

 

TroubleShooting

Event Viewer

Log Parser Studio

Queries

Query – Get All WAS Entries
Query
SELECT TOP 1000

         TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') as TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Message
  
from  '[LOGFILEPATH]'

where ( SourceName = 'WAS' ) 
 
ORDER BY
           ComputerName
         , TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') DESC

Output

 

Query – Get WAS Entries – Application Pool Disabled
Query


SELECT TOP 1000

         TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') as TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Message
  
from  '[LOGFILEPATH]'

where ( SourceName = 'WAS' ) 
 
and ( Message like '%disable%' ) 

ORDER BY
           ComputerName
         , TO_STRING(TimeGenerated, 'yyyy-MM-dd HH:mm:ss') DESC
Output

 

Summary

There are a few entries bearing the Source WAS in Windows System Event Viewer.
Inclusive are :

  1. A process serving application pool ‘DefaultAppPool’ failed to respond to a ping. The process id was ‘6208’.
  2. A process serving application pool ‘DefaultAppPool’ suffered a fatal communication error with the Windows Process Activation Service. The process id was ‘13844’. The data field contains the error number.
  3. A worker process with process id of ‘21412’ serving application pool ‘DefaultAppPool’ has requested a recycle because the worker process reached its allowed processing time limit.
  4. Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.

 

The ones most pernicious is “Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool. “

Internet Information (IIS) / Log Parser – Queries – String Pattern Matching

Background

Looking for File I/O Exceptions in the Event Viewer.

 

Query

Sample

Sample 001

Code


SELECT TOP 100 
 
         TimeGenerated
       , ComputerName
       , EventCategoryName
       , EventTypeName
       , EventID
       , SourceName
       , Message as Mesg
       , Strings as Strings
       , EXTRACT_TOKEN(Strings,1,'|') AS AppName
       , EXTRACT_TOKEN(Strings,2,'|') AS AppVersion
       , EXTRACT_TOKEN(Strings,3,'|') AS S3
       , EXTRACT_TOKEN(Strings,4,'|') AS Module
       , INDEX_OF(Message, 'System.IO.IOException') as indexOf
       , case INDEX_OF(Message, 'System.IO.IOException') 
            when 0 then 'N'
            when NULL then 'N'
            else 'Y'
         end as IOE
       , CASE strcnt(Message, 'System.IO.IOException')
             when 0 then 'No'
             else 'Yes'   
         end as IOException
 
from  '[LOGFILEPATH]'
 
WHERE ( EventType = 1 OR EventType = 2 )

and    INDEX_OF(Message, 'System.IO.IOException') > 0

 
ORDER BY
         TimeGenerated DESC


Output

 

Explanation

  1. INDEX_OF
    • We use INDEX_OF to find the position of the sought string in the Message column
      • When the column contains System.IO.IOException the query returns the starting position of the found pattern
      • When not found, null is returned
  2. STRCNT
    • We invoke STRCNT to count number of matches
      • When String not found, 0 return
      • When matched, number of matches

 

References

  1. StackOverflow
    • Log Parser Case Statement
      Link