Linux – Listing Files Based on Date

 

Background

I have a need to list files on a Linux Host based sorted by date.

 

ls Command

Earliest First

Command

[ code language=’bash’]

ls -ltr -G -g | grep -v ‘^total’ | head -5

[/code]

 

Output

 

Latest First

Command

[ code language=’bash’]

ls -lt -G -g | grep -v ‘^total’ | head -5

[/code]

Output

 

Find Command In a ditch

Unfortunately, when there are numerous files in the target folder the ls command and other file utilities can get in a ditch.

Argument List Too Long

Command


ls *.LOG

Output

Image

Text

bash: /bin/ls: Argument list too long

 

Find Command

Earliest First

Command

[ code language=’bash’]

find . -name ‘*.LOG’ | sort | head -n 5

[/code]

Output

Command

[ code language=’bash’]

find . -name ‘*.LOG’ | sort | tail -n 5

[/code]

Output

 

Script

Let us get a bit more complicated.

And, we will do do by picking up individual files, process them, and exit after N Number of files have been processed.


# get current folder
echo "Current working folder is $PWD"


#echo list all files in current log
#find . -name "*.LOG" | xargs -i echo " {} "


declare -i iFileID
declare -i iFileIDMax
declare FORMAT_FILE_PROCESSING

iFileID=1
iFileIDMax=1000
iFileIDMax=50
FORMAT_FILE_PROCESSING="%d  file %s \n"

#echo $FORMAT_FILE_PROCESSING

#list earliest top N LOG files
find . -name "*.LOG" 2> /dev/null  | sort | head -n $iFileIDMax  | while  IFS="" read name;
do



     printf "$FORMAT_FILE_PROCESSING" $iFileID  $name

     iFileID=$((iFileID+1));

     if [ $iFileID  -gt  $iFileIDMax ]
      then


        break;

     fi


done



Summary

When needing to list files by date, one can use the ls command,

In cases where there are numerous files, the type of filtering one can do is hampered by the fact that the ls command tries to accept all the files as a batch.

A good workaround is the find command.

 

Reference

  1. ls command
    • TheGeekStuff.com
      • Linux ls command examples
        Linux
    •  linuxcommand.org
  2. Find Command
    • Unix.com
      • While loop, input from find command
        Link
    • StackOverflow.com
      • Display modified datetime with Find Command
        Link
    • unix.stackexchange.com
      • Only find first few matched files using find
        Link
  3. Loop Control
    • The Linux Documentation Project ( tldp.org )
  4. If Command
    • TheGeekStuff.com
      • Bash If Statement Examples ( If then fi, If then else fi, If elif else fi, Nested if )
        Link
  5. Math
    • K-State Polythecnic
      • Math in Shell Script
        Link
  6.  printf
    • Linuxconfig.org
      • Bash printf basic commands
        Link
    • Bash Hackers wiki
      • The printf command
        Link
    • Unix.StackExchange.org
      • Printf formatting with variable format – what does this var reference?
        Link

 

Change Prompt in Bash

Background

For us that like to journal and keep notes of every command we issue and capture screens, having long prompts get in the way a bit.

Especially, when you get an incident and pick up that phone and  it says one of the DB/2 hosts is running out of Log space again.

And, the best you can do is say that if it is only at 90% it can likely suffer till the morning.

 

Current Prompt

To get the current prompt issue “echo $PS1

Image

Text

\u@\h:\w>

Revised Prompt

Let us rid ourselves of all the clutter by dropping all three; username, machine name and the current working directory are all out of here.

SESSION


export PS1='>'

 

Permanently

To make the change permanently, please edit ~/profile in your editor of choice and add same line.


					

SQL Server – SSMS – Options – “Override connection string time-out value for table designer updates”

 

Background

This is the third in a series of posts on timeouts

 

SQL Server Management Studio ( SSMS)

Designers

For those that use the Designer built into SSMS to make structural table changes, the Designer Option panel is a good place to go and make sure that the default settings are what you want them to be.

Screen Shot

Here is what that screen looks:

Initial

Post Changes

Image

Explanation
  1. Here we changed the “Transaction Time-out after:” from the default of 30 seconds to 90 seconds

 

Misnomer

I think outside of the Database World and specifically SQL Server, it is OK to use the phrase & words, but I think for a tool that targets SQL Server and only that table, a bit more care should be taken.

And, you likely ask why.

Grouping :- Override connection string time-out value for table designer updates
Phrase :- Transaction time-out after

Here are the difficulties that I have with the Group Label “Override connection string time-out value

  1. We all know that the Connection String does not expose command execution nor lock timeout
    • Command Execution can be set in the Connection or Command Object, but not in the actual connection string
    • Because of this understanding, coders have to look into the code and set the timeouts
    • They are not able to do so declaratively in the Web.config and thus not suffer the pain of recompile
  2. There is more than enough perception that Connection Open timeouts and Command Execution timeouts, are one the same

 

Microsoft – Connect

  1. Item #  :- 3130697
    • Link :- Link
      Opened By: Daniel Adeniji
      Date Opened:- 2017-March-25
      Type :- Suggestion
      Status :- Active

SQL Server – “Lock Timeouts” – Trigger Timeouts – SSMS – Table Designer

Background

In our initial post we spoke about experiencing an Outage.

Our Clients in the Development side of the house explained they have being logging “Lock Timeouts“.

We wanted to see what tools we can use to detect similar occurrences on the DB.  And, we cited SQL Server Profiler as one of those tools.

In this post, we will attempt to simulate time outs due to expired Lock requests and see how well SQL Server Profiler performs.

 

Lab

Outline

  1. Start SQL Server Profiler and set to capture event
  2. Create a bare minimum table
  3. Add data
    • Start Transaction
    • Add a couple of records
    • Temporarily leave Transaction Uncommitted
  4. In another session, return to Table designer & initiate changes
    • Add one or two new columns, attempting to make design changes

Steps

SQL Server Profiler

Here are the events that we said we will capture:

Image

Image – Events Selection

Image – Edit Filter

 

Tabulation

Tabulation – Events Selection
Event Category Event
 Locks  Lock:Cancel
 Stored Procedures  RPC:Completed
 TSQL  SQL:Batch Completed

 

 

Tabulation – Filter
Column Value
 Duration  30000

 

 

Create a Bare Minimum Table

Image

Explanation

In the screen above, we have created a table and added a couple of columns ( personID and personName).

 

Attempt to add data

Script


set nocount on
go
set XACT_ABORT on
go

use [DBLab]
go

declare @transactionComplete bit

set @transactionComplete = 0

begin tran

	insert into [dbo].[person]
	default values;


 

Table Designer

Table Designer – Adding new column – dateAdded

Image

 

Table Designer – Execution Timeout Expired

Image

Textual

'person' table
- Unable to modify table.
Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

 

SQL Server Profiler

Image

Lock:Cancel

Tabulated
Event Category Event Columns Values
 Lock  
 Lock:Cancel
 TextData
 Application  Microsoft SQL Server Management Studio
 NTUserName  dadeniji
 Duration  29999
 StartTime  2017-03-25 16:23:18.190
 EndTime  2017-03-25 16:23:18.190
 Mode  2-SCH-M
 ObjectID 110623437
 ObjectID2  110623437
 OwnerID  1-TRANSACTION
 DatabaseID  7
 DatabaseName  DBLAB
 Hostname  DADENIJI
 IntegerData2  0-LOCK

 

 

Explanation
  • You want to pay attention to ObjectID and Database ID
    • Those two will lead you to Database, Schema, and Object name
  • Mode
    • Will lead you to desired Lock
      • In our case, 2-SCH-M or Schema Modification Lock
  • And, of course Application, Host, and Username, and TextData
    • Determine causation and forensics

 

SQL: Batch Completed

Tabulated
Event Category Event Columns Values
 T-SQL  
SQL: Batch Completed
 TextData  ALTER TABLE dbo.person ADD dateAdded null
 Application  Microsoft SQL Server Management Studio
 NTUserName  dadeniji
 Duration  29999
 StartTime  2017-03-25 16:22:48.190
 EndTime  2017-03-25 16:23:18.190

 

 

Explanation
  • You want to pay attention to ObjectID and Database ID
    • Those two will lead you to Database, Schema, and Object name
  • Mode
    • Will lead you to desired Lock
      • In our case, 2-SCH-M or Schema Modification Lock
  • And, of course Application, Host, and Username, and TextData
    • Determine causation and forensics

 

 

Dedicated

Dedicated to MSFT’s SQL Server Team.

 

 

SQL Server – “Lock Timeouts” – Identifying through SQL Server Profiler

Preface

We experienced an outage this last Monday.

One of the errors logged in the WebSphere log file is the one pasted below:

Lock request time out period exceeded.

TroubleShooting

SQL Server Profiler

One of the most formidable tool for SQL Server TroubleShooting is SQL Server Profiler.

It allows the DBA to monitor several activities.

 

SQL Server Profiler – Take Care

Because it is an expensive to run it, I will suggest the following:

  1. Carefully choose which events you will like tracked
  2. Convert from Client Tracking to Server Tracing

 

SQL Server Profiler – Events

Here are the events to track on…

Tabulated

Events File Version
 Locks  
 Lock: Cancel
 Lock: Timeout
 Lock:Timeout ( timeout > 0)
 Stored Procedures
 RPC:Completed
 TSQL  
 SQL: BatchCompleted

 

Image

SQL Server Profiler – Filter

To augment the events we need to place filters and thus ensure we are not overloaded.

Tabulated

Events File Version Specific
 Duration  
 Greater or equal to
 30000  ( Please note value is in milliseconds, and so we have it set for 30 seconds )
 Exclude rows that do not contain values

 

 

Image

 

 

Dedicated

Duckworth & Borris Callens

How to find timed out statements in SQL 2005 profiler
Link

 

References

  1. How to find timed out statements in SQL 2005 profiler
    Link
  2. Transcender
    • Application Deadlock
      You are the database administrator for a banking company. You manage all the SQL Server 2008 R2 databases of the company. The company stores customer-related data in the database named Cust01. This database is accessed by most users in the company for different purposes. The users daily perform insert and updates to the database through a .NET application.
      Eric, a user in the database, complains that his transaction has frozen and that he is not able to perform any operation in the database. You find out that the problem is due to a deadlock. You want to find out the user who is the other participant in the deadlock.
      Link

 

Summary

In follow-up posts, we will take things to the Lab and do things that will trigger timeouts.

SQL Server 2005 – Express Edition – Manual Updates

Background

As promised a couple of posts back, over this last weekend we successfully patched an old SQL Server Express v2005 RTM Edition  to v2005 SP4.

In our initial post, we spoke of the difficulty we experienced trying to patch v2005 using Windows Update.

In this post we skipped having Automatic Updates apply the patch and downloaded the patch and manually applied it.

We stumbled badly but thankfully discovered workarounds through Google.

 

TroubleShooting

Event Viewer

Error 1260 – Windows cannot open this program because it has been prevented by a software restriction policy

Error Image

EventID_11260_MicrosoftSQLServerExpressEdition_Error_1260_0237PM (Brushed Up)

 

Textual

  1. Source :- MsiInstaller
  2. Type :- Error
  3. Event ID :- 11260
  4. Product :- Microsoft SQL Server Express Edition – Error 1260.  Windows cannot open this program because it has been prevented by a software restriction policy.

 

Remediation

Outline
  1. Get Application’s GUID
  2. Add it to the exception list
ScreenShot

ScreenShot – Before Adding Application GUID

WLRegistry_20170319_0243PM

ScreenShot – Before Adding Application GUID { 2AFFDD7* }

WLRegistry_20170319_0244PM

SQL Server – Error Log

Error: 17207, Severity: 16, State: 1 – FCB::RemoveAlternateStreams: Operating system error 6(The handle is invalid.) occurred while creating or opening file

Error Text

2017-03-19 14:02:52.13 spid5s      Starting up database ‘master’.
2017-03-19 14:02:52.18 spid5s      Error: 17207, Severity: 16, State: 1.
2017-03-19 14:02:52.18 spid5s      FCB::RemoveAlternateStreams: Operating system error 6(The handle is invalid.) occurred while creating or opening file ‘E:\Program Files\Microsoft SQL Server\v2005\SQLExpress\MSSQL.1\MSSQL\DATA\master.mdf’. Diagnose and correct the operating system error, and retry the operation.
2017-03-19 14:02:52.59 spid5s      Recovery is writing a checkpoint in database ‘master’ (1). This is an informational message only. No user action is required.

2017-03-19 14:02:52.82 spid5s      Starting up database ‘mssqlsystemresource’.
2017-03-19 14:02:53.13 spid7s      Starting up database ‘model’.

2017-03-19 14:02:53.18 spid7s      Error: 17207, Severity: 16, State: 1.
2017-03-19 14:02:53.18 spid7s      FCB::RemoveAlternateStreams: Operating system error 6(The handle is invalid.) occurred while creating or opening file ‘E:\Program Files\Microsoft SQL Server\v2005\SQLExpress\MSSQL.1\MSSQL\DATA\model.mdf’. Diagnose and correct the operating system error, and retry the operation.

2017-03-19 14:02:54.16 spid7s      Clearing tempdb database.
2017-03-19 14:02:54.99 spid7s      Error: 17207, Severity: 16, State: 1.
2017-03-19 14:02:54.99 spid7s      FCB::RemoveAlternateStreams: Operating system error 6(The handle is invalid.) occurred while creating or opening file ‘E:\Program Files\Microsoft SQL Server\v2005\SQLExpress\MSSQL.1\MSSQL\DATA\tempdb.mdf’. Diagnose and correct the operating system error, and retry the operation.
2017-03-19 14:02:55.97 spid7s      Starting up database ‘tempdb’.
2017-03-19 14:02:55.99 spid7s      Error: 17207, Severity: 16, State: 1.
2017-03-19 14:02:55.99 spid7s      FCB::RemoveAlternateStreams: Operating system error 6(The handle is invalid.) occurred while creating or opening file ‘E:\Program Files\Microsoft SQL Server\v2005\SQLExpress\MSSQL.1\MSSQL\DATA\tempdb.mdf’. Diagnose and correct the operating system error, and retry the operation.
2017-03-19 14:02:56.18 spid5s      Recovery is complete. This is an informational message only. No user action is required.

Explanation

It is likely another application is interfering with the database’s engine access to the database files.

It could be Anti-Virus, OS File System drivers and filters or diagnostic tools.

In our case it could have been the fact that we were running SysInternal’s process monitor and chosen to focus in on the folder where how SQL Server data files are kept.

 

SQL Server Install – Summary.txt

Exit Code Returned: 1260

Error Text

**********************************************************************************
Product Installation Status
Product                   : Express Database Services (V2005SQLEXPRESS)
Product Version (Previous): 1399
Product Version (Final)   :
Status                    : Failure
Log File                  : C:\Program Files\Microsoft SQL Server\90\Setup Bootstrap\LOG\Hotfix\SQL9Express_Hotfix_KB2463332_SQLEXPR.EXE
SQL Express Features      : SQL_Data_Files,SQL_Engine,SQL_SharedTools
Error Number              : 1260
Error Description         : Unable to install Windows Installer MSI file
———————————————————————————-

**********************************************************************************
Summary
One or more products failed to install, see above for details
Exit Code Returned: 1260

 

Remediation

Remediation – Change SQL Server Service Account to Domain Account

Review Account SQL Server is running under.  If it is running under “NT Authority\Network Services”, then change it to use to use a “Domain Account”.

Steps Outline

  1. Launch SQL Server Configuration Manager
  2. On the left panel, access the SQL Server Services Node
  3. On the right panel, review the list of services
  4. In our case, our targeted instance is v2005 SQLExpress
    • We reviewed all of the corresponding services for that instance
    • Currently, they are set to NT AUTHORITY\NETWORK SERVICES
    • We changed them to run under a least privileged domain account

ScreenShot

Sql Server Configuration Manager – List of Services

SQLServerConfigurationManager-NetworkService-20170319-0119PM (Brushed Up)

 

Sql Server Configuration Manager – Changing Service Account from Network Service to ….

changeServiceAccount_20170319_0110PM (Brushed Up)

 

Sql Server Configuration Manager – Changed Service Account from Network Service to Domain Account

SQLServerConfigurationManager-ChangedToDomainAccount-Services-20170319-0113PM (Brushed Up)

 

 

Summary

As always took the long way home.

Here are all the things we did:

  1. Whitelisted SQL Server v2005 SP4
    • Captured App GUID through Event Viewer or summary.txt ( SQL Server Log File )
  2. Experienced “Error: 17207, Severity: 16, State: 1 – FCB::RemoveAlternateStreams: Operating system error 6(The handle is invalid.) occurred while creating or opening file”
    • Stopped SysInternal’s Process Monitor
  3. Ran into Error Code 1260
    • Changed Service Account from “NT Authority\Network Services” to Domain Account
    • Ensured that the Service Account has full permissions on all folders that SQL Server has data and log files