Linux – printf – Proper Usage

Background

I ran into problems with a little bash code that I was writing.

I was trying to use the printf code format output.

 

Code

Here is the original code.

Original Code


iNumberofProducts=10
strProductName="Apple"
dCostPerProduct=1.25

dProductSubTotal=`echo "$iNumberofProducts*$dCostPerProduct"|bc`

FORMAT_PRODUCT_DETAIL="%d %s at  %.2f each for a sub-total of %.2f \n"

printf $FORMAT_PRODUCT_DETAIL  $iNumberofProducts $strProductName $dCostPerProduct $dProductSubTotal

 

Output

Textual


printf.sh: line 11: printf: %s: invalid number
printf.sh: line 11: printf: at: invalid number
printf.sh: line 11: printf: %.2f: invalid number
printf.sh: line 11: printf: each: invalid number
printf.sh: line 11: printf: for: invalid number
printf.sh: line 11: printf: a: invalid number
printf.sh: line 11: printf: sub-total: invalid number
printf.sh: line 11: printf: of: invalid number
printf.sh: line 11: printf: %.2f: invalid number
printf.sh: line 11: printf: \n: invalid number
printf.sh: line 11: printf: Apple: invalid number
printf.sh: line 11: printf: 1.25: invalid number
printf.sh: line 11: printf: 12.50: invalid number


 

Revised Code

Code


iNumberofProducts=10
strProductName="Apple"
dCostPerProduct=1.25

dProductSubTotal=`echo "$iNumberofProducts*$dCostPerProduct"|bc`

FORMAT_PRODUCT_DETAIL="%d %s at %.2f each for a sub-total of %.2f \n"

printf "$FORMAT_PRODUCT_DETAIL" $iNumberofProducts $strProductName $dCostPerProduct $dProductSubTotal


Code

The solution is pretty simple and it is to quote the format string.

Simply replace printf $FORMAT_PRODUCT_DETAIL with printf $FORMAT_PRODUCT_DETAIL .

 

 

SQL Server – Identify Lock TimeOuts via Extended Events ( from converted Trace File )

 

This is the 3rd in a series of posts on Identifying Lock Timeouts.

Here are earlier posts:

  1. SQL Server – “Lock Timeouts” – Identifying through SQL Server Profiler
    Link
  2. SQL Server – “Lock Timeouts” – Trigger Timeouts – SSMS – Table Designer
    Link

Extended Events

Extended Events introduced in SQL 2008 is lighter to run on a system compared to client side SQL Server Profiler or Server Side Trace.

Design Tooling

Unfortunately, when first introduced with SQL 2008 tooling for creating events was not included.

SQL 2008-R2 came with tooling for designing new Extended Events ( EE ) and modifying existing ones.

But, unfortunately, in my humble estimation, the tooling did not sufficiently expand on what is available via SQL Server Profiler.

In  a later post, we will touch on some items that can be bettered in the GUI for creating and modifying Extended Events.

 

 

Convert SQL Server Profiler to Extended Event

Jonathan Kehayias

SQLSkills has a very good Stored Procedure ( SP ) that generates a script for extended event from a running trace.

It is available here.

As always I am late as this tool was released back in 2012.

 

Create SP

Once downloaded, please create the SP, sp_SQLskills_ConvertTraceToExtendedEvents.

 

Use SP

To use the SP, please do the following:

  1. Start SQL Server Profiler Trace
  2. Get the Trace Number by running “select * from sys.traces
  3. Generate Extended Event Script – Pass along Trace ID
  4. Capture & Appy Generated Script
  5. Review and amend generated Extended Event
  6. Start Extended Event
  7. View Life Data

 

Get Trace Number
Code

select *

from sys.traces tblST

-- Not Default Trace
where tblST.[is_default] = 0

-- Trace is running
and tblST.[status] = 1

order by
tblST.start_time desc

 

Output

 

Generate Extended Event Script
Code
use master
go

declare	@TraceID INT
declare	@SessionName NVARCHAR(128)
declare	@PrintOutput BIT
declare	@Execute BIT

set		@TraceID = 4
set		@SessionName = 'IdentifyLockTimeOut'
set 	@PrintOutput = 1
set	    @Execute = 0


exec sp_SQLskills_ConvertTraceToExtendedEvents
		  @TraceID = @TraceID
		, @SessionName = @SessionName
		, @PrintOutput = @PrintOutput
		, @Execute = @Execute

 

Output

IF EXISTS (SELECT 1 FROM sys.server_event_sessions WHERE name = 'IdentifyLockTimeOut')
	DROP EVENT SESSION [IdentifyLockTimeOut] ON SERVER;
GO
CREATE EVENT SESSION [IdentifyLockTimeOut]
ON SERVER
ADD EVENT sqlserver.lock_cancel(
	ACTION 
	(
			  sqlserver.client_app_name	-- ApplicationName from SQLTrace
			, sqlserver.client_hostname	-- HostName from SQLTrace
			, sqlserver.client_pid	-- ClientProcessID from SQLTrace
			, package0.event_sequence	-- EventSequence from SQLTrace
			, sqlserver.is_system	-- IsSystem from SQLTrace
			, sqlserver.nt_username	-- NTDomainName from SQLTrace
			, sqlserver.request_id	-- RequestID from SQLTrace
			, sqlserver.server_instance_name	-- ServerName from SQLTrace
			, sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.server_principal_sid	-- LoginSid from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
			, sqlserver.session_resource_group_id	-- GroupID from SQLTrace
			, sqlserver.session_server_principal_name	-- SessionLoginName from SQLTrace
			, sqlserver.transaction_id	-- TransactionID from SQLTrace
			-- IntegerData2 not implemented in XE for this event
	)
	WHERE 
	(
			duration >= 30000000
	)
),
ADD EVENT sqlserver.rpc_completed(
	ACTION 
	(
			  sqlserver.client_app_name	-- ApplicationName from SQLTrace
			, sqlserver.client_hostname	-- HostName from SQLTrace
			, sqlserver.client_pid	-- ClientProcessID from SQLTrace
			, sqlserver.database_id	-- DatabaseID from SQLTrace
			, sqlserver.database_name	-- DatabaseName from SQLTrace
			, package0.event_sequence	-- EventSequence from SQLTrace
			, sqlserver.is_system	-- IsSystem from SQLTrace
			, sqlserver.request_id	-- RequestID from SQLTrace
			, sqlserver.server_instance_name	-- ServerName from SQLTrace
			, sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
			, sqlserver.session_resource_group_id	-- GroupID from SQLTrace
			, sqlserver.session_server_principal_name	-- SessionLoginName from SQLTrace
			, sqlserver.transaction_id	-- TransactionID from SQLTrace
			-- IntegerData not implemented in XE for this event
			-- BinaryData not implemented in XE for this event
	)
	WHERE 
	(
			duration >= 30000000
	)
),
ADD EVENT sqlserver.sql_batch_completed(
	ACTION 
	(
			  sqlserver.client_app_name	-- ApplicationName from SQLTrace
			, sqlserver.client_hostname	-- HostName from SQLTrace
			, sqlserver.client_pid	-- ClientProcessID from SQLTrace
			, sqlserver.database_id	-- DatabaseID from SQLTrace
			, sqlserver.database_name	-- DatabaseName from SQLTrace
			, package0.event_sequence	-- EventSequence from SQLTrace
			, sqlserver.is_system	-- IsSystem from SQLTrace
			, sqlserver.nt_username	-- NTDomainName from SQLTrace
			, sqlserver.request_id	-- RequestID from SQLTrace
			, sqlserver.server_instance_name	-- ServerName from SQLTrace
			, sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.server_principal_sid	-- LoginSid from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
			, sqlserver.session_resource_group_id	-- GroupID from SQLTrace
			, sqlserver.session_server_principal_name	-- SessionLoginName from SQLTrace
			, sqlserver.transaction_id	-- TransactionID from SQLTrace
	)
	WHERE 
	(
			duration >= 30000000
	)
),
ADD EVENT sqlserver.sql_statement_completed(
	ACTION 
	(
			  sqlserver.client_app_name	-- ApplicationName from SQLTrace
			, sqlserver.client_hostname	-- HostName from SQLTrace
			, sqlserver.client_pid	-- ClientProcessID from SQLTrace
			, sqlserver.database_id	-- DatabaseID from SQLTrace
			, sqlserver.database_name	-- DatabaseName from SQLTrace
			, package0.event_sequence	-- EventSequence from SQLTrace
			, sqlserver.is_system	-- IsSystem from SQLTrace
			, sqlserver.nt_username	-- NTDomainName from SQLTrace
			, sqlserver.request_id	-- RequestID from SQLTrace
			, sqlserver.server_instance_name	-- ServerName from SQLTrace
			, sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.server_principal_sid	-- LoginSid from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
			, sqlserver.session_resource_group_id	-- GroupID from SQLTrace
			, sqlserver.session_server_principal_name	-- SessionLoginName from SQLTrace
			, sqlserver.transaction_id	-- TransactionID from SQLTrace
	)
	WHERE 
	(
			duration >= 30000000
	)
),
ADD EVENT sqlserver.xml_deadlock_report(
	ACTION 
	(
			  sqlserver.server_instance_name	-- ServerName from SQLTrace
			, sqlserver.server_principal_name	-- LoginName from SQLTrace
			, sqlserver.session_id	-- SPID from SQLTrace
	)
)
ADD TARGET package0.ring_buffer


Review and amend generated Extended Event

Once Extended Event is created, view and amend it through GUI

Object Explorer

Here is the Object Explorer view, please select the Session, view and amend the targeted Session

 

Live Data

Once the EE is reviewed, please select the EE, start it and select the “Watch Live Data“.

We created a new transaction, added data to the table, and tried to modify the same table.

A short while later, we were able to trigger lock timeout.

Thankfully, our new EE captured same and here is what is captured

 

Live Data – lock-cancel ( Image )

 

Live Data – lock-cancel ( Tabulated )

 

Field Value
 associated_object_id  110623437
 client_app_name  Microsoft SQL Server Management Studio
 client_hostname  DADENIJI
 databse_id  7
 database_name
 duration  89997000
 owner_type  Transaction
 resource_o  110623437
 server_instance_name  DADENIJI\v2014
 server_principal_name dadeniji

 

 

 

Live Data – sql_batch_completed ( Image )

 

Live Data – sql_batch_completed ( Tabulated )

 

Field Value
batch_text  ALTER TABLE dbo.person ADD   gender char(1) NULL
 client_app_name  Microsoft SQL Server Management Studio
 client_hostname  DADENIJI
 databse_id  7
 database_name  DBLab
 duration  89993130
 result  Abort
 row_count  0
 server_instance_name  OBJECT
 server_principal_name dadeniji
 logical_reads  0
 physical_reads  0
 writes  0

 

 

 

Analysis

Processing Time Out

Previous to triggering the error, we increased our timeout by accessing Options\  Designers \ Table and Database Designers.

 

Captured Data

The two events we are tracking offers different dataset.

The “Lock Cancel” captures the following:

  1. The Database ID & Object ID of the Targeted Object
  2. Mode :- SCH_M – Schema Modification
  3. Forensics such as Application Name, Host Name, Database ID

On the other hand, the “SQL Batch Completed” captures the following:

  1. Batch Text :- “ALTER TABLE dbo.person ADD   gender char(1) NULL
  2. Pertinent Data such as Duration, Result – Abort in this case, Rowcount ( 0 )
  3. Forensics such as Application Name, Host Name, Database ID, Database name

 

Dedicated

Can’t go anywhere else but JK as in Jonathan Kehayias.

 

Summary

In later posts will discuss creating Extended Events, Monitoring, and measuring impact on the system.

Linux – Listing Files Based on Date

 

Background

I have a need to list files on a Linux Host sorted by date.

 

ls Command

Earliest First

Command


ls -ltr -G -g | grep -v '^total' | head -5

 

Output

 

Latest First

Command


ls -lt -G -g | grep -v '^total' | head -5

Output

 

ls Command In a ditch

Unfortunately, when there are numerous files in the target folder the ls command and other file utilities can get in a ditch.

Argument List Too Long

Command


ls *.LOG

Output

Image

Text

bash: /bin/ls: Argument list too long

 

Find Command

Earliest First

Command


find . -name '*.LOG' | sort | head -n 5

Output

Command


find . -name '*.LOG' | sort | tail -n 5

Output

 

Script

Let us get a bit more complicated.

And, we will do do by picking up individual files, process them, and exit after N Number of files have been processed.


# get current folder
echo "Current working folder is $PWD"


#echo list all files in current log
#find . -name "*.LOG" | xargs -i echo " {} "


declare -i iFileID
declare -i iFileIDMax
declare FORMAT_FILE_PROCESSING

iFileID=1
iFileIDMax=1000
iFileIDMax=50
FORMAT_FILE_PROCESSING="%d  file %s \n"

#echo $FORMAT_FILE_PROCESSING

#list earliest top N LOG files
find . -name "*.LOG" 2> /dev/null  | sort | head -n $iFileIDMax  | while  IFS="" read name;
do



     printf "$FORMAT_FILE_PROCESSING" $iFileID  $name

     iFileID=$((iFileID+1));

     if [ $iFileID  -gt  $iFileIDMax ]
      then


        break;

     fi


done



 

 

Dedicated

Dedicated to ….

The Electronic Toolbox
Argument list too long when copying/deleting/moving files on Linux
Link

Summary

When needing to list files by date, one can use the ls command,

In cases where there are numerous files, the type of filtering one can do is hampered by the fact that the ls command tries to accept all the files as a batch.

A good workaround is the find command.

 

Next Up

The error that I really need to fix was the one from running “gzip S066*

Image

Textual

bash: /usr/bin/gzip: Argument list too long

 

I know it has something to do with 666

 

Reference

  1. Error – Argument list too long …
    • The Electronic Toolbox
      • Argument list too long
        Link
  2. ls command
    • TheGeekStuff.com
      • Linux ls command examples
        Linux
    •  linuxcommand.org
  3. Find Command
    • Unix.com
      • While loop, input from find command
        Link
    • StackOverflow.com
      • Display modified datetime with Find Command
        Link
    • unix.stackexchange.com
      • Only find first few matched files using find
        Link
    • Superuser.com
      • How can I find the oldest file in a directory tree
        Link
  4. Loop Control
    • The Linux Documentation Project ( tldp.org )
    • LinuxCommand.org
      • Flow Control – Part 3
        Link
  5. If Command
    • TheGeekStuff.com
      • Bash If Statement Examples ( If then fi, If then else fi, If elif else fi, Nested if )
        Link
  6. Math
    • K-State Polythecnic
      • Math in Shell Script
        Link
  7.  printf
    • Linuxconfig.org
      • Bash printf basic commands
        Link
    • Bash Hackers wiki
      • The printf command
        Link
    • Unix.StackExchange.org
      • Printf formatting with variable format – what does this var reference?
        Link
  8. variables
    • Ryan

 

Change Prompt in Bash

Background

For us that like to journal and keep notes of every command we issue and capture screens, having long prompts get in the way a bit.

Especially, when you get an incident and pick up that phone and  it says one of the DB/2 hosts is running out of Log space again.

And, the best you can do is say that if it is only at 90% it can likely suffer till the morning.

 

Current Prompt

To get the current prompt issue “echo $PS1

Image

Text

\u@\h:\w>

Revised Prompt

Let us rid ourselves of all the clutter by dropping all three; username, machine name and the current working directory are all out of here.

SESSION


export PS1='>'

 

Permanently

To make the change permanently, please edit ~/profile in your editor of choice and add same line.


					

SQL Server – SSMS – Options – “Override connection string time-out value for table designer updates”

 

Background

This is the third in a series of posts on timeouts

 

SQL Server Management Studio ( SSMS)

Designers

For those that use the Designer built into SSMS to make structural table changes, the Designer Option panel is a good place to go and make sure that the default settings are what you want them to be.

Screen Shot

Here is what that screen looks:

Initial

Post Changes

Image

Explanation
  1. Here we changed the “Transaction Time-out after:” from the default of 30 seconds to 90 seconds

 

Misnomer

I think outside of the Database World and specifically SQL Server, it is OK to use the phrase & words, but I think for a tool that targets SQL Server and only that table, a bit more care should be taken.

And, you likely ask why.

Grouping :- Override connection string time-out value for table designer updates
Phrase :- Transaction time-out after

Here are the difficulties that I have with the Group Label “Override connection string time-out value

  1. We all know that the Connection String does not expose command execution nor lock timeout
    • Command Execution can be set in the Connection or Command Object, but not in the actual connection string
    • Because of this understanding, coders have to look into the code and set the timeouts
    • They are not able to do so declaratively in the Web.config and thus not suffer the pain of recompile
  2. There is more than enough perception that Connection Open timeouts and Command Execution timeouts, are one the same

 

Microsoft – Connect

  1. Item #  :- 3130697
    • Link :- Link
      Opened By: Daniel Adeniji
      Date Opened:- 2017-March-25
      Type :- Suggestion
      Status :- Active

SQL Server – “Lock Timeouts” – Trigger Timeouts – SSMS – Table Designer

Background

In our initial post we spoke about experiencing an Outage.

Our Clients in the Development side of the house explained they have being logging “Lock Timeouts“.

We wanted to see what tools we can use to detect similar occurrences on the DB.  And, we cited SQL Server Profiler as one of those tools.

In this post, we will attempt to simulate time outs due to expired Lock requests and see how well SQL Server Profiler performs.

 

Lab

Outline

  1. Start SQL Server Profiler and set to capture event
  2. Create a bare minimum table
  3. Add data
    • Start Transaction
    • Add a couple of records
    • Temporarily leave Transaction Uncommitted
  4. In another session, return to Table designer & initiate changes
    • Add one or two new columns, attempting to make design changes

Steps

SQL Server Profiler

Here are the events that we said we will capture:

Image

Image – Events Selection

Image – Edit Filter

 

Tabulation

Tabulation – Events Selection
Event Category Event
 Locks  Lock:Cancel
 Stored Procedures  RPC:Completed
 TSQL  SQL:Batch Completed

 

 

Tabulation – Filter
Column Value
 Duration  30000

 

 

Create a Bare Minimum Table

Image

Explanation

In the screen above, we have created a table and added a couple of columns ( personID and personName).

 

Attempt to add data

Script


set nocount on
go
set XACT_ABORT on
go

use [DBLab]
go

declare @transactionComplete bit

set @transactionComplete = 0

begin tran

	insert into [dbo].[person]
	default values;


 

Table Designer

Table Designer – Adding new column – dateAdded

Image

 

Table Designer – Execution Timeout Expired

Image

Textual

'person' table
- Unable to modify table.
Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

 

SQL Server Profiler

Image

Lock:Cancel

Tabulated
Event Category Event Columns Values
 Lock  
 Lock:Cancel
 TextData
 Application  Microsoft SQL Server Management Studio
 NTUserName  dadeniji
 Duration  29999
 StartTime  2017-03-25 16:23:18.190
 EndTime  2017-03-25 16:23:18.190
 Mode  2-SCH-M
 ObjectID 110623437
 ObjectID2  110623437
 OwnerID  1-TRANSACTION
 DatabaseID  7
 DatabaseName  DBLAB
 Hostname  DADENIJI
 IntegerData2  0-LOCK

 

 

Explanation
  • You want to pay attention to ObjectID and Database ID
    • Those two will lead you to Database, Schema, and Object name
  • Mode
    • Will lead you to desired Lock
      • In our case, 2-SCH-M or Schema Modification Lock
  • And, of course Application, Host, and Username, and TextData
    • Determine causation and forensics

 

SQL: Batch Completed

Tabulated
Event Category Event Columns Values
 T-SQL  
SQL: Batch Completed
 TextData  ALTER TABLE dbo.person ADD dateAdded null
 Application  Microsoft SQL Server Management Studio
 NTUserName  dadeniji
 Duration  29999
 StartTime  2017-03-25 16:22:48.190
 EndTime  2017-03-25 16:23:18.190

 

 

Explanation
  • You want to pay attention to ObjectID and Database ID
    • Those two will lead you to Database, Schema, and Object name
  • Mode
    • Will lead you to desired Lock
      • In our case, 2-SCH-M or Schema Modification Lock
  • And, of course Application, Host, and Username, and TextData
    • Determine causation and forensics

 

 

Dedicated

Dedicated to MSFT’s SQL Server Team.

 

 

SQL Server – “Lock Timeouts” – Identifying through SQL Server Profiler

Preface

We experienced an outage this last Monday.

One of the errors logged in the WebSphere log file is the one pasted below:

Lock request time out period exceeded.

TroubleShooting

SQL Server Profiler

One of the most formidable tool for SQL Server TroubleShooting is SQL Server Profiler.

It allows the DBA to monitor several activities.

 

SQL Server Profiler – Take Care

Because it is an expensive to run it, I will suggest the following:

  1. Carefully choose which events you will like tracked
  2. Convert from Client Tracking to Server Tracing

 

SQL Server Profiler – Events

Here are the events to track on…

Tabulated

Events File Version
 Locks  
 Lock: Cancel
 Lock: Timeout
 Lock:Timeout ( timeout > 0)
 Stored Procedures
 RPC:Completed
 TSQL  
 SQL: BatchCompleted

 

Image

SQL Server Profiler – Filter

To augment the events we need to place filters and thus ensure we are not overloaded.

Tabulated

Events File Version Specific
 Duration  
 Greater or equal to
 30000  ( Please note value is in milliseconds, and so we have it set for 30 seconds )
 Exclude rows that do not contain values

 

 

Image

 

 

Dedicated

Duckworth & Borris Callens

How to find timed out statements in SQL 2005 profiler
Link

 

References

  1. How to find timed out statements in SQL 2005 profiler
    Link
  2. Transcender
    • Application Deadlock
      You are the database administrator for a banking company. You manage all the SQL Server 2008 R2 databases of the company. The company stores customer-related data in the database named Cust01. This database is accessed by most users in the company for different purposes. The users daily perform insert and updates to the database through a .NET application.
      Eric, a user in the database, complains that his transaction has frozen and that he is not able to perform any operation in the database. You find out that the problem is due to a deadlock. You want to find out the user who is the other participant in the deadlock.
      Link

 

Summary

In follow-up posts, we will take things to the Lab and do things that will trigger timeouts.