Error: “The requested operation could not be completed due to a file system limitation (mscorlib”

Background

Experiencing an error when trying to run an embedded process.  Part of the functionality that is bundled in the process, is a file copying module.

Error Message

Here is the error message :-

*** Error: The requested operation could not be completed due to a file system limitation (mscorlib) ***

Trouble Shooting

Application Message

Thankfully the particular file that is copied is logged, as well.


Copying log backup file to temporary work file. Source: '\\LABDB\Backup\hrdb_20180703104507.trn', Destination: '\\LABMirror\backup\HRDB_20180703104507.wrk'

Fast Copy

Attempted same file copy in FastCopy.

And, again thankfully it reported an error message; along with an error number.

Image

FastCopy_20180705_0712PM_v2.png

Text


WriteFileWait(The requested operation could not be completed due to a file system limitation665) : \\LABMirror\backup\HRDB_20180703104507.wrk
TotalRead  = 40,414 MB
TotalWrite = 40,289 MB
TotalFiles = 0 (0)
TotalTime  = 26:50
TransRate  = 25.0 MB/s
FileRate   = 0.00 files/s 

Explanation

  1. Error
    • Error Operation :- WriteFileWait
    • Error Description :- The requested operation could not be completed due to a file system limitation
    • Error Number :- 665

 

Error Causation

There are a few probably causes for error number 665.

Compression?

File

Finally, we checked the compression setting on others files in the targeted folder.

File Property

Image

file_20180706020000_trn_brushedup.png

Textual
  1. Size :- 2.07 GB
  2. Size on Disk :- 760 MB
Explanation

We can see that our file is compressed.

Folder Settings

Reviewed the Folder’s Advanced Attributes.

Image

AdvancedSettings_20180705_0717PM.png

Explanation

  1. Compress Contents to save disk space ( is enabled )

 

Remediation

Compression

Setting

We turned off compression on the targeted folder.

Fast Copy

Retried Fast Copy and it worked successfully.

NoError__20180705_1014PM_v2.png

 

Summary

Wished the original application displayed the OS’s error number and not just a simple error text that read “Error: The requested operation could not be completed due to a file system limitation (mscorlib)“.

Thankfully, there was enough diagnostic data in regards to the original file and destination folder that allowed us to replay via a more illuminating tool.

 

 

 

 

 

Fastcopy – Scheduling through Scripting

Background

Have a need to schedule weekly restoration of Production DB unto BETA.

As our DB is a bit big, I just did not want to use DOS Copy/XCOPY and so was happy that fastcopy is already available on server.

Code

List of files

  1. Files.txt
    • full file names of each source file
    • Delimited by newline
  2. CopyFiles.cmd
    • Invokes the fastcopy.exe
    • Added options not to display error message as we will be scheduling on Server and so UI will be counter-productive
    • Added speed=7
      • Files are copied in parallel and if we are not careful, we can overwhelm network
  3. RestoreDB.cmd
    • Calls copyfiles.cmd
    • Pause
      • Added pause and left it commented out
      • Will enable it, if we need to debug MS Windows Task Scheduler

 

Files.txt


\\HRDBPROD\SQLBackups\HRDB\HRDB.bak

CopyFiles.cmd


set "app=D:\Downloads\FastCopy\FastCopy311_x64\fastcopy.exe"

@rem set "srcfile=D:\Scripts\SyncFiles\Backup\files.txt"
set "_HOME_DIR=%CD%"

set "_srcfile=files.txt"
set "_srcfileFull=%_HOME_DIR%\%_srcfile%"
set "_logFolder=%_HOME_DIR%\log\"

set "_destFolder=F:\Microsoft\SQLServer\SQLRestore"

@rem if Folder Log does not exist, please create it
if not exist log mkdir log

@rem Initiate Fastcopy
%app% /cmd=diff ^
	 /srcfile=%_srcfileFull% ^
         /auto_close /force_close ^
	 /estimate ^
	 /no_ui ^
	 /no_confirm_del ^
	 /no_confirm_stop ^
	 /log=true ^
	 /logfile=%_logFolder%\fastcopy.log ^
	 /filelog=%_logFolder%\filecopy.log ^
	 /speed=7 ^
	 /to=%_destFolder%"

RestoreDB.cmd

copyfiles.cmd
@rem pause

Fastcopy Log files

Fastcopy produces plentiful throughput and error info.

There are two file fastcopy.log and filecopy.log

fastcopy.log

Here is what fastcopy.log looks like:

fastcopy

 

Monitoring

Windows Task Manager

WindowsTaskManager-20160513-0918AM

Explanation

  1. Our process, fastcopy.exe, is using some I/O

Resource Monitor

One can monitor ongoing network and IO throughput via built-in MS Windows Resource Monitor.

Disk

ResourceMonitor-20160513-0847AM

Explanation

  1. Note that Image is not fastcopy.exe, but the system
  2. We are getting about 150MB/sec for our copying process
  3. Overall 180 MB/sec IO

 

Network

ResourceMonitor-NetworkIO-20160513-0907AM

 

Explanation

  1. Again, Image is not fastcopy.exe, but the system
  2. Remote Port is 445
  3. Marked Latency

It seems obvious that Network IO is not reflecting true and overall Network I/O

 

Scheduling

MS Windows Task Scheduler

Task – General

Customization

  1. Pay attention to “Configure for:
    • You are given the choices of
      • Windows 7, Windows Server 2008 R2
      • Windows Vista, Windows Server 2008
    • To avoid errors, please choose “Windows Server 2008” even on “Windows Server 2008/R2

 

Task-General

 

Task-General-ConfigureFor

 

Event Viewer

Looking through MS Windows Event Viewer is a great way to troubleshoot scheduled tasks.

The specific branch to look at is Microsoft-Windows-TaskScheduler/Operational

Branch

Branch-Microsoft-Windows

Branch-Microsoft-Windows-TaskScheduler-Operational

 

Filter

Again, there is a lot to choose from and so please filter on …

Filtering Elements

  1. Event ID :-
    • ID is 201
      • 201 means “Action Completed
    • User is the name the task is scheduled to “run as”

 

FilterCurrentLog

Sample Entries

Return Code – 4294967295

EventDetails-4294967295

Return Code – 1

EventDetails-1

Return Code – 0

EventDetails-0

 

Interpretation

Log – General Causation
 Task Scheduler successfully completed task “\Corp\Restore DB” , instance “{76835bc9-9cb6-4d08-82ae-1bbe783f4d71}” , action “C:\Windows\SYSTEM32\cmd.exe” with return code 4294967295. Invalid arguments passed to fast copy
 Task Scheduler successfully completed task “\Corp\Restore DB” , instance “{482d0822-91ea-43a3-abce-fd7a53d14e1d}” , action “C:\Windows\SYSTEM32\cmd.exe” with return code 1. The Task has been configured for “Windows 7, MS Windows 2008/R2“.  Please configure for “Windows Vista, MS Windows 2008“.
 Task Scheduler successfully completed task “\Corp\Restore DB” , instance “{a4c53952-7caa-49f9-8df2-85b11b259b13}” , action “C:\Windows\SYSTEM32\cmd.exe” with return code 0.  We are good

 

 

Performance Comparison

Microsoft Windows 2008/R2

Xcopy


set appLogger="D:\Program Files (x86)\GnuWin32\bin\tee.exe"
set _appLoggerOption=--append 
set "_HOME_DIR=%CD%"
set "_LOG_FOLDER=%CD%\LOG"
set "_LOG_FILE=%CD%\LOG\benchmark_%DATEANDTIME%.log"
set "_SRC_FILES=\\HRDB\f$\SQLBackups\*.bak"
set "_DEST_FOLDER=F:\Microsoft\SQLServer\SQLRestore\"
set "_OPTIONS_XCOPY=/D /Y"

if not exist %_LOG_FOLDER% mkdir %_LOG_FOLDER%

set DATESTAMP=%DATE:~10,4%_%DATE:~4,2%_%DATE:~7,2%
set TIMESTAMP=%TIME:~0,2%_%TIME:~3,2%_%TIME:~6,2%
set DATEANDTIME=%DATESTAMP%_%TIMESTAMP%

ECHO "Before Copy %DATEANDTIME%  .... " | %appLogger% %_appLoggerOption% %_LOG_FILE%

xcopy %_SRC_FILES%  %_DEST_FOLDER% %_OPTIONS_XCOPY% | %appLogger% %_appLoggerOption% %_LOG_FILE%

set DATESTAMP=%DATE:~10,4%_%DATE:~4,2%_%DATE:~7,2%
set TIMESTAMP=%TIME:~0,2%_%TIME:~3,2%_%TIME:~6,2%
set DATEANDTIME=%DATESTAMP%_%TIMESTAMP%

ECHO "After Copy %DATEANDTIME%" | %appLogger% %_appLoggerOption% %_LOG_FILE%

Resource Monitor

Disk I/O

xcopy-Disk-20160526-0939AM

 

Network I/O

xcopy-network-20160526-0939AM

 

Performance Monitor

PerfomanceMonitor-20160526-1212

 

Matrix

 

Tool Disk I/O Network I/O Time Taken
 fastcopy 165 MB/sec 4 Hours 30 minutes
 xcopy 110 MB/sec  115 MB/sec 1 Hour

 

 

Summary

I think the SMB protocol has been greatly optimized in MS Windows 2008/R2 and later OS and so this only applies to OSes prior to that version .

Even if then…

Quick, Quick Copy of datafiles across WAN

As I tried to sneak off for a late Post Office run, ran into the Director.

He was standing, guarding the elevator.  Not quite, but I still thought it best to let him know how I am doing with one of his more pressing project.

Told him the database engine is installed and now I am starting with copying the database files from the old system unto the new one.

He asked me how much longer would it take and I really wanted to let him know that I wish I knew.

But, did not think he was in much mood for the vulgarity of my usual “smart remarks”.

As so upon returning from the Post Office run, I  goggled for “copy fast windows 2003”.

I ran into a few promising links.  The one that caught my eye was from the SQLTeam.com

http://weblogs.sqlteam.com/tarad/archive/2009/03/03/FastCopy—fastest-Windows-copy-product.aspx

Since the database files are MS SQL Server data and log files, I thought I might as well peruse MS SQL Server related blog.

Well, it proved to be useful and now I have a few more more utilities to add to Robocopy.

  1. FastCopy
    http://ipmsg.org/tools/fastcopy.html.en
  2. Code Sector – TeraCopy
    http://www.codesector.com/download.php