Introduction

What is HttpCopy?

HttpCopy is a utility designed to replicate files from a remote server to the local system. It provides the following features:
  • Byte-level restart. If a copy operation is interrupted because the connection is lost or time runs out (see below), the job can be relaunched later and copying will resume from the point where it was interrupted.
  • Bandwidth control. A maximum bandwidth can be specified that will not be exceeded during the copy operation.
  • Maximum duration. If bandwidth is available over the WAN links only during certain times of the day, a time to stop can be specified.
  • Will replicate individual files or directories (recursive).

HttpCopy is freely available to customers of LANDesk products. However, it remains copyrighted material of LANDesk Software. The user is not allowed to redistribute it to third parties.

Disclaimer

This program has undergone limited testing and is not supported by LANDesk Software customer support. Nothing in this document constitutes a guarantee, warranty, or license, express or implied. LANDesk Software disclaims all liability for all such guaranties, warranties, and licenses, including but not limited to: fitness for a particular purpose; merchantability; non-infringement of intellectual property or other rights of any third party or of LANDesk Software; indemnity; and all others.

Direct any comments to jan.buelens@landesk.com


New Page 1

Prerequisites and theory of operation

The web server must support the HTTP 1.1 byte range feature. The byte range feature is essential for restartable copying. Without it, HttpCopy will still work, but the file that was being copied at the time the job was interrupted will need to be copied again from the beginning.

For directory level copying, the web server needs to allow directory browsing.

HttpCopy relies on the web server sending the file’s last modified time. If the server’s response does not include a last modified time, the file’s time stamp will be today’s date and subsequent HttpCopy jobs may copy the file unnecessarily.

HttpCopy uses the “ETag” header defined by HTTP 1.1 to avoid inconsistencies in the case when a copy is restarted and the file has changed since the time when an earlier job was interrupted. This causes an incomplete copy of the out-of-date file to be discarded when the job is restarted.

HttpCopy uses two local directories, a work directory (WorkDir) and a download directory (DownloadDir). Files are first stored in WorkDir as they are being received. They are moved to DownloadDir when complete. If a job is interrupted and there are incomplete files, they remain in WorkDir. WorkDir is also used to store a file’s ETag. Restarting an interrupted file copy is not possible without the file’s ETag.

HttpCopy keeps a log of all files copied. The log file is HttpCopyLog.csv in WorkDir. The information that is logged includes date and time, filename, file size, whether or not the file was complete, elapsed time and average transfer speed. Information about files that were not copied (because same file already exists) is not logged.


Installing and using HttpCopy

Installation

There is no installation. Just copy the executable to a directory of your choice and run it. To use the byte-level differencing feature, copy libdfp.dll to the same directory. Hcmkdiff.exe and libdf.dll are not required at httpcopy run-time.

Using HttpCopy

You can run HttpCopy either in interactive mode, specifying the required parameters through a GUI, or in unattended mode, specifying the required parameters from the command line or in an INI file. One way to build the INI file for unattended mode is to run the GUI and click the Configure button. When you have specified all the parameters (including unattended or silent checkboxes), exit the program. You now have an httpcopy.ini file ready for unattended operation.

HttpCopy command line syntax

The parameters needed to run HttpCopy in unattended mode can be specified either from the command line, or in an INI file. Command line parameters override equivalent parameters in the INI file. You probably want to use the GUI to build the INI file.

HttpCopy can copy either a single URL (individual file or directory) specified on the command line, or multiple files listed in an INI file.

Command line syntax:

HTTPCOPY [parameters …]

Valid parameters are described in the table below. All parameters (with the exception of the last 3) can be abbreviated to 3 characters, e.g. /sto=08:00 and /stop=08:00 are equivalent to /stopat=08:00. Abbreviations are allowed only on the command line, not in the INI file. The last 3 parameters (retries, retyInterval, LogDebug) can only be abbreviated to 5 characters.

/ini=<ini_file> | INI file. This can be a URL. Default is httpcopy.ini in same directory as the EXE.
/WorkDir=<local_path> | Work directory. Files being received are first created here. When a file is complete, it is moved to <DownloadDir>. There is no default. This parameter must be specified either on the command line, or in the INI file
/DownloadDir=<local_path> | Download directory. When a received file is complete, it is moved from the work directory to the download directory. There is no default. This parameter must be specified either on the command line, or in the INI file.
/url=<url> File or directory to be downloaded. Valid formats include
http://myserver/packages/pkg.exe
//myserver/packages/pkg.exe
myserver/packages/pkg.exe
http://myserver/packages/
http://myserver/packages
/user=<user> | User name (if required to access the URL from which to download). This user name is also used to access the ini file (if that is a URL) and the profile info (if that is a URL).
/password=<password> | Password (if required)
/maxtime=<time> Maximum duration. Default unlimited. Examples:
/maxtime=3600 (3600 seconds)
/maxtime=30:00 (30 minutes)
/maxtime=2:0:0 (2 hours)
/StopAt=<time> Stop at specified time (24 hour clock, local time). Examples:
/stopat=23:30
/stopat=07:59:30
If both maxtime and stopat are specified, the job will stop at the earliest of the two values.
/StartAt=<time> Do not run before specified time (24 hour clock, local time). If httpcopy is started outside of the time window specified by (start at, stop at), it will exit with an error. For more info, see Specifying a time window.
Examples:
/startat=23:30
/startat=07:59:30

Caution: specifying this parameter does not cause HttpCopy to start at the specified time. It only causes it to exit when started at the "wrong" time. If you want HttpCopy to start at a specific time, use a scheduler.

/speed=<BytesPerSecond> Maximum transfer rate. Default unlimited. Examples:
/speed=8192 (8 KB/s, 64 Kbit/s) /speed=8K (equivalent to above)
/delete=1 Delete orphan files. If set to 1, only those files that, by current settings, are to be replicated from the source server are kept in DownloadDir. Any other files will be deleted. Situations in which files will be deleted include the following:
  • If files were deleted from the source server, the same files will be deleted from the local machine.
  • If a machine is removed from a profile, the files that came with that profiles will be deleted. See profiles.
  • If files exist in DownloadDir or WorkDir that, by current httpcopy setting, do not belong there (because they do not satisfy the replication rules), they will be deleted.

Caution: inappropriate use of the delete parameter can cause irreversible damage. Example: if you set "DownloadDir=C:\" and "delete=1", httpcopy will wipe your C: drive.

/unattended=1 | Unattended mode. The GUI will still be shown, but all dialog boxes will be closed automatically after a timeout (see DlgTimeout parameter). See unattended mode.
/silent=1 | Silent mode. Assumes unattended.
/keepdir=1 | Keep directory structure. Default 1. If set to 0, the directory structure is "flattened".
/DlgTimeout=<seconds> | Dialog timeout. In unattended mode, dialogs will be closed after the specified number of seconds. Default 10. The timeout will be used to close message boxes displayed when an error occurs, as well as to close the main httpcopy dialog when the program terminates.
/log=<generations> | Log file generations. HttpCopy keeps a log file called HttpCopyLog.csv in the work directory. This parameter indicates the number of generations of the log file that should be preserved. The old generations will be named HttpCopyLog001.csv, HttpCopyLog002.csv etc. The default is 5.
/utf8=1 | Use UTF-8 encoding for log files. See here for details.
/profileList=<path> | Path to the profile list file. This can be a URL. See here for details.
/name=<name> | Normally, the machine running HttpCopy uses its machine name to look up its name in the profile list file. If you want to use a different name, use this parameter. See here for details.
/AvoidHead=1 | Use HTTP GET requests instead of HEAD requests where possible. See known issues. Do not use this parameter unless you are sure you have the problem it is meant to work around.
/Retries=<retries> | Number of retries. Default is 3.
/RetryInterval=<seconds> | Seconds between retries. Default is 3.
/LogDebug=1 | Set to 1 to log additional info in the log. Default is 0.


New Page 1

HttpCopy.ini File

All of the command line parameters listed here (except /ini) can also be specified in the INI file. Parameters can’t be abbreviated in the INI file. By default, the ini file is HttpCopy.ini in the same directory as the httcopy.exe executable.

You will probably want to use the GUI to build the INI file.

INI file example 1:

[Params]
URL=http://myserver/packages
speed=8192
StopAt=08:00
DlgTimeout=5
DownLoadDir=c:\packages
WorkDir=c:\Work

[Files]
acrobat.exe
treesize.exe

This INI file requests two files to be downloaded to c:\packages. If you used the GUI, the [Files] section is the list that you specified under the Files tab. The two files are http://myserver/packages/acrobat.exe and http://myserver/packages/treesize.exe. Note that a file list can only be specified in the INI file – not from the command line. The file list will be ignored if a URL is specified on the command line.

HttpCopy will not copy the files again if they already exist on the local system. HttpCopy uses the file size and date (last modification time) to check if an existing file needs to be copied again.

The maximum bandwidth used during the transfer is 8192 bytes/sec (64 Kbps). The transfer will be stopped at 8 AM.

INI file example 2:

[Params]
URL=http://myserver/packages
maxSpeed=8192
StopAt=08:00
DlgTimeout=5
DownLoadDir=c:\packages
WorkDir=c:\Work

This INI file does not list any files. If the specified URL is a directory, all files within the directory will be downloaded to c:\packages. The copying is recursive.

INI file example 3:

[Params]
URL=http://myserver/packages
maxSpeed=8192
StopAt=08:00
DlgTimeout=5
DownLoadDir=c:\packages
WorkDir=c:\Work

[Files]
acrobat.exe
treesize.exe
Office2003
IE6

[Exclude]
Office2003/SP1

This INI file is like example 1, except that the [Files] list contains a mix of individual files and folders (Office2003 and IE6 are presumably folders). Also, there is an [Exclude] list. Like the [Files] list, the [Exclude] list can be a mix of files and folders. See here how to create the exclude list from the GUI.


Introduction

The HttpCopy GUI

When you launch HttpCopy, the following dialog is displayed:

The Start Download button is greyed out until enough parameters are specified to start the session. These parameters may be specified on the command line, in httpcopy.ini, or by clicking Configure and completing the dialogs shown below:

For Delete orphan files, see here.
For Keep directory structure, see here.
For Do not run before, see here.

If you want to synchronise the URL defined under the Paths tab in its entirety (all files and directories), leave blank. If you only want to specify specific files and directories, specify them here (paths relative to the base URL define under the Paths tab).
More info on profiles here.

When you complete these dialogs, the info is saved in the httpcopy.ini file.

The screenshot below shows the program in action:

The 3 progress bar reflect the status as follows:

  • The top progress bar shows the progress of the current file.
  • The middle progress bar shows the progress of the session.
  • The bottom progress bar shows the the overall progress.

In the above example, there are 34 files totalling 445 MB on the source server. Some of these files had already been fully or partially copied in previous sessions. Therefore, as the middle progress bar indicates, the data to be downloaded during this session is 11 files totalling 236 MB.


Unattended mode

Using the GUI to build the INI file

To run HttpCopy in unattended mode, you need to build an INI file. Rather than building the INI file manually, run HttpCopy in interactive mode and specify all the necessary parameters. HttpCopy saves all the settings in an INI file that you can then use in unattended mode.

Running HttpCopy in unattended mode.

To run HttpCopy in unattended mode, specify a /unattended=1 command line parameter or include an unattended=1 parameter in the INI file.

In unattended mode, the main HttpCopy dialog window remains visible, including the cancel button. To hide the dialog, use the silent parameter. When you set the silent parameter using the GUI, silent mode takes effect the next time HttpCopy runs.

In unattended mode, HttpCopy waits 5 seconds before starting. If you click the Configure button within these 5 seconds, you inhibit the automatic start. This provides an opportunity to change settings in an unattended job without manually editing the INI file.

Running HttpCopy in a batch file

You can test success or failure of HttpCopy in a batch file by examining its return code (errorlevel). The return code obeys the following rules:

  1. When HttpCopy terminates because of a syntax error, the return code is 9999.
  2. When HttpCopy was unable to read the ini file specified on the command line, the return code is 9998.
  3. When HttpCopy was started outside the (startat, stopat) time window, the program exits with a return code of 9997.
  4. When HttpCopy terminates because of an unrecoverable Win32 error, the return code is the Win32 error code.
  5. When HttpCopy terminates normally but one or more files could not be copied due to an error, the return code is 1.
  6. When HttpCopy terminates because of the maximum time limit, the return code is 0.

The return code alone does not distinguish between a session that terminated successfully and a session that timed out. Assuming there were no errors, the return code will be 0 either way. To allow a batch file to make this distinction, include a CALL SETSTAT.BAT in the batch file that launches HttpCopy. SETSTAT.BAT is a batch file generated by HttpCopy in the "Work Directory". It sets a number of environment variables that you can then test in your batch file. Below is a sample SetStat.bat file:

SET HC_SUMMARY="Stopped (time limit). Copied: 23 files, 197 MB. Remaining: 11 files, 247 MB. Failed: 3 files."
SET HC_DONE=0
SET HC_TIMEOUT=1

HC_DONE will have a value of 0 (not all work was done, either because there were errors, or because the session timed out) or 1 (all work done).

HC_TIMEOUT will have a value of 0 (no timeout) or 1 (the session timed out, or the "Stop Download" button was pressed in the GUI.

The following batch file shows how some of this works:

del workdir\setstat.bat
HttpCopy
REM line below assumes that Work Directory as defined in httpcopy.ini is "workdir"
call workdir\setstat.bat
if %errorlevel%==9999 echo HttpCopy syntax error
if errorlevel 2 echo HttpCopy terminated abnormally
if %errorlevel%==1 echo HttpCopy terminated with errors
if %errorlevel%==0 echo HttpCopy terminated with no errors
if "%HC_DONE%"=="1" echo HttpCopy terminated with all work done
if "%HC_TIMEOUT%"=="1" echo HttpCopy timed out
echo HttpCopy summary: %HC_SUMMARY%

Note that SetStat.bat is not always generated when there is a syntax error (exit code 9999). Replace workdir with the correct path of the httpcopy work directory (as specified here or here).

Running HttpCopy in a LANDesk Management Suite Software Distribution job

One way to schedule regular HttpCopy jobs is to use LDMS Software Distribution. In LDMS 8.5 and later, you would use a package of type "batch". This allows you to display the HttpCopy summary at the console, in the "message" column of the status line. Use the sdclient /msg feature to achieve this, as in the following example:

"c:\program files\landesk\ldclient\sdclient.exe" /msg=Running
HttpCopy
call workdir\setstat.bat
SET HC_ERRLEVEL=%errorlevel%
"c:\program files\landesk\ldclient\sdclient.exe" /msg=%HC_SUMMARY%
if %HC_ERRLEVEL%==0 goto end
exit /b %HC_ERRLEVEL%
:end
Replace workdir with the correct path of the httpcopy work directory (as specified here or here).

The result at the console might look something like this:

Specifying a time window

HttpCopy is designed to be launched on e.g. a daily basis by an external scheduler, such as the LDMS core server's scheduler, or the LDMS local scheduler, or some other scheduler. In the above example, the core server's scheduler is used, which has the advantage that status info can be fed back to the LDMS console. Suppose you need to restrict HttpCopy to a specific time window, e.g. from 2 AM to 6 AM. You would set up your scheduler to start HttpCopy every day at 2 AM. You would use the StopAt parameter to force HttpCopy to stop at 6 AM.

When using the LDMS core server scheduler, this leaves one potential problem. You might have specified a start time of e.g. 2 AM for an LDMS scheduled task. But if you have a large number of target machines, the task will start at 2 AM only for the first <number of simultaneous distributions> targets, where <number of simultaneous distributions> is a parameter of the distribution method. Typical values are from 60 to 240. The remaining targets are put on a queue. Each time one of the active targets reports end-of-task, one of the queued machines gets serviced. Therefore, if the total number of targets exceeds the distribution method's <number of simultaneous distributions>, you can't be sure when the task will start on a given target.

If you need to be sure that HttpCopy is not started outside a specified time window, either by mistake or due to the effect explained in the previous paragraph, use the do not start before setting, which can be set from the GUI or in the INI file ("StartAt" parameter). Caution: this parameter does not cause HttpCopy to start at the specified time. It only causes HttpCopy to exit with an error when started at the "wrong" time. Use a scheduler to start HttpCopy at a specific time.

Profiles

If different package servers need to receive different subsets of the packages at the master server, profiles can help to centrally manage which servers receive which packages. Instead of file lists specified locally in httpcopy.ini files, you refer each package server to a profile list file using the ProfileList parameter. You can set this parameter from the GUI, see here. The ProfileList file is a list of "profiles" and associated package servers.

To use profiles, put a line similar to the following in the [Params] section of the ini file:

ProfileList=http://master/packages/ini/profiles.ini

The path can be a UNC or URL path. The profiles.ini file must have the following format:

[profile1]
server1
server2

[profile2]
server3

[profile3]
server1
server3

[common]
*

Each section header is the name of an ini file that lists files and directories to be copied. Under the section header is a list of servers (NetBios machine names) that are to copy the file and directories listed in the ini file. In the example above, server1 is listed under [profile1] and under [profile3]. When httpcopy runs on server1, it will look for files profile1.ini and profile3.ini and use the file lists in them. These files must exist in the same folder as the ProfileList. Under the [common] section, we find '*'. The machine name matching supports wildcards, so server 1 will also use common.ini. The effective file list will be the combination of the file list in server1's local httpcopy.ini, plus profile1.ini, profile3.ini and common.ini.

By default, the machine running HttCopy uses its NetBios machine name to look up its profile membership. If you want to use a different name, use the /name parameter.

Each of the ini files lists files and directories to be replicated, in the same format as the [Files] section of the httcopy.ini file. Below is an example:
[Files]
pkg1.exe
dir1

The log file

Full details of an httpcopy session can be found in the log file. The log file is a csv file created in the work directory. Below is a sample log file. A one minute time limit was specified, therefore the session times out with a "Max time elapsed" status.

As of V3.4, httpcopy is unicode based. This allows non-ANSI characters (e.g. Cyrillic, Japanese...) in filenames. If you are going to have non-ANSI characters in filenames, you must choose between two options:

  • Use UTF-8: the log file will be written in UTF-8 encoding. Advantage: non-ANSI characters are displayed correctly when the log file is opened in many tools, including notepad. Disadvantage: at the time of writing, Excel does not understand UTF-8. The file will be "generally readable" in excel, but with with some "funny characters".
  • Do not use UTF-8: the log file will be written in your native 8 bit character set, e.g. ANSI. Advantage: excel will not display "funny characters". Disadvantage: if there are filename with non-ANSI characters, the non-ANSI characters are replaced with question marks.

  • If an English-only environment, or more generally in an ANSI-only environment (e.g. West-European languages only), leave blank, i.e. do not UTF-8.
  • This setting only affects the way filenames appear in the log. It makes no difference in terms of the filename under which the file is created.


Unpack-on-arrival feature

Introduction: single-file packages or multi-file packages?

If you are using httpcopy to distribute software distribution packages, the recommendation is to use a packaging method where one package = one file. There are two possible issues with multi-file packages:
  1. Performance. If you need to replicate a large number of files (tens or hundreds of thousands), httpcopy may take a long time simply to compare source and target directories and decide which files need copying. Inevitably, it takes a finite amount of time to look at each file's size and time stamp. Given enough files, total run time may become unacceptably long.
  2. Package integrity. Httpcopy guarantees that only completed files will exist in the download folder. If a package consists of multiple files, there will be a time window in which only some of the files are present.

On the other hand, there may be situations where single file packages are not practical. If you have multi-file packages with thousands of files in them, and you are unable to repackage them to single files, consider using the technique explained below.

Httpcopy unpack-on-arrival feature - summary

The main idea is to archive packages to single files and unpack them on arrival. Archiving the packages is you, the user's responsibility. The unpacking is done by httpcopy, using instructions provided by you in control files.

Sequence of events:

  1. You use an archiving tool of your choice to compress each package to a single file. One critical requirement is that there must be a command line tool to unpack the archived packages.
  2. For each archived package, you create a control file in which you store appropriate command line(s) to unpack the archive.
  3. What httpcopy replicates are only the archived packages and the control files.
  4. While downloading files, httpcopy recognises the control files and runs the commands contained in them to unpack the archived packages.
  5. To avoid the overhead of storing both packed and unpacked copies of packages on the target systems, the packed copy can be deleted after unpacking. A stub file is created to stand in for the deleted file, storing the size and time stamp of the deleted file. Next time httpcopy runs, it will see the stub file and not download the file again unless it has changed.

Example: suppose you have a large MSI package called MYPKG consisting of thousands of files. Use winzip (or any other archiving tool) to pack all the files into a single package called e.g. MYPKG.ZIP. Also create a control file called MYPKG.ZIP_unpk.ini, in which you store appropriate command lines to unpack the archive. For this example, the control file might look as follows:

[params]
delete=1

[commands]
wzunzip -d -o -e MYPKG.ZIP d:\packages\MYPKG

The control file (MYPKG.zip_unpk.ini) must exist in the same folder on the source server as the file to which it refers (MYPKG.ZIP). The unzip tool shown in this example is wzunzip.exe, which is a command line tool available from www.winzip.com. This is only an example. It is the user's responsibility to find an appropriate command line driven unpacking tool. The delete=1 parameter is an instruction to delete the packed file after unpacking.

Assuming you have only this one package (MYPKG.ZIP) to replicate, an httpcopy session will run as follows:

  1. At the start of the session, there are 2 files in the source folder (MYPKG.ZIP and MYPKG.ZIP_unpk.ini), neither of which exist in the target machine's download folder.
  2. Httpcopy downloads MYPKG.ZIP to the work folder and moves it to the download folder when it is complete.
  3. Httpcopy downloads MYPKG.ZIP_unpk.ini to the work folder and, when moving it to the download folder, notices from the filename that it is a control file. Httpcopy checks if the corresponding archive file (MYPKG.ZIP) exists. If the file exists, it runs the command line(s) stored in the control file.
  4. Httpcopy has no knowledge of what the command line(s) do. It just runs them and waits for them to terminate. In our example, the wzunzip command line tool unzips the package to d:\packages\MYPKG.
  5. On completion of the commands stored in the control file, if there is a parameter delete=1 in the [params] section, httpcopy deletes the archived package (MYPKG.ZIP) from the download folder and creates a small stub file (MYPKG.@stub@) in its place, in which it stores the file's original size and time stamp.

    In a subsequent httpcopy session, the following will happen:
  6. Httpcopy compares source and target folders and decides which files need downloading. The MYPKG.ZIP file exists in the source folder and not in the download folder. But in its place, there is the stub file. If the date and size stored in the stub file still match the source file, the file will not be copied again.

Command line parameters

The way httpcopy runs the commands contained in the control file is by writing them to a batch file (e.g. MYPKG.ZIP_unpk.bat). It then runs this batch file, passing it the full path of the archived package as a command line parameter. This makes it possible to make the unzip command line generic, e.g. if the archive is called MYPKG.ZIP, we could replace the following command line:

wzunzip -d -o -e MYPKG.ZIP d:\packages\MYPKG

with the following:

wzunzip -d -o -e "%~nx1" %~d1\packages\"%~n1"

This syntax should be generic enough to allow the same command line to be used for many different packages. If you are not familiar with syntax like %~nx1, note that this is generic syntax that can be used in any batch file, see e.g. http://technet.microsoft.com/en-us/library/cc732835.aspx (if the article is no longer there, try googling "to a file name and extension only"). In the example above, %~nx1 expands to the filename + extension (i.e. MYPKG.ZIP), %~n1 to the filename only (i.e. MYPKG).

Note that the full pathname of the archive file is not required because the batch file is invoked with current directory = the folder of the archive file.

Package integrity

Performance is only one problem we are trying to solve. Another one is "package integrity", i.e. we want all files to be there before we "publish" the package. The unpack-on-arrival mechanism reduces the time window in which the package is incomplete to the time needed to unpack. If it is desirable to reduce this time window even further, you could unpack to a temp folder, which you only rename at the end, i.e. something like this:

[commands]
wzunzip -d -o -e "%~nx1" %~d1\packages\"%~n1.tmp"
ren %~d1\packages\"%~n1.tmp" "%~n1"

Delete handling

As explained so far, unpack-on-arrival provides mechanisms to transfer packages initially and keep them up-to-date. But what happens when a package is deleted? Normally, when a file is deleted from the source server, the file will also be deleted from the local download folder next time httpcopy runs. With unpack-on-arrival, the original file is no longer there (it has been unpacked and deleted) and httpcopy has no knowledge of where it was unpacked to. If you want the unpacked files to be deleted when the archived package is deleted, you need to include an [OnDelete] section in your _unpk.ini files, storing commands to be run when the package is deleted.

The details of the feature are as follows:

  1. You include an [OnDelete] section in your control file (_unpk.ini file).
  2. At the time httpcopy initially receives the archived package and the control file, it runs the commands stored in the [commands] section of the control file to unpack the archive. It then deletes the archived package and creates a stub file.
  3. When creating the stub file, if there is an [OnDelete] section in the _unpk.ini file, it is copied to the stub file.
  4. If at some later time the original archived package no longer exists on the source server, httpcopy will delete the stub file from the download folder. On doing so, it will notice the [OnDelete] section in the stub file and run the commands contained in it.

The [OnDelete] section will also be executed when an archived package is updated. On receiving the updated archived package, httpcopy will, as per usual, run the commands stored in the [commands] section to unpack the package, after which it will delete the archived package and create a stub file. But if a stub file already exists for an earlier version of the same package, httpcopy will first run the [OnDelete] section contained in the stub file. The [OnDelete] section of the old stub file is run before the [commands] section of the _unpk.ini file, i.e. the old copy of the unpacked package will have been deleted before the new package is unpacked.

The same command line parameters (see above) can be used in the [OnDelete] section as in the [commands] section. Httpcopy runs the [OnDelete] section by creating a batch file (e.g. MYPKG.ZIP_del.bat), which is called with the full path of the archived package (which no longer exists) as a command line parameter. Although the archived package no longer exists, it is still useful as command line parameter, e.g. if the following command line was used to unpack the package:

[commands]
wzunzip -d -o -e "%~nx1" %~d1\packages\"%~n1"

... then the following command line can be used to delete the unpacked package:

[OnDelete]
rmdir /s /q %~d1\packages\"%~n1"

As a protection against possible corruption or tampering, a sumcheck is generated at the time the [OnDelete] section is copied to the stub file and checked before executing it. The checksum algorithm includes the machine name, path, filesize and file date, therefore it is "not easy" to simply paste malicious content into the stub file.

Detailed syntax of the control file

The control file must exist in the same folder as the archived package to which it refers. Its filename must be the name of the archived file (including extension), with a suffix of _unpk.ini appended. Example: the control file for MYPKG.ZIP is MYPKG.ZIP_unpk.ini.

There are 2 or 3 sections in the control file:

[params]

[commands]

[OnDelete]

Parameters in the [params] section:

Parameter | Values | Explanation
delete | 0 | 1 | If delete=1, httpcopy will delete the archive file after executing the [commands] section and create a stub file. Default is 0.
conditional | 0 | 1 | Modifies the meaning of delete=1. If delete = 1 and conditional = 1, the archive file will not be deleted if [commands] exits with a non-zero return code.
visible | 0 | 1 | If visible=1, a "DOS box" will appear when httpcopy executes the [commands] or [OnDelete] section. Default is 0.
priority | LOW | BELOWNORMAL | NORMAL | Can be used to run [commands] and [OnDelete] at reduced CPU priority. Default is normal.

Under [commands], specify the command lines that will unpack the archived package.

Under [OnDelete] (optional), specify the command lines that will delete the unpacked package.

Both the [commands] and [OnDelete] sections can use parameters as explained above.

When is the unpacking done?

The explanation given above is not complete. During the initial download of the archived packages, each package will be unpacked immediately upon being received. What triggers the unpacking is the control file (_unpk.ini) being received. During subsequent sessions, however, there may be a considerable delay between the time a package is received and the time it is unpacked. Suppose you have multiple archived packages in the folder. You have updated some or all of the archived packages, but not the control files. In this situation, since the control files will not be downloaded again, packages will not be unpacked immediately upon being received. Instead, it will take until the complete folder has been processed before httpcopy will look for the control files.

If you want to avoid this delay, "touch" the control files when you update the packages on the source server.

Logging

If visible = 0, stdout and stderr of [commands] and of [OnDelete] will be saved in log files named filename_unpk.log and filename_del.log. These files are created in the work directory. In addition, non-zero return codes are reported in the log file.


Byte-level Differncing

Byte-level Differencing

Overview

Byte-level differencing is a feature that can save large amounts of bandwidth when updating existing files. When a new version of a package is "not too different" from the old version, try building a difference file. Downloading only the difference may be a much less costly operation than sending the new package.

Example: you have a 1 GB package. You add one file to it. Do you need to transfer the complete package again? Create a difference file instead and you save 1 GB of bandwidth.

Process summary

To use byte-level differencing, you would follow these steps (the filenames serve purely as an example):

  1. You have an existing version of a package called MyPkgV1.zip
  2. You build a new version of the package called MyPkgV2.zip
  3. You create a difference file (MyPkg.df) by running the following command line: hcmkdiff /old=MyPkgV1.zip  /new=MyPkgV2.zip  /df=MyPkg.df
  4. On your source server, you copy MyPkgV2.zip and MyPkg.df to the same folder where MyPkgV1.zip resides. You may chose to delete MyPkgV1.zip, or you may chose to keep it.
  5. In the next httpcopy session, httpcopy sees the MyPkg.df file and decides from its extension that it may be a difference file. Looking inside the difference file, it finds all the info needed to create MyPkgV2.zip based on MyPkgV1.zip and MyPkg.df: old filename, filesize and date, new filename, filesize and date.
  6. Httpcopy will not download MyPkgV2.zip because it has already been created based on MyPkgV1.zip and MyPkg.df.

Notes:

  1. The above was written with a .zip file as an example. Please realize that it is only an example. There is no assumption that the files involved are zip files.
  2. The only naming convention is that difference files must have a .df extension. But the base name of the difference file is not subject to any naming convention. In the above example, the difference file was called MyPkg.df. You could have called it xyz.df and the example would still have worked. If there are ordinary data files with a .df extension, there will be no ill effects except for a warning message in the log.
  3. There is no danger that the wrong difference file will be applied to the wrong base file. Control info contained in the .df file includes not only the filename of the base file, but also its size, date and MD5 checksum. Therefore, any attempt to apply the wrong difference file to the wrong base file will fail.
  4. To understand how the process works, it may be important to take note of the following: httpcopy processes .df files before all other files. Therefore, in the above example, when time comes to download MyPkg2.zip, the file will not need to be downloaded because the mechanism described above will already have created it.
  5. Byte-level differencing and unpack-on-arrival can work together. Your package might unpack itself as described under unpack-on-arrival. If you keep a copy of the original archive file (delete=0 in the filename_unpk.ini file), you can update the package by sending only a difference file.
  6. The old file must exist on the local machine at the time the diff file is received. In the example above, the old file (MyPkg1.zip) existed on the local machine before the start of the httpcopy session and the diff file was therefore applied. But suppose all 3 files (old file, new file, diff file) exist on the source server and none of the 3 exists on the local machine. A possible expectation is that httpcopy would download the old file and the diff file, then build the new file. This is not what will happen.
  7. It is essential that both the difference file and the new file exist on the source server. If the new file does not exist on the source server, the difference file will be applied (provided that the old file exists) and the new file will be created on the local machine, but it will be considered an orphan and will be deleted (unless "delete orphan files" is disabled).

Building the difference file

Use the hcmkdiff (httpcopy make diff) command line program to create difference files. The syntax is as explained in the help info:

hcmkdiff V1.0 - copyright LANDesk Software

This program builds a difference file for httpcopy. Given an old version and a
new version of a file, it builds a difference file for use by httpcopy.
Usage: to build a diff file:

hcmkdiff [/q] /old=oldfile /new=newfile /df=diff-file

The diff file must have a .df extension. The optional /q parameter suppresses
progress info.

To display info contained in an existing diff-file:

hcmkdiff /i /df=diff-file

DLLs required

Two DLLs are delivered with this version of HttpCopy: libdf.dll and libdfp.dll. The former is required by hcmkdiff. The latter is required by HttpCopy when byte-level differencing is desired. HttpCopy will work without the DLL, but it will not apply diff files. A warning message will appear in the log if diff files were found but the DLL could not be loaded.


Known Issues

Error 12029, A connection with the server could not be established

This problem has been seen when the number of files to be synchronised exceeds 3500 or so. The problem is believed to be caused by a bug in wininet, which opens a new TCP connection for each HTTP HEAD request, even though the http "keep-alive" feature is enabled. During the discovery phase, HttpCopy performs a HEAD request on each of the files on the source server. If there are enough files, and if the network is fast enough, the machine that is running HttpCopy may accumulate enough TCP connections in TIME_WAIT state that it is unable to make any further connections to the same server/port (it will still be able to connect to other machines, and it will still respond to incoming connections). The result will be HttpCopy failing with status 12029. The http keep-alive feature is designed to avoid this problem by reusing the same TCP connection rather than opening a new one for each http request, but wininet does not seem to support keep-alive for HEAD requests.

A connection in TIME_WAIT state is a connection that is already terminated, but not quite "forgotten". When a TCP connection is closed, the computer that initiated the connection keeps the connection in the TIME_WAIT state for "a while", just in case any packets come in late and get confused with a new connection.

How to recognise the problem:

  1. Do not assume you have this problem unless you have an Error 12029.
  2. Do not assume you have this problem unless you have several thousand files.
  3. If you suspect that you might have this problem, open a command prompt and type netstat -np tcp on the machine running httpcopy immediately after receiving the error 12029. If netstat returns thousands of lines with status TIME_WAIT, you have the problem.

If you have confirmed that you have the problem, there are two workarounds:

  1. On the machine running httpcopy (not on the server), reduce the system's TIME_WAIT timeout. The default TIME_WAIT timeout is 240 seconds (4 minutes). Create a DWORD named TcpTimedWaitDelay with a value such as 30 under HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters.
  2. Use the AvoidHead parameter. This is a parameter that can be set in the INI file or on the command line. There is no GUI support to set it. It causes HttpCopy to "avoid" HTTP HEAD requests where possible, using GET requests with a 1 byte range instead. This causes a slight server side overhead (the server must now open the file rather than report some its attributes), but substantially reduces the number of TCP connections. The keep-alive feature works for GET requests.


Change History

Changes in V1.2 (13 July 2001)

  1. Changed command line syntax. Command line parameters and INI file parameters are now identical.
  2. HttpCopy now sets the file’s last modification time.
  3. V1.1 could copy individual files but not a directory. Another program (ftpsync), based on FTP, was needed to replicate a directory. This version can replicate a directory using only HTTP. Directory browsing must be enabled at the web server.
  4. In addition to the maxTime parameter, there is a StopTime parameter, e.g. to stop at 8 AM, specify /stop=08:00:00.
  5. Username and password.

Changes in V1.3 (19 Nov 2001)

  1. New /delete=1 parameter to delete files that no longer exist in source URL.

Changes in V1.4 (8 Jan 2003)

  1. New /silent=1 parameter.

Changes in V2.0 (16 July 2004)

  1. GUI to configure parameters.
  2. New /unattended=1 parameter. Previous versions were unattended by default and failed if parameters were missing. To allow V2.0 to be started with no command parameters and no INI file, it was necessary to introduce an incompatibility with previous versions. V2.0 will no longer run in unattended mode by default.
  3. Username and password are stored encrypted in the INI file. To maintain compatibility with old versions, unencrypted usernames and passwords are still understood, however.
  4. Better cleanup of work directory.
  5. Bug fix: if the delete parameter was specified and a job was interrupted, any files that httpcopy had not yet visited would be deleted from DownloadDir.
  6. Some Apache based web servers did not understand the range header. As a result, byte level restart would not work.

Changes in V2.0.1 (23 July 2004)

  1. Bug fix. V2.0 ignored the Files list. It was not possible to download individual files.

Changes in V2.1 (24 Feb 2005)

  1. New feature: "Keep Directory Structure" setting (default true). Previously, the directory structure was always kept. Uncheck this parameter if you want to "flatten" the directory structure.

Changes in V3.0 (10 May 2006)

  1. Bug fix. Copying a directory from an IIS 6 server (Win2003) would fail with a status 301 unless a trailing slash was appended to the url.
  2. New feature: exclude list. See example here. See also here.
  3. HttpCopy now calculates the number of files and bytes to download before starting. It will then report progress against those numbers. Two additional progress bars show the overall progress. See this screenshot.
  4. Exit code. Previous versions of HttpCopy were not entirely consistent in terms of the exit code (errorlevel) they produced. More details about the exit code here.
  5. HttpCopy generates a file called setstat.bat in the WorkDir folder. If you are running HttpCopy from a batch file, CALL setstat.bat from your batch file to set a number of environment variables that provide further detail of the HttpCopy success or failure status. More details here.
  6. Log file. Previous versions of HttpCopy kept using the same log file between jobs until it reached 1 MB, after which it was saved as httpcopy.log.old. If you prefer a new log file to be created for each job, use the new log parameter (/log=n) to indicate how many generations of the log file are to be kept (max 999). Old generations are named HttpCopyLognnn.csv. If you prefer the old behaviour (grow to 1 MB), set /log=0. The log file is now in csv format. More details here.
  7. Previous versions of httpcopy might hang when the connection to the server was lost.

Changes in V3.1 (11 June 2006)

  1. Bug fix. Previous versions of httpcopy had trouble with files bigger than 2 GB. This version has been tested with files up to 6 GB.

Changes in V3.2 (04 July 2006)

  1. The ini file parameter (/ini=<filename> can be a URL path. This allows the ini files to be kept in one place at a central location. Example: call each ini file <machine_name>.ini, use a batch file to run httpcopy and specify a /ini parameter similar to /ini=http://myserver/packages/ini/%computername%.ini.
  2. Profiles. If different package servers need to receive different subsets of the packages, this feature allows you to maintain a central list list of which servers receive which packages. More details here.

Changes in V3.3 (27 July 2006)

  1. In previous versions, there were 2 limitations to the delete orphan files feature. This feature causes orphan files (files that no longer exist on the source or that do not satisfy the replication rules) to be deleted. First, the delete parameter would only apply if a single directory was being replicated. Second, removing a machine from a profile did not cause the files that came with that profile to be deleted.

    These limitations have been removed. Any files in both the download folder and in the work folder that do no satisfy the replication rules will be deleted.

    Caution: you may have existing httpcopy.ini files with a /delete parameter that would have been ignored by previous versions of httpcopy, but that is no longer ignored by this version. Make sure you do not lose files inadvertently.
  2. Bug fix. The code was always meant to retry after a "connection reset" status. But previous versions didn't.
  3. The GUI for the profiles was changed. You can now test the profile settings. The test button allows you to see what profiles apply to the local machine. You can also see the list of files and folders to be replicated as a result profile memberships. See here.

Changes in V3.3.1 (19 Dec 2006)

  1. Bug fix. The program would crash when no httpcopy.ini file existed.
  2. Bug fix. When the login name exceeded 16 characters, it would be truncated.
  3. Bug fix. Folder names on the exclude list would only work if they had a trailing slash.
  4. New feature: "do not run before". This allows a time window to be set within which httpcopy is allowed to run. If started outside of the allowed window, httpcopy will exit with an error code of 9997. For more details, see Specifying a time window.

Changes in V3.4 (2 Feb 2007)

  1. Bug fix. When the server name could not be resolved, all files would be deleted.
  2. Previous versions had problems with filenames that had accented characters in them. This version is Unicode based and no longer has any such problems. It not only supports accented characters in filenames, but also Japanese, Chinese, Cyrillic etc. Filenames will always look right in the GUI and on the hard drive, but you may need a new setting to make sure that filenames are logged correctly.
  3. Better retry handling. This version will survive certain error conditions that previous versions didn't.
  4. Changes to the UI. The progress info and the numbers hopefully make more sense than before.
  5. See known issues.

Changes in V3.4.1 (16 Feb 2007)

  1. Bug fix. V3.4 had broken the option to flatten the directory structure. The program would loop in the discovery phase.
  2. Bug fix. The GUI cleared the file list and exclude list in the INI file under certain conditions.

Changes in V3.4.2 (04 June 2007)

  1. This version supports https. The URL parameters that you specify in the GUI or in the INI file can have an https prefix.

Changes in V3.4.3 (03 July 2007)

  1. Bug fix. When there was a Files list, there could be an error "The system cannot find the path specified (3)".

Changes in V3.4.4 (28 Nov 2007)

  1. Bug fix. Httpcopy would sometimes misinterpret responses from web servers that do not include a content-length header. The result would be a temp.html file in the work directory that keeps growing forever.

Changes in V3.4.5 (07 Dec 2007)

  1. Bug fix. Httpcopy would misinterpret responses from web servers that include a particular type of "content-type" header. The result would be an error message "unable to create <workdir>" at the start of the session.
  2. Accented characters. The UTF-8 encoding of URLs, introduced in V3.4 and recommended by W3C (see http://www.w3.org/International/O-URL-code.html), is not understood by certain web servers. The result would be http 404 status ("file not found") for files whose filename includes accented characters. This version tries to adapt dynamically to this behaviour. If the source server encodes its directory pages in ansi, httpcopy will respond in ansi rather than UTF-8.

Changes in V3.4.6 (24 Dec 2007)

  1. Bug fix. Some web servers include embellishments such as sort headers in their directory listings (click on one link to sort by name, click on another link to sort by size etc). There is no foolproof way to distinguish these links from links to files and directories. This version accommodates some web servers by ignoring links with question marks in them (question mark is not a valid character in a filename).

Changes in V3.5 (21 May 2008)

  1. New feature: unpack-on-arrival. This is useful when replicating large numbers of files. If you find that httpcopy takes an unacceptable length of time because of the large number of files, consider zipping your files. You can use the feature described here to unzip them automatically on arrival.

Changes in V3.5.1 (04 June 2008)

Minor changes to the unpack-on-arrival feature:
  1. In V3.5, unpack-on-arrival would work when synchronising folders but not when synchronising individual files. In this version, the feature also works for individual files.
  2. The temporary bat files (filename_unpk.bat and filename_del.bat) are now deleted after use.
  3. The log files (filename_unpk.log and filename_del.log) are now created in workDir rather than downloadDir.
  4. New conditional parameter. Modifies the meaning of the delete parameter. If conditional=1, the archive file will not be deleted, and no stub file will be created, if the filename_unpk.bat file returned a non-zero exit code (i.e. if something went wrong with the unpacking).

Changes in V3.5.2 (04 July 2008)

  1. Bug fix. V3.4.1 (yes, 3.4.1) had introduced a problem whereby the "bytes remaining" count in the end-of-session message was always zero.

Changes in V3.6 (04 Dec 2008)

  1. New: byte level differencing.
  2. More robust retry handling. Previous versions made 1 retry per file, and only on errors believed to be recoverable. This version improves error handling in 3 respects: -1- the number of retries is configurable (default 3) -2- After a successful recovery from an error (i.e. at least some data was received after retries), attempts will be made to recover from further errors on the same file (previous versions would not retry on subsequent errors) -3- there is no attempt to judge whether or not a particular error code is recoverable.
  3. Changes were made to unpack-on-arrival. In previous versions, the feature worked well if the original archive file was deleted after unpacking (delete=1 in filename_unpk.ini). But with delete=0, there was a problem in that the unpack commands would be run again in every subsequent session of httpcopy. This version changes things as follows: -1- a stub file is created irrespective of the delete parameter (previously, a stub file was created only if delete=1) -2- before running the unpack commands, httpcopy will check if the the file size & date stored in the stub file still match the archive's file size & date. If they still match, httpcopy decides that the unpack commands have already been run for this file and doesn't run them again.
  4. Httpcopy may be faster when copying large numbers of files across a slow link. As before, httpcopy performs a discovery phase followed by a copy phase. The purpose of the discovery phase is to be able to generate progress info and end-of-session statistics. What's new is that httpcopy now caches the results of the discovery phase. Previous versions discovered the files to be copied again during the copy phase.

Changes in V3.6.1 (12 Dec 2008)

  1. Previous versions would sometimes delete a partial file unnecessarily. The reason has to do with an identifier known as etag that a web server associates with every web resource. The purpose of the etag is to avoid downloading a file again if the same file has been downloaded before and it hasn't changed. The client receives an etag with every files that it downloads and it can later use the etag to verify with the server whether or not the file needs to be downloaded again. So far so good. The only problem with this is that in the case of IIS, a file's etag changes when the IIS server is reset or the system is rebooted. This version works around the problem by also checking a partial file's date & size. If the etag doesn't match but its size & time stamp do, the partial file will be considered valid.

Changes in V3.6.2 (07 January 2009)

  1. Bug fix. V3.6 had broken the option to flatten the directory structure. The symptom would be http status 404.