Generate Window

The digest calculation functionality is provided by the docking window named Generate. For the default configuration of the Digester software, the button for this docking window is located on the right edge of the main window. Here you can see five tabs Formats, Options, OpenPGP, XML, and Refresh to control the calculation process. The generate window provides support for the recursive processing of directories including subdirectories.

Please note that the generate functionality is not available in the Basic Edition.

     

The requirements for the use of digests and signatures may be very different: for FTP servers often SHA and signature files are used, while the Digester XML format is appropriate for archiving backups. In order to support arbitrary environments appropriately, templates may be defined. You can create a new template using the button with plus symbol on top of the generate window. You will be prompted to specify a unique template name. The current settings of the generate window are used to create the new template. You may adjust the settings at any time: please select the desired template and adjust settings as desired. The templates are automatically saved if the program gets terminated and restored when restarted. Templates can be deleted with the red X button after user commit and be renamed with the button showing the pen symbol. The templates can be displayed in alphabetical order or in the order the items were created.

Tab Formats

As the figure shows, the Formats tab can be used to configure the digest algorithms and their associated file extensions. By selecting the various algorithms, you configure the Digester software to use these algorithms for calculations. You can specify any combination of algorithms but at least one algorithm needs to be selected here. If you like to select all algorithms in one step, please activate the context menu (right mouse button) directly on the algorithm checkboxes.

In the fields to the right of the algorithms you can enter the file extensions to be used. The fields are already pre-configured with widely used extensions. If you calculate a hash with the MD5 algorithm for example, the software creates a digest file with the name of the original file plus the extension specified here. The configured file extensions are not only used by the generate window but for all calculation variants including the context menu (right mouse button) and the generate wizard.

Please note, that changes to the file extensions within the generate window do not affect the file associations of the installer variants of the software. The file associations to open files on the operating system level are the same as the default settings of the generate window and can be adjusted only with the appropriate programs (e.g. Windows Explorer).

The option Hex digest format configures the case of the calculated hexadecimal digests. It defaults to "Lower-case", the values may alternatively be generated with upper-case letters. This option does not influence OpenPGP signatures.

Tab Options

This tab provides several additional options to control the created file formats:

If files are created (Create files), then you can precisely control how this will be done. You can choose between the supported file formats GNU format, BSD format, the OpenPGP signatures, and the Digester XML format. You can use either the GNU or BSD format since these formats are based on the same naming scheme. The XML format can always be configured in addition to GNU/BSD since this file format produces unique filenames due to a separate file extension which does not represent the digest algorithm. The generate window helps you to find an appropriate combination here by rejecting unallowed option combinations.

If you like GNU files being created then you can specify whether GNU files shall be created on a per-target basis (per original file) or if multiple digests for a whole directory shall be saved into a single GNU file (per directory). You can also use both options in parallel. While for the GNU files per target the name of the original files plus a file extension for each algorithm is used, the GNU files per directory are named based on the setting in the Summary file input field (defaults to summary). While the GNU files per target only contain one digest for the used algorithm, the GNU files per directory possibly contain multiple digest values for the algorithm. The digests are stored per line according to the GNU format. A typical GNU file per target for the franz.txt original file and the MD5 algorithm is named franz.txt.md5 for example, a typical GNU file per directory is named summary.md5 based on the default settings.

The settings for BSD files are the same as for GNU files, only that the format of the generated files is slightly different. OpenPGP signature files are limited to the per-target creation. If multiple signatures e.g. for multiple keys shall be stored together please use the Digester XML format.

If you like to create Digester XML files which may contain any number of final digest values plus intermediate digests calculated by arbitrary algorithms, please select the XML file per target or XML file per directory option or both. While for the XML files per target the name of the original files plus a configurable file extension is used (see the XML extension input field on the XML tab), the XML files per directory are named based on the setting in the illustrated input field (defaults to summary). While the XML files per target only contain one digest for each original file and algorithm, the XML files per directory possibly contain multiple digest values for each of the original files and algorithms. A typical XML file per target for the franz.txt original file is named franz.txt.digest for example, a typical XML file per directory is named summary.digest based on the default settings.

By setting the Overwrite existing files option, the software is instructed to overwrite existing files, otherwise the calculated digests will not be saved. The Apply target file timestamp option should always be selected if the timestamp of the processed original file should be set for the created message digest file (only possible when files are created per target). If digests are calculated for large amounts of data such as backups, then the software can be configured not to produce or display any results by deselecting the Show results option.

Overwritten digest/signature files cannot be restored programmatically, so please backup your data first. A dialog is displayed if this option is selected.

 

The Miscellaneous section of this tab provides additional generate options:

In order to generate only certain files for certain original files you may enter a regular expression in the Include input field. If the field is empty then all recognized original files will be processed by default. If certain original files shall not be processed then you can specify a regular expression with the Ignore input field. These options are particularly useful, if directories or entire directory trees including subdirectories are processed. You can activate recursive scanning of subdirectories by selecting the Include subdirectories option. In order to get detailed messages on the generation process please select the Verbose messages option.

Tab OpenPGP

In addition to message digests the Digester software is capable of calculating OpenPGP signatures and signature files. Signature files may be created either int the ASCII signature file format (default) or in the Binary signature file format. Binary files have the advantage that a little less space in required for storage. Within Digester XML files OpenPGP signatures are always stored in the ASCII format, since binary data cannot be stored here directly. In contrast to the other file formats described here each signature file may contain only one single signature so that signature files for whole directories is not possible here.

     

Typical OpenPGP extensions for signature files are .asc and .sig, but the Digester software may be configured to create arbitrary file extensions. In order to label created signature files one can specify an arbitrary single-line text to be stored in the signature file headers (OpenPGP version). For instance, you can add your company's or your organization's name here.

If you already have created secret OpenPGP keys using the key management they will be displayed by the list at the bottom. In addition to the optional URL (network address), the Name column shows the key name. The eMail column shows the e-mail address of the key owner and the hexadecimal key identifier is displayed by the Key Id column. In order to generate signatures according to the OpenPGP standard please select the desired key entry in the list.  If a password was deposited for the key this password will be automatically used for signing, the user is prompted by a dialog otherwise just before the signing. The key URL may be configured (optional) to store the key address within Digester XML files. You can set a key URL by clicking the button showing the pen symbol, a key may be deleted with the red X button. Only valid URLs should be entered here since the information is used to download public keys. On top of the key list you can see the main features of the key management whcih may be called conveniently directly from here.

The OpenPGP functions can be used only if the unlimited key strength has been activated. You can find detailed pieces of information about the legal background for the deployment of cryptography on our website. With the Check Key Strength functionality you can determine the currently supported key strength of your Java environment.

Tab XML

This tab provides settings in connection with the Digester XML format. Besides the above mentioned input field to specify an XML file extension, you can set a comment for XML files (optional). This is stored in the header of each Digester XML file and can for example be used to transport copyright information or notes for the FTP server. The default format of digests in Digester XML files is the hexadecimal format, you may also choose Base64 encoding.

With the Include absolute paths option, all digest values are stored along with the absolute file reference for the computer. Under Windows for example this could be the path C:\Temp\franz.txt. Even after having copied such XML files to another drive of the same computer, the software can correctly resolve such paths and properly find the files associated with the digests. However, Digester XML files get bigger when absolute paths are added.

The Include URLs input field may contain a fixed network address or a SSI pattern. Here you can specify a static URL pattern e.g. for SSI (Server Side Includes) which can be processed by web servers like the Apache HTTP server. Such URLs can be translated dynamically into the appropriate URLs for the original files on the server. If Digester XML files are being downloaded with the software, then the referenced files can be identified using the URL entries to be downloaded, checked, and processed in other ways.

If Digester XML files are being created and OpenPGP keys have been specified to calcuate signatures then the signature values may be added to the Digester XML files (Include OpenPGP signatures with). Signatures are represented by separate XML elements which are stored on the level together with digests (see DTD Digester XML). The associated flags Key ID, Key name, Key eMail, and Key URL may be configured to include the key identifier, key name, key owner's eMail addresse, and a static network address to download the public key (all optional).

A special feature of the Digester software is the capability to create interim results - the so-called intermediate digests - in addition to the final digest values. While GNU/BSD files can only contain one single digest per original file, the use of Digester XML files allows to store an arbitrary number of intermediate digest values. Message digest files based on the GNU/BSD format may only contain the final digest value which is created when all bytes of the original files have been processed. The Digester software can now be configured to create interim results at defined positions which are then stored into the Digester XML files along with a position indication. The use of intermediate digests allows to quickly detect deviations of examinee files and original files since these often differ already in the first data blocks. In such cases the calculated intermediate digests for examinee files are different from the intermediate digests found in the digest files. Especially when checking files on remote servers, this effect may lead to significant time savings since less data needs to be transferred and processed. 

If you do not like intermediate digests be produced then select the No intermediates option. Select Linear mode to have intermediate digests be created at fixed intervals. The interval can be entered into the Create every input field where the default value is 10M, causing an intermediate digest being created every 10 megabytes. In order to limit the size of the Digester XML files even for large original files, you can set the total number of intermediate digests per original file to a maximum value (see the first Max digests input field).

The option Exponential mode allows the creation of intermediate digests beginning at a certain position (see the Starting at input field, default is 8K = 8 kilobytes). The next intermediate digests will be produced after the next 16 kilobytes, 32 kilobytes, etc. While in this mode a relatively large number of intermediate digests is produced for the first data blocks of the original files and less at the end. This allows to quickly detect deviations of examinee files and the original files since these often differ already in the first kilobytes of data. In order to limit thesize of the Digester XML files even for large original files, you can set the total number of intermediate digests per original file to a maximum (see the second Max digests input field).

Tab Refresh

In generate mode all message digests are calculated and files are created always. If only a few files on an FTP server for example have been added or modified this kind of behavior is not appropriate since too many calculations must be done unnecessarily. Here the refresh function of the Digester software can help. On the Refresh tab you can configure under which conditions existing digest files are recognized as being still correctly so that the calculations can be skipped.

For the refreshing, all detected digest/signature files are being used if the Analyze all formats option is selected. If the Analyze write formats only is selected instead then only those formats are being analyzed which are configured to be written and deemed relevant for the refreshing.

If all formats are being analyzed then it is possible to switch from GNU files over to XML files for example and let the software delete the previous GNU files programmatically. This way only those files are kept which are deemed relevant for the synchronization (write formats). Activate the Remove other format files option to achieve this. By setting the Remove artifact files option so-called artifact files may be automatically removed. Artifacts are files with digests/signatures where no original files exist at all. This way useless files can be cleared automatically.

The section titled GNU / BSD / OpenPGP contains options to control the program behavior for such files. Such files are skipped (not calculated again) if digests match and if the file timestamps match (GNU/BSD file timestamps match). OpenPGP files are skipped if signatures match and if the file timestamps match (OpenPGP file timestamps match).

The section titled Digester XML contains options to control the program behavior for Digester XML files. Such files are skipped (not calculated again) if digests/signatures match and if the file timestamps match (XML file timestamps match). Such files are skipped if digests/signatures match and if the file sizes within the XML entries match (XML entry file sizes match). Such files are skipped if digests/signatures match and if the file timestamps within the XML entries match (XML entry timestamps match). Last but not least missing intermediates may be ignored with the option Ignore missing intermediates. Maybe intermediates have not been generated the last time so this can be compensated here as long as the overall digests match for the files.

Please keep in mind that the options XML entry file sizes match, XML entry timestamps match, and Ignore missing intermediates refer to the element contents within the XML files, not the XML files themselves. The XML files especially contain information about the original files at the time the XML files were created (sizes, timestamps) which may be evaluated here.

If file timestamps should be analyzed please select the option Apply target file timestamp as well (see options tab). The timestamps of the original files are then applied to the generated files so that this information can be used for the refresh functionality.

Start Generating

Once all the settings within the generate window are as desired, the calculation can be started by clicking on the Generate button at the bottom of the window. If the settings make sense and appropriate files and/or directories have been selected within the file system window, this button is enabled. The checking process is monitored by a special dialog where the current state is shown. Upon completion of the calculations this progress dialog shows some statistics for the operation and the generate results. Besides the time consumption of the calculations (top left corner) the dialog shows the remaining time (ETA = Estimated Time Access).

Please note, that it is possible to select directories within the file system window to be processed recursively including subdirectories, while this is not possible if you start generating from the context menu. The Refresh button is disabled if the option No files, just calculate is selected. The refreshing functionality is executed the same way as the generating process.

When all digests have been calculated and the progress dialog was closed, the results are displayed by the results window which is automatically opened. The generate results are divided into two tabs Files and File Digests or URLs and URL Digests, if the generating has been started from the network file system window.

Command-line Arguments

In order to make generating from the command-line with digestercli as easy as possible the necessary arguments may be assembled by the Digester software. Please select the files/directories to be processed within the file system window in the first step. Then configure the generate/refresh options with the generate window as if the process would be started directly from here. By clicking the button with the console symbol a dialog is displayed showing the corresponding digestercli call. Either the generate or refresh mode has to be selected first if the button is clicked. This code can be easily copied into the clipboard to be used for shell or batch scripts. As you can see programming is not required here.

Ant Build Scripts

Like the command-line arguments also XML fragments for Ant build scripts may be generated using the button with the ant symbol. This code can be used for own shell or batch scripts. Please select the files/directories to be processed within the file system window in the first step. Then configure the generate/refresh options with the generate window as if the process would be started directly from here. By clicking the button a dialog is displayed showing the corresponding XML code. Either the generate or refresh mode has to be selected first if the button is clicked. This code can be easily copied into the clipboard to be inserted into a text editor for example. As you can see programming is not required here.

Additional Links