MultiHash

MultiHash builds MD5 hashes for anything from a single file all the way up to an entire hard drive. Every individual file will have an MD5 hash generated, and then an overall hash is produced which can be used to compare directory trees to assure the entire tree is identical without you having to verify that each individual file’s hash value matches. These hash values are useful for verifying downloads from sites which provide hashes for their downloads. Typically, you will find these for ISO files, Linux distributions, and many applications. Hashes are also useful for verifying software escrow deposits, since they often require multiple identical submissions to verify the data is not corrupt. MultiHash makes all of this easy, saving a log file of the hashes that are calculated, along with providing an overall hash value

With Multihash the user is given the freedom to compute hash values using combinations of algorithms. Multihash offers the same core features as similar tools while adding many features not currently found elsewhere:

• Find-mode: Instead of hashing the command-line arguments, treat them as lists of files to be hashed.
• Match-mode: Compare the hashes of two files, or recursively compare two directory structures.
• Uses the OpenSSL library to support fast and efficeint hash value generation, optimized for multiple platforms.
• Compute hash values from different algorithms in parallel for maximum efficeincy.
• Support for md5sum, md5deep, and OpenSSL output-modes for drop-in compatability.

For a full list of features, their uses, and examples please see the man page.

Multihash is able to offer all this while maintaining performance comparable to other tools. When using recursive mode Multihash is able to match performance with md5deep. Using the OpenSSL library means Multihash has to pay a higher process creation cost than tools with in-core hash functions. This causes benchmarks involving many processes (such as the -exec find(1) option) to suffer. However, when inputs are aggregated using the xargs(1) utility Multihash will perform as well as other tools. Multihash also offers the find-mode alternative to cust the cost of process creation down. In the future, once the rest of the code has had a chance to air out, I hope to move to some more time saving techniques.

Multihash is a protocol for differentiating outputs from various well-established hash functions, addressing size + encoding considerations. It is useful to write applications that future-proof their use of hashes, and allow multiple hash functions to coexist.

Multihash is particularly important in systems which depend on cryptographically secure hash functions. Attacks may break the cryptographic properties of secure hash functions. These cryptographic breaks are particularly painful in large tool ecosystems, where tools may have made assumptions about hash values, such as function and digest size. Upgrading becomes a nightmare, as all tools which make those assumptions would have to be upgraded to use the new hash function and new hash digest length. Tools may face serious interoperability problems or error-prone special casing.

How many programs out there assume a git hash is a sha1 hash?

How many scripts assume the hash value digest is exactly 160 bits?

How many tools will break when these values change?

How many programs will fail silently when these values change?

This is precisely where Multihash shines. It was designed for upgrading.

When using Multihash, a system warns the consumers of its hash values that these may have to be upgraded in case of a break. Even though the system may still only use a single hash function at a time, the use of multihash makes it clear to applications that hash values may use different hash functions or be longer in the future. Tooling, applications, and scripts can avoid making assumptions about the length, and read it from the multihash value instead. This way, the vast majority of tooling – which may not do any checking of hashes – would not have to be upgraded at all. This vastly simplifies the upgrade process, avoiding the waste of hundreds or thousands of software engineering hours, deep frustrations, and high blood pressure.

Publisher Description

Generate hash values for files, directories, and entire drives. These hash values can be used to verify that two files or directory trees are identical, or to verify against published hash values when downloading. These hash values can be used to publish your own values so your customers can verify their downloads. Hash values can also be used to verify identical copies submitted for escrow or publication to CD. The user interface allows you to add files in a freeform manner, and it automatically begins processing the files. You can watch as it progresses from file to file. There is a progress bar for the entire file set, and for individual files. Two typical ways to use this software are to either compute a hash of a large archive file containing all of your individual files (like an installer or .iso image), or to calculate individual hash values for every file. To make it simpler to verify a large number of files, MultiHash can generate an overall hash for the file set, and save a data file showing the hash of the individual files.