runtime comparison of boost SHA1 hashing with md5sum/sha1sum

Question

I used Boost to compute SHA1 hash, and then compared the rum time with Linux command md5sum and sha1sum. The file size is 28G. The md5sum and sha1sum are both version 8.4.

This is how I used Boost to compute SHA1:

void Sha1_Key(const std::string& inputfilename_, uint32_t* inputfile_sha1)
{
    std::string sha1_key_string;
    uint32_t local_sha1[5];

    for (uint32_t i = 0; i < 5; ++i)
    {
        inputfile_sha1[i] = local_sha1[i] = 0;
    }

    std::ifstream ifs(inputfilename_,std::ios::binary);

    boost::uuids::detail::sha1 sha1;
    std::stringstream sha1_key;
    char buf[128*1024];

    clock_t begin = clock();
    while(ifs.good()) {
      ifs.read(buf,sizeof(buf));
      sha1.process_bytes(buf,ifs.gcount());
    }

    ifs.close();
    sha1.get_digest(local_sha1);
    for(std::size_t i=0; i<sizeof(local_sha1)/sizeof(local_sha1[0]); ++i) {
      sha1_key<<std::hex<<local_sha1[i];
      inputfile_sha1[i] = local_sha1[i];
    }
    sha1_key_string = sha1_key.str();
    clock_t end = clock();
    std::cout << "run time for Boost SHA1: " << double(end - begin)/CLOCKS_PER_SEC << " sec"; 
}

This is the run time comparison:

Boost SHA1: 170s
md5sum: 54.719s
sha1sum: 81.795s

The complexity of sha1sum is higher than md5sum, so sha1sum takes 50% more time. But the Boost SHA1 method run time is twice compared with the Linux sha1sum. Why?

Is there anything I can improve my code? Or any suggestion to other hash method to use?

Overlapping I/O and calculation. It's not very efficient to do it alternating in the same thread. — Ext3h, Apr 10 '18 at 18:56
@Ext3h This is the function used to calculate SHA1 so it should include all overlapping and overhead. Otherwise I just need to call md5sum in my code. — werk, Apr 10 '18 at 19:04
Perhaps sha1sum simply contains a better optimized SHA1 implementation than Boost. — President James K. Polk, Apr 10 '18 at 19:44
＋1 for actually getting the `read` loop right, with `gcount` (I think :)) — sehe, Apr 10 '18 at 20:58

score 4 · Accepted Answer · answered Apr 10 '18 at 20:56

boost::uuids::detail::sha1 sha1;

See that? ::detail::

That right there tells you you are abusing an undocumented implementation detail, that wasn't intended for your use.

That SHA implementation underpins UUID generation and wasn't designed to be fast for digesting large sequential streams.

That's all.

Use a proper sha1 implementation (libcrypto, cryptoc++ etc)

Your suggestion works. I tried with openssl, and the runtime reduced by 50% compared Linux sha1sum. — werk, Apr 12 '18 at 02:00

runtime comparison of boost SHA1 hashing with md5sum/sha1sum

1 Answers1