It looks like "hashbreaker" and "CodingCrypto" are both running through nearly the exact same word sets. Either they are from to different people using the exact same algorithm or (what I suspect is more likely) they have created multiple twitter accounts to get around the submission limit.
I didn't scrutinize the sha-1 algorithm, but is it possible to optimize its computation (e.g., precompute intermediates) when using a small number of repeated words as input? This might explain why a group of people all arrived at similar solutions, since it only matters how quickly you can guess if all guesses are pretty much equal.
I lucked out and got a solution at 35. Coded something up in python and only got about 85k hashes/sec on one core. The CUDA program, though, was about 100x faster on my low-end Nvidia 8400 GS.
Yes: very roughly, you could precompute the internal state after 'abcdefg' -- essentially checkpointing the algorithm -- and then compute SHA1('abcdefgX'), SHA1('abcdefgY'), etc. more easily than other strings. (In fact, you'd want your checkpoint aligned at some multiple -- maybe 512 bits? -- but that's the general idea.)
I figured the 'final 5 arbitrary characters' was a nod to this approach. So the best strategy I could think of -- and I didn't try coding anything up for the contest -- was to (1) guess the word dictionary; (2) precompute a whole bunch of checkpointed-except-for-last-five-characters function states; (3) at the moment the real dictionary and target was announced, try many different last-five-character extensions against any valid precomputed states.
Disregarding the last 5 characters, the 'bugs blank' and 'cows dirty' strings are exactly 64 characters (512 bits) long, as are many of the other leading entries. Savvy teams are checkpointing the hash at the 64 character point, then rerunning it from there with alternate endings.
The function as applied to a stream of bytes has a certain amount of internal state. You could at any moment take a snapshot of the entire state, and from that point, feed it different suffixes of the same previously-fed data. In this way, you calculate many final hashes without always repeating the redundant part (the shared prefix).
For a concrete example, consider let us say you wanted to take the SHA1 of the input that is 1MB of 'a' followed by an 'a', and then 1MB of 'a' followed by a 'b', etc.
You don't have to run the SHA1 function over 'a...aa', then reset from scratch and run it over 'a...ab'. You can run it over 'a...a' without then feeding it the final byte or performing its final steps, snapshot its state (which is only some small multiple of its final output size), and then quickly calculate 'a...aa', 'a...ab', 'a...ac', etc. without redundantly feeding the identical 1MB prefix in each time.
From the hashlib documentation, yes, it appears so.
Note, though, SHA1 works in 512-bit (64-byte) blocks, so 'snapshotting' after feeding it (for example) 63 bytes hasn't saved any hard work -- the routine was only buffering those 63 waiting for byte #64 (or the digest-finish operation) to do the real crunching.
And, the tiny EngineYard strings don't offer a giant opportunity for precalculation savings. The usual pattern for digesting is:
1 x new()
N x update()
1 x digest()
The copy() lets you snapshot state after M update()s, M<N, to speed future calculations. Obviously if M is a lot -- there is a large shared prefix -- there could be big savings. But with the EngineYard strings, M should never be more than 1 block/update(). (If it is, your 10-word preamble was wastefully long.) So the potential savings are capped relative to the cost of the copy() and the (unavoidable) second update() and digest(). If the second update() and final digest() are roughly as costly as the first update() and original new(), and if the copy() is nearly costless, I suppose the speedup could approach 2x.
But also: I suspect an optimized C SHA1 operation on tiny amounts of data is going to scream compared to your Python loop and calling overheads -- so any improvement from your copy() reuse may be noise compared to what's taking up most of your loop time.
