I am aware that you may not wish to hear what follows, but here goes:


Firstly, perhaps 'deciphering' is the wrong term since the strength of the algorithm is not an issue at all. Anyone with moderate ASM experience and a disassembler could work out quite quickly where your decryption loop is and from where in the code it draws its key. Thats if they cared to waste any time, there are faster solutions for those that just want a timely answer without the requirement to understand every step of the code.

For example, a simpler method would be to execute the binary in a secure debugging environment (Say, on a spare machine or a VM) One could then simply step through it and pause once the code has run its decryption and read the plaintext right off the screen. Thats the no-brainer solution.

So, the point that should be understood here is that nobody need waste their time breaking or even looking at your cipher as the binary will decrypt it for us during the course of its run.

Please don't feel I am criticising you or your code here. I just feel that this is a point you should be aware of.


Personaly, if I were investigating the code, I would look at all the calls it made during an unhindered execution and find the more interesting ones (Registry writes and certain other DLL calls) this would tell me if the code was running batches, spawning an embedded exe (As in an exebinder), setting keys, injecting code into other processes, calling WSAStartup, etc. and generaly give me an idea of what TYPE of things the code is the code is doing. Once I've identified and shortlisted the calls I wish to inspect more closely I'd then run it again under softice with breakpoints on the interesting calls and read the parameters and, optionaly, return values from the stack and looking at any buffers passed by reference. This approach would generaly give me everything I needed to know quickly whilst minimising he chances I may miss something important.


This is all standard procedure and is well within the capabilities of any competent incident investigator, coder or forensic analyst.

So, in answer to your original question 'could one decrypt the payload' my answer is simply that if it was specificaly requested it could be done trivialy with a little time, but this is seldom of interest and therefore not usualy done. It is sufficient to understand what the code is actualy doing rather than how it stores, encrypts or packs itself and its data. Once the former is understood, the latter becomes an irrelevance.


Anyway, best of luck with your rootkit. Its always nice to see someone actualy getting off their ass and challenging themselves. Well done

SG