I'm trying to reverse an encryption scheme, but I seem to have fallen into a pit when it comes to reversal using algebra.
The encryption scheme is as follows for a single char (using registers and constants):
encrypted_char= (original_char XOR dl) + al where: eax = eax.previous * c1 +c2 edx = (eax >> c3) eax.0 is a known seeded constant.
I want to solve this equation algebraically for original_char, but I'm running into a few problems, namely with order of operations for getting original char on it's own. Thinking about wraparound for negative numbers is also giving me a headache.
If anyone had any pointers for how to solve for the original_char, it would be appreciated. My first thoughts are to just subtract al and then xor with dl, but I'm starting to feel confused at this point.
I played with a toy example before posting and my answer is as follows:
bitwise xor has the same precedence as multiplication, I just flip it over. I already knew that XOR was the inverse of XOR, but I thought I should state it here.
The resulting formula is as follows:
(encrypted_char - al) XOR dl = al
What goes into the larger registers doesn't need to be toyed with to arrive at the correct solution.
I will solve the wraparound using the modulus operation with the correct size for my variables.
Using the above methods I was able to reverse the code.