Poor Man's Bin-Diff

Sometimes you happen upon a binary file, nearly identical to another, that requires identification of very minor differences. Usually the first n bytes are identical, than deviations or complete differences exist beyond that point. If you don't have access to a tool like bindiff, or the file format is not applicable to bindiff (like the 2 ISO files I had to compare today, or anything other than an executable), there is a very easy way to identify the start and potentially all differences between the files. With some gnu kung-fu we'll use xxd (the hex dump utility) and diff (the programmer's difference identification tool) to locate the changes and potentially the start of changes. 

First, use xxd to dump the binary file contents to a ASCII hex representation:

# xxd File1.exe > File1_dump 
# xxd File2.exe > File2_dump

Standard PE/Executable file, dumped to hex with xxd.

As you can see above, xxd makes for a very convenient binary file viewer when looking for plain text meta-data, basic data structures and other potentially interesting items. We've essentially converted the binary files into an ASCII based file.

Some useful notes about this method: if there's an offset from the start of similarity, you'll need to use the  following options when creating your xxd dump:

-seek bytes_to_skip Skip to the identical starting point of the two files.
-ps  Create the output without the byte offset column and without the ASCII representation columns.

Next, use diff to view the areas of difference between these files.

# diff File1_dump File2_dump

We can clearly see changes (an IP address) starting at file offset 0x4f70

Using the switch --suppress-common-lines will reduce the content you have to review before identifying areas of interest.

In this manufactured example, we see the difference between the two executable files is an IP address.

References:
http://linux.die.net/man/1/diff
http://linux.die.net/man/1/xxd

Digital Evidence Formats

Folks, this is a little off our normal topic of "Intrusions and Malware Analysis," but I think it's entirely relevant.

The Plea
Over the last 6 years that I've been involved with computer forensics/computer security I regularly reflect on the lack of open standards for handling digital evidence. Can we please, as a community, make a grassroots decision to standardize our imaging on a common format?

The Problem
I, personally, have experienced problems with proprietary image formats that are password protected or have incompatible format versions for the analysis software that I am using. When these problems arise it often costs a ridiculous amount of time and potentially money to "fix" the problem. The problem is proprietary evidence container formats. There, I've said it.

Today
Currently, if I had to choose, I would say that "dd", or raw bitstream, would be the best format to standardize on. Split, complete, whatever. Raw format allows for the most flexibility in performing forensics examinations. Every forensics suite (that I know of) supports the format, there are never issues or concerns regarding format or software version, we're able to split the image to support different file systems if need be... In my opinion, it's the best option available to us.

Tomorrow
Now, you may be sitting there thinking, "But raw doesn't support compression" or "But raw doesn't allow me to password protect the evidence." You're right on both counts. I don't think that Raw format is the end-all fix to this problem, but let's start with the raw format and see how we can meet various requirements of evidence containers while staying in the realm of open standards.

Compression
If there is one thing that Open Source/Open Standards software has given us it's compression algorithms. One might argue that there are too many compression formats on open platforms and I would agree. We need to identify a compression algorithm that would bring reasonable compression and ease of split file output for our increasingly large image files.

Protection
The selected compression standard must also allow for password protection. Some evidence may not be able to be processed with a chain of custody at every point and should be protected from illicit viewing and potential tampering.

Other Details
I would like to think of the file format as more of a wrapper than the actual content itself. In addition to handling some level of compression and protection, I'd like to be able to insert a 'hand receipt' into the package. Generate a hashlog of the drive and insert into the overall package. Think about it more like a portable filesystem than a file itself:

\ImageFileName.fez
- PackageContents.xml (Contains MD5 of image, information about origin, etc.)
- Hashlog.txt
- ImageFile.raw.dd
This is very similar to how the Open Office Document format works, it makes sense and could make working across company/organization boundaries much more efficient.

Questions
Did I miss any major features that you would want to see? Let me know in the comments!

RAID Tool Update

I have released a FUSE file system that can put together an asymmetrical RAID 5 with any parity level and any iteration level. I don't have any controllers to test the thing on other than the Compaq one it was developed with. Remember this tool is a proof of concept, use it accordingly. I am working on the update to this tool that will use one single fuse mount command and plugins to add "files" at will. Reassembled RAID images will be the first of these files and it will allow for compound RAID with only one fuse instance running.

If your interested in the tool, check it out at http://raidfs.sourceforge.net/ or download at http://sourceforge.net/search/?type_of_search=soft&words=raidfs

Jason

Powered by Qumana