October 2013

Restoring file systems from iTunes backups

I’ve been reading lots of interesting information about iTunes and IOS, so I thought I would investigate, just what is in an iTunes backup.  Typically, on OS X, you can find your iTunes backup here, under the appropriate OS X user profile: /Library/Application Support/MobileSync/Backup When you look at the backup, it’s a giant list of 40 character hexadecimal file names.   After doing some quick reading on theiphonewiki.com, those file names are SHA-1 hashes.  Each of the files, is a backed-up file from the iPhone.   The problem of restoring the file system then is that we need to find the original file names from the hashes. Luckily, Apple provides an index.  There is a file called “Manifest.mbdb” which is a binary index of all the SHA-1 files.  There is a pretty good description of the format of that file here.   After reading the Manifest.mbdb into memory, we have enough information to generate all the SHA-1 hashes.  From there, we can match the generated hashes to the filenames on the file system, and we have enough information to regenerate the backed-up filesystem. Once we have the file system, it’s interesting to look around and find out what information was actually backed-up.  Here’s some highlights: SMS messages:  “Library/SMS/sms.db”.  This is a sqlite database. Address Book: “Library/AddressBook/AddressBook.sqlitedb”.  sqlite database. Notes: “Library/notes/notes.sqlite”.  sqlite database. Call History: “Library/CallHistory/call_history.db”. sqlite database. Photos: “Media/DCIM/”.  File systems of JPG files. SMS photos: “Library/SMS/Attachments”. File system of JPG files. Safari bookmarks: “Library/Safari/Bookmarks.db”.  sqlite database. I have working proof of concept code, however, in the interest of being a good guy, I’m keeping it private.  

Spelunking Apple DOS 3.3 DSK files

If you happen to have had an Apple][, you might be interested in retrieving your data from old DOS 3.3 floppies.  Or maybe you’re just interested in retrocomputing and downloaded some floppies at Apple2Online.  In my case, I had an ulterior motive.  I’ve been learning about FreeBSD device drivers and was wondering how difficult it would be to add support for Apple][ DOS 3.3 to FreeBSD.  I used Virtual][ as a desktop emulator to verify my DSK files. In short; use the md device to mount a DSK image, which is simply a binary dump of a floppy disk.  This is exactly how FreeBSD hackers build .img files or mount .iso images.  easy. Prior to writing code, the first task was to investigate the DSK image.  I used this image, for nostalgic reasons, however any valid DSK file should work.  I referenced this document for technical details of the VTOC, which I won’t explain here.  It is, important, however, that on a DOS 3.3 disk, the VTOC is located on track 17, sector 0. A DOS 3.3 disk has these characteristics: 35 tracks per disk 16 sectors per track 256 bytes per sector So a simple formula, in base-10, to find the byte offset of the first byte of track T, sector S is: Offset = T*16*256 + S*256 For a typical disk with 35 tracks, the total storage space works out to 35*16*256 = 143360 bytes / disk.  Not surprisingly, thats the byte size of a DSK image.   The first byte of the VTOC by the above formula, is therefore 17*16*256 = 69632. The arrangement of the VTOC, from the manuals is this: Byte        Description 00 Not Used 01 Track number of first catalog sector 02 Sector number of first catalog sector 03 DOS release number 04-05 Not Used 06 Volume number 07 – 26 Not Used 27 Maximum number of track-sector pairs which will fit in a track-sector list 28 – 2F Not Used 30 Last allocated track 31 Track allocation direction 32 – 33 Not Used 34 Tracks / diskette 35 Sectors / track 36 – 37 Bytes / sector 38 – FF Track-Sector allocation bitmaps In my case the first couple bytes of the VTOC are: 04 11 0F 03 00 00 FE 00 Disk catalog is at track 0x11, sector 0x0F.  Not surprisingly track 0x11 is track 17 in decimal. DOS 3.3 disk Volume number 0xFE Once we’ve read the VTOC, we can find the disk catalog, which contains the list of every file on the disk.  In my case the byte offset for the catalog would be 69632 + 10*256 = 72192 (0x11A00) A catalog sector looks like Byte        Description 00 Not Used 01 Track number of next catalog sector 02 Sector number of next catalog sector 03 – 0A Not used 0B – FF File descriptor entries (34 bytes each) There is room for 7 file descriptor entries in a catalog sector, each having this format Byte        Description 00 Track of the first track-sector list entry 01 Sector of the first track-sector list entry 02 File type 03 – 20 File name 21 – 22 File length, in units of sectors The start of my catalog is: 00 11 09 00 00 00 00 00 00 00 00 0F 0C 84 C7 CF D4 C8 C9 C3 AE D3 C5 D4 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 05 The next catalog sector is at track 0x11, sector 0x09 The first track sector list for the first file is at track 0x0F, sector 0x0C The file type is 0x84 (a locked binary file) The file name bytes are “C7 CF D4 C8 C9 C3 AE D3 C5 D4”. The file is 5 sectors in size The A0 character bytes are ASCII extended character codes for spaces.  The file names are ASCII with the high bit set, so the bytes “C7 CF D4 C8 C9 C3 AE D3 C5 D4”, as decimal with the high bit unset are “71 79 84 72 73 67 46 80 69 81” which is “G O T H I C . S E T”.   Therefore the file name is “GOTHIC.SET”. The track-sector list for “GOTHIC.SET” is track 0x0F, sector 0x0C.  The offset, in decimal, is therefore: 15*16*256+12*256 = 61440 + 3072 = 64512 The format of a track-sector list is: Byte        Description 00 Not Used 01 Track number of next track-sector list, or 00 02 Sector number of next track-sector list 03 – 04 Not used 05 – 06 Sector offset 07 – 0B Note used 0C – FF Track and sector pairs for data sectors, or 00 The bytes at this offset are: 00 00 00 00 00 00 00 00 00 00 00 00 0F 0B 0F 0A 0F 09 0F 08 There is no next track-sector list The track and sector pairs are for data for this file are all on track 0x0F, and they are sectors {0x0B, 0x0A, 0x09, 0x08}. So, to read this file, I would read the entire data of those four sectors.

Building FreeBSD-9.2 for Soekris using Crochet-BSD

Crochet-BSD is a github project which supports building FreeBSD disk images for various platforms such a RaspberryPi, BeagleBoard and others.  I had a Soekris 4501 laying around, so I decided to extend Crochet to build images for it. So, firstly, I needed a kernel.  Building the kernel config for Soekris is explained pretty well here.  Essentially: Add support for AMD Geode Remove everything that isn’t needed, for example SCSI support Add in various useful network goodies, such as pf and ALTQ. The resulting kernel configuration is here. The basic configuration for a new Crochet board is very well explained here.  I simply copied that and made modifications as necessary.   My Github fork of Crochet is here.  The end result is a handful of changes which ended up merged into Crochet-BSD. The boot log is here.  If you want to try the image on your Soekris box, it’s here. It turns out that building the kernel for Soekris is quite easy.   There are, however, some gotchas in building a working image: /boot/loader.conf needs to specify that the console device for loader is serial. /boot.config needs to pass “-h” to tell the boot loader to use the serial console /etc/ttys needs to be configured so that FreeBSD will use the Soekris RS-232 port as the system console.