Apache Commons Math

Commons Math is an Apache library which includes a variety of mathematical tools, including 1st and 2nd-order ODE solvers.  In order to familiarize myself with the ODE solvers, I wrote a simple program to solve an RC charging circuit.

The circuit has these parameters:

  • R (ohms), the resistance of the resistor
  • C (farads), the capacitance of the capacitor
  • V (volts), the voltage of the supply current

Differential equation solvers need initial conditions, so I've supplied the voltage across the capacitor as zero.  That is, the capacitor starts uncharged.

Commons Math requires that all equations are provided in the form

y' = f(y, t)

That is, the equation must provide the differential in terms of time, and the current state: y.   Note that both y' and y can be vectors.

In my case, the equation I've supplied is:

Vc' = - (Vc-Vin)/RC;

Where

  • Vc' is the derivative with respect to time of the capacitor charge
  • Vc is the current capacitor charge
  • Vin is the supply voltage

This equation is coded as an implementation of FirstOrderDifferentialEquations.  Note that I could have supplied an multiple equations using this interface, and I could have done something more complex using SecondOrderDifferentialEquations.  My implementation is here.

Once I've defined equations, there are multiple implementations of ODE solvers.   For simplicity I chose ClassicalRungeKuttaIntegrator, however any of the implementations of AdaptiveStepsizeIntegrator might be faster.  My code to invoke the integrator with a step size of 1e-6 seconds is here.

The output of the test using

  • R = 100KOhm
  • V = 12V
  • C = 10 nF
  • Vc(0) = 0 V
  • step size 1e-6 seconds
  • T (0) = 0 seconds
  • T (N) = .1 seconds

Starts like this:

1.0E-6,0.0011999400019999497
2.0E-6,0.0023997600159991993
3.0E-6,0.0035994600539959497
4.0E-6,0.0047990401279872
4.9999999999999996E-6,0.005998500249968752
5.999999999999999E-6,0.007197840431935207
6.999999999999999E-6,0.008397060685879965
8.0E-6,0.009596161023795232
9.0E-6,0.010795141457672009
1.0E-5,0.0119940019995001
1.1000000000000001E-5,0.01319274266126811
1.2000000000000002E-5,0.014391363454963448
1.3000000000000003E-5,0.01558986439257232
1.4000000000000003E-5,0.016788245486079736
1.5000000000000004E-5,0.017986506747469506
1.6000000000000003E-5,0.019184648188724243
1.7000000000000003E-5,0.020382669821825364
1.8000000000000004E-5,0.021580571658753083
1.9000000000000004E-5,0.02277835371148642
2.0000000000000005E-5,0.02397601599200319
2.1000000000000006E-5,0.025173558512280026

and ends like this:

0.09998900000007933,11.99945460123407
0.09999000000007933,11.99945465577122
0.09999100000007934,11.999454710302917
0.09999200000007934,11.99945476482916
0.09999300000007934,11.999454819349952
0.09999400000007934,11.999454873865291
0.09999500000007934,11.99945492837518
0.09999600000007934,11.999454982879616
0.09999700000007934,11.999455037378603
0.09999800000007934,11.99945509187214
0.09999900000007934,11.999455146360228

Pretty much exactly what you'd expect; the capacitor charges to ~12V, exponentially.

 

phpGrammar

I've recently becoming interested in porting legacy PHP sites to JSPs.   It seemed to me that one of the hardest parts of this problem was parsing the PHP code.  Once a parse tree was created, the next step would be to emit equivalent JSP code.

I went looking for an ANTL4 grammar for PHP, but could only find an ANTLR3 grammar, so I went to work updating the ANTLR3 grammar to ANTLR4 and writing a very simple validation suite.  The github project is here, and the resulting grammar is here.

 

jvmBasic 2.0

I've always had a fascination with compilers.  As a Java geek, I'm also quite interested in the JVM.  In order to learn a little more about both, and as a way to contribute to the open source world, I decided to implement a compiler for BASIC.   So, jvmBasic consumes BASIC code and emits .class files.

The first step was to build a parser and lexer for BASIC.  I decided to define an ANTLR4 grammar and use it to generate the lexer and parser.  BASIC is a fairly simple language, so the grammar was not difficult to define.  However, there are numerous BASIC dialects, so I had to pick a simple dialect.  jvmBASIC syntax looks much like Integer BASIC, but could easily be extended to parse GW-Basic, or maybe VB.  The resulting grammar is here.

Once ANTLR has generated a parser and lexer, it's possible to generate a parse tree for any BASIC input and then walk the tree emitting bytecode.  I used ASM to emit the bytecode.  An example BAS input file looks like this:

100 PRINT "Hello world"

The generated parse tree from jvmBASIC debug output looks like

- [1 line]
-  [3 linenumber]
-   [120 NUMBER] 100
-  [4 amprstmt]
-   [5 statement]
-    [7 printstmt1]
-     [4 'PRINT'] PRINT
-     [8 printlist]
-      [66 expression]
-       [60 func]
-        [118 STRINGLITERAL] "Hello world"
-  [122 CR]

Because there is no concept of functions, methods or classes in BASIC, I chose to enclose the generated code in a single method, of a single class.  The classname is the name of the BASIC input file, and the single method is:

public static void main(String[] args)

The class has two fields:

public InputStream inputStream;
public OutputStream outputStream;

The default values of inputStream and outputStream are System.in and System.out respectively.  However, in the case of jvmbasicwww, I replace them with HTTP input and output streams.

BASIC doesn't have new, delete, malloc, or free, or really any analogue of those.  Additionally, methods such as MID$ or perhaps VAL have certain semantics and behaviour.  In order to as closely as possible emulate BASIC, I implement jvmbasicrt.  Inside jvmbasicrt are implementation of each BASIC function, as well as a class called ExecutionContext. ExecutionContext includes the "guts" of a BASIC runtime:

  • A stack.  Similar to many programming languages, BASIC needs a stack.
  • All variables.  This is simple a hashtable of Values, keyed on the Variable name.

Additionally there is Value which implements a variable with BASIC semantics.

There is a maven mojo which wraps jvmbasicc.  The mojo jvmbasicmojo, compiles all BASIC files in "/src/main/bas" and produces a .class file for each one.  This mojo can be used to incorporate BASIC files into any normal maven project and then link them into a .jar file.

An additional example BASIC file is:

10 REM this is a comment
20 PRINT "13"
30 PRINT "hi"
40 PRINT 10
50 PRINT 15.55
60 LET x = 12
70 PRINT "hihi"
80 PRINT x
90 LET y = 1+2
100 LET z = 3*6
110 LET d= y+z
120 PRINT d

The maven pom file that uses jvmbasicmojo is here.

The javap output for the generated .class file is:

public class EXAMPLE1 {
 public com.khubla.jvmbasic.jvmbasicrt.ExecutionContext executionContext;
 public java.io.InputStream inputStream;
 public java.io.PrintStream outputStream;
 public EXAMPLE1();
 public static void main(java.lang.String[]);
 public void program() throws java.lang.Exception;
}

There isn't a big demand, that I'm aware of, for bytecode compilers for BASIC.  Two potential applications that come to mind are:

  • Running VB code on the JVM.  Theoretically it would be possible to extend the grammar to include VB, and then to emit bytecode for VB programs.  This would form the foundation of technology to run .asp applications on the JVM.  The VB standard library would have to be implemented too.
  • Cross-compilation.  Again, theoretically, it should be possible to use the grammar file to implement a cross compiler which consumes VB code and emits JSP code, or even PHP code.

 

Building the Wandboard boot loader

I've bee hacking away trying to set up Crochet-BSD to build boot images for Wandboard.  Wandboard uses Cortex A9 processor, so it's ARM.  The linux distros for it use U-Boot, so that seemed like a likely place to start for FreeBSD.

Looking at the Wandboard Wiki, there is an article on U-Boot which explains how to build U-Boot for Wandboard.   Firstly, download the October 2013 release of U-Boot here.  If you are building on FreeBSD, you'll need this patch to the Makefile to include libc.

Then, build U-Boot for wandboard:

make wandboard_quad_config
make

and copy the imx file to the sd card

sudo dd if=u-boot.imx of=/dev/mmcblk0 bs=1 seek=1024

This writes the imx file 1K into the disk; which is where the i.mx6 processor expects to find the U-boot image.  The processor manual is here.  It specifies offset 0x400 as the Program Image start on the SD card.  The U-boot startup looks like this:

CPU: Freescale i.MX6Q rev1.2 at 792 MHz
Reset cause: POR
Board: Wandboard
DRAM: 2 GiB
MMC: FSL_SDHC: 0, FSL_SDHC: 1
*** Warning - bad CRC, using default environment
In: serial
Out: serial
Err: serial
Net: FEC [PRIME]
Hit any key to stop autoboot: 1 0 
mmc0 is current device
SD/MMC found on device 0

Keep in mind that the default serial protocol for Wandboard is 8n1 and 115200.

Once U-boot is started, you will need to boot the FreeBSD kernel.  My disk image is partitioned with a FAT partition on which I have kernel.bin, and a UFS partition which is the FreeBSD root file system.  kernel.bin is the raw kernel.  The partition table looks like this:

Disk: /dev/rdisk1 geometry: 3880/255/63 [62333952 sectors]
Signature: 0xAA55

Starting       Ending

#: id  cyl  hd sec -  cyl  hd sec [     start -       size]
------------------------------------------------------------------------
*1: 0C    8   5   1 -   58  29  63 [     16443 -     102375] Win95 FAT32L
2: A5   58  30   1 -  992   1  63 [    118818 -    1881180] FreeBSD
3: 00    0   0   0 -    0   0   0 [         0 -          0] unused
4: 00    0   0   0 -    0   0   0 [         0 -          0] unused

You should notice that the FAT partition starts at 16443.  I've moved the start of the FAT partition further into the disk to accomodate u-boot at disk offset 0x400.

Once you've got an SD card with u-boot and kernel.bin on it, and you've booted to u-boot, you'll want to start the kernel.  At the u-boot console, load the kernel.bin into memory at address 0x12000000.

fatload mmc 0:1 12000000 kernel.bin

The address 0x1200000 is specified in the FreeBSD ARM configuration for Freescale.  Check here. Once the kernel is loaded, go ahead and start it:

go 12000000

This will start FreeBSD:

KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r259276: Thu Dec 12 17:20:23 MST 2013
 tom@bernice:/storage/home/tom/crochet/crochet-wandboard/crochet-freebsd/work/obj/arm.arm/storage/home/tom/crochet/src/FreeBSDHead/head/sys/WANDBOARD-QUAD arm
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
Preloaded elf kernel "kernel" at 0xc24afdac.
CPU: Cortex A9-r2 rev 10 (Cortex-A core)
 Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext
 WB disabled EABT branch prediction enabled

In my case, I've run into a kernel crash starting the kernel, so I don't have a fully working system yet, but I do have a working boot loader.  The Crochet-BSD code to build an image file containing USB, a partitioned disk image, kernel.bin and the userland is here:

Building a NetBSD Release

Having built FreeBSD numerous times, I thought it would be interesting to know how to build NetBSD.  The instructions from the NetBSD documentation are here.

Firstly get the sources.  An easy way is to simply FTP them.  I got the NetBSD 6.1.2  sources from here.  These files are all tarballs, so they'll need to be extracted.

Once the sources are extracted, it's necessary to build the toolchain.  This toolchain can be used to cross-compile for any architecture supported by NetBSD, however I decided to build for i386.

cd usr/src/
./build.sh -m i386 tools

Once you've done this you can simply build a kernel:

./build.sh -u -m i386 kernel=GENERIC

Or, you can build a release which includes the userland:

./build.sh -U -u -m i386 release

The completed binaries will be in usr/src/obj/

 

Restoring file systems from iTunes backups

I've been reading lots of interesting information about iTunes and IOS, so I thought I would investigate, just what is in an iTunes backup.  Typically, on OS X, you can find your iTunes backup here, under the appropriate OS X user profile:

/Library/Application Support/MobileSync/Backup

When you look at the backup, it's a giant list of 40 character hexadecimal file names.   After doing some quick reading on theiphonewiki.com, those file names are SHA-1 hashes.  Each of the files, is a backed-up file from the iPhone.   The problem of restoring the file system then is that we need to find the original file names from the hashes.

Luckily, Apple provides an index.  There is a file called "Manifest.mbdb" which is a binary index of all the SHA-1 files.  There is a pretty good description of the format of that file here.   After reading the Manifest.mbdb into memory, we have enough information to generate all the SHA-1 hashes.  From there, we can match the generated hashes to the filenames on the file system, and we have enough information to regenerate the backed-up filesystem.

Once we have the file system, it's interesting to look around and find out what information was actually backed-up.  Here's some highlights:

  • SMS messages:  "Library/SMS/sms.db".  This is a sqlite database.
  • Address Book: "Library/AddressBook/AddressBook.sqlitedb".  sqlite database.
  • Notes: "Library/notes/notes.sqlite".  sqlite database.
  • Call History: "Library/CallHistory/call_history.db". sqlite database.
  • Photos: "Media/DCIM/".  File systems of JPG files.
  • SMS photos: "Library/SMS/Attachments". File system of JPG files.
  • Safari bookmarks: "Library/Safari/Bookmarks.db".  sqlite database.

I have working proof of concept code, however, in the interest of being a good guy, I'm keeping it private.

 

Spelunking Apple DOS 3.3 DSK files

If you happen to have had an Apple][, you might be interested in retrieving your data from old DOS 3.3 floppies.  Or maybe you're just interested in retrocomputing and downloaded some floppies at Apple2Online.  In my case, I had an ulterior motive.  I've been learning about FreeBSD device drivers and was wondering how difficult it would be to add support for Apple][ DOS 3.3 to FreeBSD.  I used Virtual][ as a desktop emulator to verify my DSK files.

In short; use the md device to mount a DSK image, which is simply a binary dump of a floppy disk.  This is exactly how FreeBSD hackers build .img files or mount .iso images.  easy.

Prior to writing code, the first task was to investigate the DSK image.  I used this image, for nostalgic reasons, however any valid DSK file should work.  I referenced this document for technical details of the VTOC, which I won't explain here.  It is, important, however, that on a DOS 3.3 disk, the VTOC is located on track 17, sector 0.

A DOS 3.3 disk has these characteristics:

  • 35 tracks per disk
  • 16 sectors per track
  • 256 bytes per sector

So a simple formula, in base-10, to find the byte offset of the first byte of track T, sector S is:

Offset = T*16*256 + S*256

For a typical disk with 35 tracks, the total storage space works out to 35*16*256 = 143360 bytes / disk.  Not surprisingly, thats the byte size of a DSK image.   The first byte of the VTOC by the above formula, is therefore 17*16*256 = 69632.

The arrangement of the VTOC, from the manuals is this:

Byte        Description
00          Not Used
01          Track number of first catalog sector
02          Sector number of first catalog sector
03          DOS release number
04-05       Not Used
06          Volume number
07 - 26     Not Used
27          Maximum number of track-sector pairs which will fit in a track-sector list
28 - 2F     Not Used
30          Last allocated track
31          Track allocation direction
32 - 33     Not Used
34          Tracks / diskette
35          Sectors / track
36 - 37     Bytes / sector
38 - FF     Track-Sector allocation bitmaps

In my case the first couple bytes of the VTOC are:

04 11 0F 03 00 00 FE 00
  • Disk catalog is at track 0x11, sector 0x0F.  Not surprisingly track 0x11 is track 17 in decimal.
  • DOS 3.3 disk
  • Volume number 0xFE

Once we've read the VTOC, we can find the disk catalog, which contains the list of every file on the disk.  In my case the byte offset for the catalog would be

69632 + 10*256 = 72192 (0x11A00)

A catalog sector looks like

Byte        Description
00          Not Used
01          Track number of next catalog sector
02          Sector number of next catalog sector
03 - 0A     Not used
0B - FF     File descriptor entries (34 bytes each)

There is room for 7 file descriptor entries in a catalog sector, each having this format

Byte        Description
00          Track of the first track-sector list entry
01          Sector of the first track-sector list entry
02          File type
03 - 20     File name
21 - 22     File length, in units of sectors

The start of my catalog is:

00 11 09 00 00 00 00 00 00 00 00 0F 0C 84 C7 CF D4 C8 C9 C3 AE D3 C5 D4 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 A0 05
  • The next catalog sector is at track 0x11, sector 0x09
  • The first track sector list for the first file is at track 0x0F, sector 0x0C
  • The file type is 0x84 (a locked binary file)
  • The file name bytes are "C7 CF D4 C8 C9 C3 AE D3 C5 D4".
  • The file is 5 sectors in size

The A0 character bytes are ASCII extended character codes for spaces.  The file names are ASCII with the high bit set, so the bytes "C7 CF D4 C8 C9 C3 AE D3 C5 D4", as decimal with the high bit unset are "71 79 84 72 73 67 46 80 69 81" which is "G O T H I C . S E T".   Therefore the file name is "GOTHIC.SET".

The track-sector list for "GOTHIC.SET" is track 0x0F, sector 0x0C.  The offset, in decimal, is therefore:

15*16*256+12*256 = 61440 + 3072 = 64512

The format of a track-sector list is:

Byte        Description
00          Not Used
01          Track number of next track-sector list, or 00
02          Sector number of next track-sector list
03 - 04     Not used
05 - 06     Sector offset
07 - 0B     Note used
0C - FF     Track and sector pairs for data sectors, or 00

The bytes at this offset are:

00 00 00 00 00 00 00 00 00 00 00 00 0F 0B 0F 0A 0F 09 0F 08
  • There is no next track-sector list
  • The track and sector pairs are for data for this file are all on track 0x0F, and they are sectors {0x0B, 0x0A, 0x09, 0x08}.

So, to read this file, I would read the entire data of those four sectors.

Building FreeBSD-9.2 for Soekris using Crochet-BSD

Crochet-BSD is a github project which supports building FreeBSD disk images for various platforms such a RaspberryPi, BeagleBoard and others.  I had a Soekris 4501 laying around, so I decided to extend Crochet to build images for it.

So, firstly, I needed a kernel.  Building the kernel config for Soekris is explained pretty well here.  Essentially:

  • Add support for AMD Geode
  • Remove everything that isn't needed, for example SCSI support
  • Add in various useful network goodies, such as pf and ALTQ.

The resulting kernel configuration is here.

The basic configuration for a new Crochet board is very well explained here.  I simply copied that and made modifications as necessary.   My Github fork of Crochet is here.  The end result is a handful of changes which ended up merged into Crochet-BSD.

The boot log is here.  If you want to try the image on your Soekris box, it's here.

It turns out that building the kernel for Soekris is quite easy.   There are, however, some gotchas in building a working image: