Introduction

This is my 3rd blog posting on the topic of the Crochet-FreeBSD ARM boot process.  The other two are here and here.   At long last, I have the Crochet-FreeBSD build for Wandboard working properly, with U-boot and ubldr.  This article will serve, I hope, to document the entire process and give others a place to start in booting FreeBSD other embedded ARM devices.   If you want to follow along via the boot log, it's here.  Bootable images are here.

Booting FreeBSD on an ARM device has three primary steps:

1. U-Boot
2. ubldr
3. FreeBSD kernel

Diagrammatically, it looks like this:

When the Wandboard starts, it loads a boot image from the mmc , at a known location on the disk.  In the case of Crochet-FreeBSD this image is U-boot.  After U-boot starts it loads and runs ubldr, which in turn loads the FreeBSD kernel and boots it.

Disk Layout

The Wandboard documentation here shows the basic requirements of the disk layout on the mmc card.  In particular:

• The MBR is at the start of the disk, and is about 0x200 bytes
• The U-boot boot loader is expected to be at offset 0x400 (1024) on the disk.

In the case of the Crochet-FreeBSD image for Wandboard, I've chosen this layout

• MBR at the start of the disk, about 0x200 bytes long
• At offset 0x400, is my U-boot boot loader, as required by the Wandboard
• A FAT partition 50MB in size at offset 16384 (0x4000) on the disk.   This is the partition that the U-Boot configuration file, and ubldr will live on.
• A UFS partition which is the remainder of the disk.  This will be the FreeBSD root filesystem.

For FAT32 partitions of less than 512MB size, the block size is 4KB.  So, 0x4000 blocks from the start of the disk is 64MB into the disk.  Given that U-boot is about a 300KB binary, we can be quite sure that the FAT partition will not overlap with U-Boot.  ubldr compiles to a 250KB binary and is stored on a 50MB partition; also plenty of space.

The disk layout looks like this

root@wandboard:~ # gpart show
=> 63 30678977 mmcsd0 MBR (15G)
 63 16380 - free - (8.0M)
 16443 102375 1 !12 [active] (50M)
 118818 1881180 2 freebsd (919M)
 1999998 28679042 - free - (14G)
=> 0 1881180 mmcsd0s2 BSD (919M)
 0 1881180 1 freebsd-ufs (919M)

U-Boot

U-Boot is a very capable boot loader, that can boot a variety of architectures, of which one is ARM.   In the case of Wandboard, however a couple changes are needed to the configuration.   The patch files are here.  The primary requirements are:

• U-Boot needs to be configured to read ELF binaries
• U-Boot needs to be configured to include the U-Boot API, a feature which ubdlr requires.
• The Makefile needs to be changed, to include libc specifically

When U-Boot starts, it looks for the file "uEnv.txt"' on the file system.  It's very important that the first partition on the file system be FAT, since U-Boot doesn't include UFS support.  The contents of uEnv.txt are instructions to U-Boot to load ubldr and start it.  Specifically:

uenvcmd=fatload mmc 0:1 88000000 ubldr;bootelf 88000000;

These U-Boot commands mean:

• From the FAT disk unit "mmc 0" on the 1st slice load "ubldr" into RAM location 0x88000000.
• Boot the ELF image at 0x88000000

From here, ubldr will start.  The reason we had compiled U-Boot with ELF support is that ubldr is an ELF binary, so we needed the U-Boot command "bootelf".

The choice of memory address 0x88000000 is mostly arbitrary.  According to the manuals, the RAM starts at address 0x100000, so this number has to be larger than 0x100000, smaller than the physical size of the RAM, and not conflict with the memory address that U-Boot was loaded at.  I suspect the Wandboard loaded U-Boot at 0x100000.

ubldr

ubldr is an ARM implementation of loader(8).  It's not technically necessary to use ubldr; U-Boot could just boot the kernel directly, but having an implementation of loader(8) is quite useful.  For example, it provides a serial console for kernel debugging, and it allows passing flags to the kernel at startup time.   Some drivers, such as urtwn, for example require passing flags to the kernel to accept licensing terms for binary blobs.

Since ubldr is not relocatable, it's necessary to compile it with the memory address that it will be loaded at.  If you look here, you will notice that the address 0x8800000 was passed to the compile command.

An important aspect of ubldr for Wandboard is that the FDT is compiled into the kernel.   ubldr can use external device blobs, or it can use device blobs that are compiled into the kernel.  If you look at the kernel config for Wandboard here, you can see that it specifies compiled-in device blobs.  This message from ubldr shows that it's using a device tree blob (DTB) compiled into the kernel

Booting [/boot/kernel/kernel]...               

Using DTB compiled into kernel.

When ubldr starts, it'll mount the UFS root file system, and read the file /boot/loader.conf.  In that file we can configure the boot loader, including passing kernel parameters, setting the serial console speed, configuring boot loader menus, etc.   It's worth noting that ubldr configurations are coded in FORTH.

Another aspect of ubldr which is important is that it uses the U-Boot API.  If you're interested to know how that works, the ubldr source is here.  The Makefile pulls in U-Boot deps, and they are referred to in conf.c.

On the topic of ubldr, there is an excellent blog post here that is worth reading.

FreeBSD Kernel

Finally, ubldr will boot the kernel.  The default location for the kernel on the root filesystem is /boot/kernel/kernel.   The file /etc/fstab should be configured to mount appropriate file systems when the kernel starts.  If you look at the kernel configuration for Wandboard here, you can see that the Kernel is configured to find the root file system on the first mmc device, on the second slice.   Specifically:

# U-Boot stuff lives on slice 1, FreeBSD on slice 2.
options ROOTDEVNAME=\"ufs:mmcsd0s2a\"

Finally

One trick that was used for Crochet-FreeBSD's Beaglebone build was to configure /etc/fstab to mount the FAT partition to /boot/msdos.  The specific configuration is

/dev/mmcsd0s1   /boot/msdos     msdosfs rw,noatime      0 0

This is interesting because uEnv.txt is on this partition. We can modify it if we want to try new configurations without rebuilding the image.