Cross-compiling FreeBSD ARM on AMD-64

I've been learning about FreeBSD kernel development, and discovering that, not surprisingly, it's very, very slow to compile on a Raspberry Pi.  So, I decided to figure out how to compile the Pi kernel on a nice fast AMD-64.

Firstly, get the kernel sources to a subdirectory under a non-root user profile:

mkdir fbsd
cd fbsd
svn co http://svn0.us-east.freebsd.org/base/head src/FreeBSD/
cd src/FreeBSD

Next, we need to set some environment variables:

export MAKEOBJDIRPREFIX=/storage/home/tom/fbsd/obj
export TARGET_ARCH=armv6

Then, in order to cross-compile we need the kern build tools, so build those:

make kernel-toolchain

Then finally, build the kernel

make buildkernel KERNCONF=RPI-B

The completed kernel should show up in '/storage/home/tom/fbsd/obj/arm.armv6/storage/home/tom/fbsd/src/FreeBSD/sys/RPI-B'.  There will be a large number of .ko files in there and of course a file named "kernel".  Copy everything to the directory '/boot/kernel' on the Raspberry Pi, and reboot.

The FreeBSD FDT, the Raspberry PI SPI

How the FDT works

In specific, I'm looking at this file bcm2835.dtsi which contains the hardware definitions for the BCM2835, the SOC which powers a Raspberry Pi. Primarily, i'm interested in how to read it.   To really make sense of this, you should have this pdf at hand; it's the hardware manual for BCM2835.  If you're interested the FreeBSD code for OpenFirmware is here.

devicetree.org has a pretty good explanation of how to read a dts file, here.

Looking at the start of the BCM2835 dts, I see this:

cpus {
  cpu@0 {
    compatible = "arm,1176jzf-s";
  };
};

This defines the CPU with the syntax "<manufacturer, model>.  So the BCM2835 has an ARM CPU and the model is "1176jzf-s".

Next I see:

SOC: axi {
   compatible = "simple-bus";
   #address-cells = <1>;
   #size-cells = <1>;
   reg = <0x20000000 0x01000000>;
   ranges = <0 0x20000000 0x01000000>;

This defines an axi bus, which is compatible with the FreeBSD simple-bus.

These two lines:

 #address-cells = <1>;
 #size-cells = <1>;

Describe that an address cell is 32 bits wide, and that size cells are also 32 bits wide.

This line

reg = <0x20000000 0x01000000>;

describes that the axi bus is memory mapped to the memory range at address 0x20000000  and that the memory range is 0x01000000 bytes wide.

So, looking at some peripherals, I see this for the gpio

reg = <0x200000 0xb0>;

So the gpio is at address 0x200000 within the range specified for the bus, and is 0xb0 cells wide (176, 32-bit bytes).  This is consistent with page 90-91 of the BCM2835 PDF.  Note that the PDF gives the memory range as starting at 0x7E200000.  The PDF specifies memory addresses as bus addresses, but the FDT specifies them as physical addresses.  Page 5 of the PDF explains the difference with a diagram.  Section 1.2.4 on  page 6 of the PDF explains that bus addresses are used in the pdf.

The SPI

The SPI base address is specified on page 152 of the PDF.  It's as bus address 0x7E204000 and its 6 - 32bit cells wide. So, an appropriate declaration for SPI0, in polled mode, could look like:

spio0 {
	compatible = "broadcom,bcm2835-spi";
	reg = <0x204000 0x06>;
};

 

The skeleton of a FreeBSD driver

It's been a goal of mine for a while to understand how FreeBSD device drivers work.  So, firstly I wrote a simple module; it's here.  The next step is to write the skeleton of a device driver.  That is, a device driver that does nothing.  The resulting source code is here.

The Makefile is very similar to that of a module.  In fact, it's basically identical:

SRCS+=ofw_bus_if.h bus_if.h device_if.h foodev.c
KMOD=foodev
.include <bsd.kmod.mk>

I include the headers I need, to inform the make to build them from .m files, and I include my C file which contains my driver.  The last line includes everything else I need to make a driver.

The source code for my driver, for the imaginary "foo" device is here.

Firstly, I declared a struct called "foodev_softc".  This struct defines the state data for my driver.  Every driver in FreeBSD can have a softc, containing the drivers state data.

struct foodev_softc 
{
	 device_t sc_dev;
};

In the case of my driver I'm storing only the "device_t" that the kernel passed for this device.  However other drivers store resources here such as memory, irqs or DMA.

There are three functions declared "foodev_probe", "foodev_attach" and  "foodev_detach".  You can learn about those here.  They are called by the kernel, in order when the device is probed, attached and detached.

Next, I declare an array of type "device_method_t", which contains the entry points of the probe, attach and detach methods for my driver:

static device_method_t foodev_methods[] = {
	DEVMETHOD(device_probe, foodev_probe),
	DEVMETHOD(device_attach, foodev_attach),
	DEVMETHOD(device_detach, foodev_detach),
	DEVMETHOD_END
};

device_method_t is defined in "bus.h"

Similar to a module, there is a struct that defines a driver.  Here is mine:

static driver_t foodev_driver = {
	"foodev",
	foodev_methods,	
        sizeof(struct foodev_softc),
};

This struct contains the name of my module, the struct which contains my module's entry points, and the size of the foodev_softc struct, since the kernel needs to know how many bytes of memory to allocate for my driver's state data.  "driver_t" is defined in bus.h also.

Finally, also similar to a module, I use a macro to declare my driver:

DRIVER_MODULE(foodev, simplebus, foodev_driver, foodev_devclass, 0, 0);

"DRIVER_MODULE" is defined in bus.h

The last piece which remains to be explained is this:

static devclass_t foodev_devclass;

Every driver defines a static instance of "devclass_t", and "devclass_t" is defined in bus.h.  A "devclass_t" is a pointer to a "devclass", which you will find in subr_bus.c.  So, the declaration above simply declares a pointer to a "devclass" which the kernel will allocate and use to store information such as the parent of the device and all devices owned by the device.

The results of loading the driver on my Pi look like this:

root@raspberry-pi:/home/pi/foodev # kldload ./foodev.ko
 foodev probe
 foodev probe
 foodev attach
 foodev probe
 foodev probe
 foodev attach
 foodev probe
 foodev probe
 foodev attach
 foodev probe
 foodev probe
 foodev attach
 root@raspberry-pi:/home/pi/foodev # kldstat -v | grep foo
 5 1 0xc262b000 9000 foodev.ko (./foodev.ko)
 108 simplebus/foodev
 root@raspberry-pi:/home/pi/foodev # kldunload ./foodev.k
 foodev.kld foodev.ko*
 root@raspberry-pi:/home/pi/foodev # kldunload ./foodev.ko
 root@raspberry-pi:/home/pi/foodev # kldstat -v | grep foo

dmesg shows this:

foodev0 mem 0x2000b400-0x2000b423 irq 0 on simplebus0
foodev1 mem 0x2000b800-0x2000b84f irq 2 on simplebus0
foodev2 mem 0-0xdec0addd on simplebus1
foodev3 mem 0x3-0xdec0ade0 on simplebus1
foodev0: detached
foodev1: detached
foodev2: detached
foodev3: detached

 

 

Writing a FreeBSD loadable module

In preparation for writing a device driver, I thought I would firstly write a module.  The resulting source code is here.   Firstly the Makefile:

SRCS+=bus_if.h device_if.h examplemodule.c
KMOD=examplemodule
.include <bsd.kmod.mk>

The first line defines the sources.   examplemodule.c is my module source.  I needed to include bus_if.h and device_if.h since I'm writing my module for newbus, and those files are generated from bus_if.m and device_if.m, respectively.  Including the .h files as sources tells the FreeBSD make that it needs to generate those from the .m files.

The second line simply defines the name I wish to give my module.  This Makefile will generate "examplemodule.ko".   Finally the last line is the magic the FreeBSD team has defined which contains everything else I need to make my module.

The entry point for my module looks like this:

static int
examplemodule_modevent(module_t mod, int type, void *unused)
{

	switch (type) {
	case MOD_LOAD:
		uprintf("Loaded examplemodule \n");
		return (0);
	case MOD_UNLOAD:
		uprintf("Unloaded examplemodule \n");
		return (0);
	}
	return (EINVAL);
}

When FreeBSD loads my module, or unloads it, this method will be called.  The body of the module is pretty self-explanatory.

There is also this:

static moduledata_t examplemodule_mod = {
        "examplemodule",
        examplemodule_modevent,
        0
};

This is the definition of my module. It contains the name of my module ("examplemodule") and the entry point "examplemodule_modevent".  If you're curious about "moduledata_t", it's a struct defined in sys/module.h

Finally a macro is used to declare the module.  Of course, this macro refers to the module definition "examplemodule_mod"

DECLARE_MODULE(examplemodule, examplemodule_mod, SI_SUB_DRIVERS, SI_ORDER_MIDDLE);

The macro "DECLARE_MODULE" is also defined in sys/module.h

Loading the module produces this:

root@raspberry-pi:/home/pi/freebsd-module # kldload ./examplemodule.ko 
Loaded examplemodule 
root@raspberry-pi:/home/pi/freebsd-module # kldunload ./examplemodule.ko
Unloaded examplemodule

 

Building a Beaglebone FreeBSD Kernel

Building a FreeBSD kernel for Beaglebone is not terribly difficult.  Firstly, you need the FreeBSD-Current sources:

svn co http://svn0.us-east.freebsd.org/base/head /src/FreeBSD/

Next, build the cross-development tools for FreeBSD:

cd /src/FreeBSD/head
make XDEV=arm XDEV_ARCH=armv6 xdev

Then get Crochet-FreeBSD.  This is by far the easiest way to build a FreeBSD kernel for Beaglebone

git clone git://github.com/kientzle/crochet-freebsd.git

Once you have Crochet, you'll need u-boot. You can get the latest sources here.  You'll need to drop them into your crochet directory.

Crochet needs the "auditdistd" user, so if you're not on FreeBSD-10, or CURRENT, you'll need to add that user:

pw useradd auditdistd  -s /usr/sbin/nologin -g wheel

Finally, once you've editted "config.sh" to set up the source directory for FreeBSD and to select the Beaglebone build, you can run it:

sh crochet.sh -c config.sh

Once crochet has finished, it will have produced an image file into the /work/ subdirectory of crochet-freebsd.  Write that image file to a sd card.

dd if=FreeBSD-armv6-BEAGLEBONE.img of=/dev/rdisk1 bs=1m

You'll likely want to set up a serial console.  In order to do that, you need one of these, a PL2303 serial cable.  You can get one here.  The pin connections are as described at circuitco.

Function      Beagleboard    Wire
Ground        Pin1           Black
Receive       Pin4           Green
Transmit      Pin5           White

On OS X, you'll need PL2303 serial drivers.  You can get those here.  You should then have this device on OS X

/dev/tty.PL2303-000013FA

Once the Beagleboard is plugged in, you can get a serial console like this:

screen /dev/tty.PL2303-000013FA 115200

To boot the SD card, follow the instructions here.  For simplicity sake, I have included the instructions:

  1. Disconnect the power to the Beagleboard
  2. Connect the serial cable to both the Beagleboard and the computer
  3. Insert the micro-SD card
  4. Hold the Boot switch.  This is the switch at the opposite end of the board from the LEDs, adjacent to the SD card slot
  5. Connect the power to the Beagleboard
  6. Release the Boot switch

In my case, I had connected the serial cable and brought up screen previously; that way I could see the boot-up messages.

If you want to try it for yourself, the .img file is here.

 

Beaglebone Serial Console on OS X

I recently got a Beaglebone Black.   I was very excited; it's similar to a Pi, but has 60 GPIO ports!

Getting a serial console up on it was not quite a simple as I thought, since the BeagleBone uses FTDI.  Following the instructions here I installed the OS X serial driver for FTDI.   After plugging in the device, I had this new serial device:

/dev/tty.usbmodemfa133

So to get a serial console up I typed:

screen /dev/tty.usbmodemfa133 115200

Once I had the Beaglebone started and had a network connection, the next step was to update the package manager.  If you're interested, the default OS on the Beaglebone is Angstrom Linux, and the package system is Itsy.  You can view the available packages here.

opkg update
opkg upgrade

Since I had a USB Wifi Adapter, it seemed to make sense to use it.  My adapter is an PCUSBW1150.

opkg list 'linux-firmware-rt*'opkg install linux-firmware-rtl8192cu

After a reboot, I now have a wlan0 device:

wlan0 Link encap:Ethernet HWaddr 00:02:72:C3:AB:72
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Angstrom uses connman to manage network settings.  My connman configuration for my wifi is in the file '/var/lib/connman/wifi.config' and looks like:

[service_home]
Type=wifi
Name=shrek2.5
Security=wpa
Passphrase=passpasspass

Once connman is configured, restart it

systemctl restart connman.service

Finally, some additional configuration. I installed nano

opkg install nano

To change the hostname, edit the file '/etc/hostname'

 

Embedding Jasper

Jasper is the JSP compiler inside Tomcat. For reasons, mainly of curiosity, I wanted to build a Pragmatach plugin which exposes Jasper.  Pragmatach supports some template engines such as FreeMarker, ThymeLeaf and Velocity, but I thought Jasper would be a good addition.

I chose to use Tomcat 6, mainly because Tomcat 7 uses Servlet 3.0.  Pragmatach is currently on Servlet 2.5 .   Luckily, there is a really helpful example of compiling JSPs right inside Tomcat; the JspC shell.  JspC is a simple command-line executable which can consume jsp files and produce both .java files and .class files.

The code I ended up with, is here.

Pragmatach

Firstly, here's the site, and here's the code.  There's a slideshow, here.

Pragmatach is a Java-based web development platform that I wrote, based on ideas from Struts2Play, JSF2 and Grails.   My goals in writing it were:

  • Configuration via annotations.  I wanted to be able to fully configure a controller using only annotations and without having to write routes files, or XML
  • Simplicity.  I wanted to be able to quickly create a skeleton project with only a couple keystrokes
  • Modularity.  I wanted to be able to extend the framework quickly and easily
  • JSON and XML support. Client-side programmers prefer to communicate with servers via JSON and XML, so I wanted simple, easy JSON and XML support
  • Multiple Template Engines. I wanted to have support for multiple template engines.
  • Maven support.  I didn't want to use any other build system than maven.
  • Multiple container support.  I wanted to always produce a generic war file, and to have that war file run in any standard J2EE container.

Pragmatach supports these goals.  To create a Pragmatach program, simply use the Maven Archetype, here, at the command line.  This will create a fully functional Pragmatach application with:

  • A working pom.xml
  • A fully maven-compliant file system
  • Support for multiple containers, detailed here
  • The pragmatach Admin console
  • A single "hello world" Controller and Template.

The generated controller looks like this:

import java.util.Date;
import com.khubla.pragmatach.framework.annotation.Controller;
import com.khubla.pragmatach.framework.annotation.Route;
import com.khubla.pragmatach.framework.annotation.View;
import com.khubla.pragmatach.framework.api.PragmatachException;
import com.khubla.pragmatach.framework.api.Response;
import com.khubla.pragmatach.plugin.freemarker.FreemarkerController;
@Controller(name="IndexController")
@View(view="index.ftl")
public class IndexController extends FreemarkerController {
 /**
 * the message
 */
 private final String message = "Welcome to Pragmatach";
 public String getMessage() {
 return message;
 }
 @Route(uri = "/")
 public Response render() throws PragmatachException {
 return super.render();
 }
 public String getTime() {
 return new Date().toString();
 }
}

A key aspect of the generated controller is the annotations.

  • The @Controller annotation tells Pragmatach that the class is a Controller.  By default controllers are session-scoped, but other scopes are available.
  • The @View annotation tells Pragmatach where to find the view template to render.  In this case "index.ftl", a FreeMarker template
  • The @Route annotation on the method "render" tells Pragmatach to bind the method "IndexController:render" to the HTTP context "/".  It's possible to bind routes to multiple methods on a class, which can be quite useful.   Additionally, via the @RouteParameter annotation, it's possible to bind parameters in the HTTP context path to Java method parameters.

In the case of the generated application, the view template is quite simple also.  It looks like this:

<!DOCTYPE html>
<head>
 <title>Pragmatach</title>
</head>
<body>
<h1>Pragmatach</h1>
${controller.message}
<h2>Current Time</h2>
${controller.time}
<a href="${session.getServletContext().getContextPath()}/pragmatach/admin">Admin</a>
</body>

Since we have chosen FreeMarker as the template engine, we bind controller class methods using the ${} syntax.  The code "${controller.message} ends up binding the method "IndexController:getMessage".

There is a lot more to Pragmatach, including the Admin console, ORM support, and a small ecosystem of plugins.  The documentation is here.  If you'd like to see some working examples, they are here.

The pragmatach web site is here.  If you're interested in contributing, have a comment or need to ask a question, you can do that on Google Groups.  The documentation is here and the latest Javadoc is here.

 

 

 

Servlet mocking with Mockito

When writing custom servlets, it can be quite useful to unit test them.  In my case, I used a combination of testng and mockito.  The basic idea is simple; mock a HttpServletRequest, and a HttpServletResponse and pass them to the custom servlet.  There is a bit more too it, so here's an example.

final HttpServletRequest httpServletRequest = mock(HttpServletRequest.class);
when(httpServletRequest.getPathInfo()).thenReturn("/lineup/world.xml");
final HttpServletResponse httpServletResponse = mock(HttpServletResponse.class);
final StubServletOutputStream servletOutputStream = new StubServletOutputStream();
when(httpServletResponse.getOutputStream()).thenReturn(servletOutputStream);
final ServletConfig servletConfig = mock(ServletConfig.class);
when(servletConfig.getInitParameter("defaultPool")).thenReturn("testpool1");

This sets up an HttpServletRequest, an HttpServletResponse, a ServletConfig where i can pass in parameters that the container would have read from web.xml, and a "StubServletOutputStream", which is just a convenient wrapper around a ByteArrayOutputStream.

My StubServletOutputStream looks like this:

public class StubServletOutputStream extends ServletOutputStream {
 public ByteArrayOutputStream baos = new ByteArrayOutputStream();
   public void write(int i) throws IOException {
    baos.write(i);
 }
}

It's necessary to init() the servlet, something the container would normally do:

final MyServlet myServlet = new MyServlet();
myServlet.init(servletConfig);

Finally; I invoke my servlet, and check the output:

restCacheServlet.doGet(httpServletRequest, httpServletResponse);
final byte[] data = servletOutputStream.baos.toByteArray();
Assert.assertNotNull(data);
Assert.assertTrue(data.length > 0);

The mystery of uart0

The FDT file for Raspberry Pi, declares UART.  I've had quite a bit of trouble understanding how declaring a uart in the FDT results in /dev/tty0 and /dev/cuau0 appearing in /dev/

Here are the relevant lines from the FDT:

uart0: uart0 {
  compatible = "broadcom,bcm2835-uart", "broadcom,bcm2708-uart", "arm,pl011", "arm,primecell";
  reg = <0x201000 0x1000>;
  interrupts = <65>;
  interrupt-parent = <&intc>;

  clock-frequency = <3000000>;	/* Set by VideoCore */
  reg-shift = <2>;
};

Eventually this uart device gets bound to the sio driver, and exposed as /dev/cuau0 and /dev/tty0.

From the FDT declaration I can see that:

  • There are some declarations of the devices that this uart driver is compatible with
  • It uses irq 65, and the parent interrupt controller is "intc" which is declared in the same FDT
  • There are some registers declared

It's also interesting to note that the uart device is not a child of the gpio device, it's a child of axi, the bus.

There is a clue, in the kernel conf for Pi, here; it declares a uart device and pl011

device uart
device pl011

The source tree for the uart devices is at /sys/dev/uart ad pl011 is at /sys/dev/uart/uart_dev_pl011.c.

Therefore:

  • The kernel conf file specifies that uart and pl011 need to be compiled into the kernel
  • The FDT file declares that there is a uart and that it's compatible with pl011.

The file /sys/dev/uart/uart_bus_fdt.c, is the uart bus driver for devices with a FDT . When the kernel boots, it will call "probe" on this device.  The code below probes the FDT (via Open Firmware) and will detect that a pl011 is compatible.

static int
uart_fdt_probe(device_t dev)
{
  struct uart_softc *sc;
  phandle_t node;
  pcell_t clock, shift;
  int err;

  sc = device_get_softc(dev);
  if (ofw_bus_is_compatible(dev, "ns16550"))
    sc->sc_class = &uart_ns8250_class;
  else if (ofw_bus_is_compatible(dev, "lpc,uart"))
    sc->sc_class = &uart_lpc_class;
  else if (ofw_bus_is_compatible(dev, "fsl,imx-uart"))
    sc->sc_class = &uart_imx_class;
  else if (ofw_bus_is_compatible(dev, "arm,pl011"))
    sc->sc_class = &uart_pl011_class;
  else if (ofw_bus_is_compatible(dev, "cadence,uart"))
    sc->sc_class = &uart_cdnc_class;
  else
    return (ENXIO);
  node = ofw_bus_get_node(dev);

  if ((err = uart_fdt_get_clock(node, &clock)) != 0)
    return (err);
  uart_fdt_get_shift(node, &shift);

  return (uart_bus_probe(dev, (int)shift, (int)clock, 0, 0));
}

So, the UART bus device knows to bind the device that was specified in the FDT, because it matches this string "arm, pl011".

So, the mystery is solved this way:

  • The kernel conf ensures that the uart and pl011 devices are compiled in
  • The FDT declares the uart and declares that its compatible with "arm, pl011" which is a string that pl011 recognizes
  • The hardware configuration that the pl011 needs, it reads via Open Firmware, which in turn reads from the FDT blob

If you look at the datasheet for the BCM2835, you will see that there is indeed a Primecell pl011 UART on the chip.   On page 177 of the datasheet there is this:

The PL011 USRT is mapped on base adderss 0x7E20100.

This address mapping of "0x7e20100" matches what was declared int the FDT above.

reg = <0x201000 0x1000>;

The reason for the difference between 0x7E20100 and 0X201000 is the I/O mapping on the ARM chip, described on page 6 of the datasheet.