February 2014

Pragmatach-1.32 Released

The latest release of the Pragmatach Framework is version 1.32.  Changes in this release include: Bug fix in JMX registration.  Web contexts can now be reloaded by the container and the JMX beans are gracefully unregistered and re-registered Addition of the @Startup and @Shutdown annotations to allow methods to be run at container shutdown and startup.  An example is here. Addition of @BeforeInvoke and @AfterInvoke annotations which can be used to register methods which are run before and after controller method invocations.  Example is here. Null pointer violation when POSTing empty forms fixed Updated various libraries to the latest including org.apache.httpcomponents.httpclient org.apache.httpcomponents.httpmime org.hibernate.hibernate-core com.google.code.gson.gson org.apache.openjpa.openjpa org.thymeleaf..thymeleaf com.thoughtworks.xstream.xstream org.yaml.snakeyaml commons-beanutils.commons-beanutils commons-fileupload.commons-fileupload org.antlr.antlr commons-codec.commons-codec asm.asm-util Updated the default J2EE containers to the latest versions including jetty tomcat7 tomcat6 tomee jboss-as glassfish Updated the supported databases to the latest version including hsqldb derby h2 Added prototypical MongoDB support Switched from cglib to javaassist for proxy generation  

How does the Crochet-FreeBSD ARM boot work?

Introduction This is my 3rd blog posting on the topic of the Crochet-FreeBSD ARM boot process.  The other two are here and here.   At long last, I have the Crochet-FreeBSD build for Wandboard working properly, with U-boot and ubldr.  This article will serve, I hope, to document the entire process and give others a place to start in booting FreeBSD other embedded ARM devices.   If you want to follow along via the boot log, it’s here.  Bootable images are here. Booting FreeBSD on an ARM device has three primary steps: U-Boot ubldr FreeBSD kernel Diagrammatically, it looks like this: When the Wandboard starts, it loads a boot image from the mmc , at a known location on the disk.  In the case of Crochet-FreeBSD this image is U-boot.  After U-boot starts it loads and runs ubldr, which in turn loads the FreeBSD kernel and boots it. Disk Layout The Wandboard documentation here shows the basic requirements of the disk layout on the mmc card.  In particular: The MBR is at the start of the disk, and is about 0x200 bytes The U-boot boot loader is expected to be at offset 0x400 (1024) on the disk. In the case of the Crochet-FreeBSD image for Wandboard, I’ve chosen this layout MBR at the start of the disk, about 0x200 bytes long At offset 0x400, is my U-boot boot loader, as required by the Wandboard A FAT partition 50MB in size at offset 16384 (0x4000) on the disk.   This is the partition that the U-Boot configuration file, and ubldr will live on. A UFS partition which is the remainder of the disk.  This will be the FreeBSD root filesystem. For FAT32 partitions of less than 512MB size, the block size is 4KB.  So, 0x4000 blocks from the start of the disk is 64MB into the disk.  Given that U-boot is about a 300KB binary, we can be quite sure that the FAT partition will not overlap with U-Boot.  ubldr compiles to a 250KB binary and is stored on a 50MB partition; also plenty of space. The disk layout looks like this root@wandboard:~ # gpart show => 63 30678977 mmcsd0 MBR (15G) 63 16380 – free – (8.0M) 16443 102375 1 !12 [active] (50M) 118818 1881180 2 freebsd (919M) 1999998 28679042 – free – (14G) => 0 1881180 mmcsd0s2 BSD (919M) 0 1881180 1 freebsd-ufs (919M) U-Boot U-Boot is a very capable boot loader, that can boot a variety of architectures, of which one is ARM.   In the case of Wandboard, however a couple changes are needed to the configuration.   The patch files are here.  The primary requirements are: U-Boot needs to be configured to read ELF binaries U-Boot needs to be configured to include the U-Boot API, a feature which ubdlr requires. The Makefile needs to be changed, to include libc specifically When U-Boot starts, it looks for the file “uEnv.txt”‘ on the file system.  It’s very important that the first partition on the file system be FAT, since U-Boot doesn’t include UFS support.  The contents of uEnv.txt are instructions to U-Boot to load ubldr and start it.  Specifically: uenvcmd=fatload mmc 0:1 88000000 ubldr;bootelf 88000000; These U-Boot commands mean: From the FAT disk unit “mmc 0” on the 1st slice load “ubldr” into RAM location 0x88000000. Boot the ELF image at 0x88000000 From here, ubldr will start.  The reason we had compiled U-Boot with ELF support is that ubldr is an ELF binary, so we needed the U-Boot command “bootelf”. The choice of memory address 0x88000000 is mostly arbitrary.  According to the manuals, the RAM starts at address 0x100000, so this number has to be larger than 0x100000, smaller than the physical size of the RAM, and not conflict with the memory address that U-Boot was loaded at.  I suspect the Wandboard loaded U-Boot at 0x100000. ubldr ubldr is an ARM implementation of loader(8).  It’s not technically necessary to use ubldr; U-Boot could just boot the kernel directly, but having an implementation of loader(8) is quite useful.  For example, it provides a serial console for kernel debugging, and it allows passing flags to the kernel at startup time.   Some drivers, such as urtwn, for example require passing flags to the kernel to accept licensing terms for binary blobs. Since ubldr is not relocatable, it’s necessary to compile it with the memory address that it will be loaded at.  If you look here, you will notice that the address 0x8800000 was passed to the compile command. An important aspect of ubldr for Wandboard is that the FDT is compiled into the kernel.   ubldr can use external device blobs, or it can use device blobs that are compiled into the kernel.  If you look at the kernel config for Wandboard here, you can see that it specifies compiled-in device blobs.  This message from ubldr shows that it’s using a device tree blob (DTB) compiled into the kernel Booting [/boot/kernel/kernel]… Using DTB compiled into kernel. When ubldr starts, it’ll mount the UFS root file system, and read the file /boot/loader.conf.  In that file we can configure the boot loader, including passing kernel parameters, setting the serial console speed, configuring boot loader menus, etc.   It’s worth noting that ubldr configurations are coded in FORTH. Another aspect of ubldr which is important is that it uses the U-Boot API.  If you’re interested to know how that works, the ubldr source is here.  The Makefile pulls in U-Boot deps, and they are referred to in conf.c. On the topic of ubldr, there is an excellent blog post here that is worth reading. FreeBSD Kernel Finally, ubldr will boot the kernel.  The default location for the kernel on the root filesystem is /boot/kernel/kernel.   The file /etc/fstab should be configured to mount appropriate file systems when the kernel starts.  If you look at the kernel configuration for Wandboard here, you can see that the Kernel is configured to find the root file system on the first mmc device, on the second slice.   Specifically: # U-Boot stuff lives on slice 1, FreeBSD on slice 2. options ROOTDEVNAME=\”ufs:mmcsd0s2a\” Finally One trick that was used for Crochet-FreeBSD’s Beaglebone build was to configure /etc/fstab to mount the FAT partition to /boot/msdos.  The specific configuration is /dev/mmcsd0s1   /boot/msdos     msdosfs rw,noatime      0 0 This is interesting because uEnv.txt is on this partition. We can modify it if we want to try new configurations without rebuilding the image.  

Spelunking the Crochet-FreeBSD Wandboard boot

Some time ago I was part of the effort to add Wandboard support to Crochet-FreeBSD.   I wrote a blog posting about it here.   An explanation of how Crochet-FreeBSD boots the Wandboard is below: The Wandboard uses U-Boot as its initial boot-loader.  A number of patches to the U-Boot source are required, and you can view them here. Creating the Disk Image First, when Crochet builds the disk image for Wandboard, it first writes an MBR to the disk.  This is in setup.sh, in the method “wandboard_partition_image”. The second step is for Crochet to write the U-boot binary to the disk in the method “wandboard_uboot_install”.  The binary artefact it writes is “u-boot.imx”.  It writes directly to the disk, 1024 bytes into the disk, like this: sudo dd if=${WANDBOARD_UBOOT_SRC}/u-boot.imx of=/dev/${DISK_MD} bs=512 seek=2 At this point the image has U-boot at the 1024th byte of the disk.  When the Wandboard starts, it looks for a boot loader at disk offset 1024, according to this.  So, we now have a disk image that has U-boot at the right spot on the disk to boot it. The next step in building the image is adding a FAT partition to hold the FreeBSD kernel.  This is accomplished in setup.h using disk_fat_create 50m 32 16384 This creates a 50 MB FAT partition of type 32 at disk offset 16384.  16384 was chosen to ensure that the first partition on the disk starts well into the disk, and beyond the U-boot loader.  The FreeBSD kernel is written to this partition like this: cp ${FREEBSD_OBJDIR}/sys/${KERNCONF}/kernel . cp ${FREEBSD_OBJDIR}/sys/${KERNCONF}/kernel.bin . The final step in image creation is creating a UFS partition for the FreeBSD file system.  This is done with the method “disk_ufs_create”.  I’ve chosen to write both kernel and kernel.bin.   kernel.bin is straight-up executable binary code.  kernel is the ELF binary wrapped around kernel.bin.  It’s notable that one of the patches that was made to U-Boot was to enable it to load ELF binaries. Booting FreeBSD with U-Boot Once the image is created we have: Partitioned the disk as MBR Written U-Boot to the disk at offset 1024 (0x400). Written the Kernel to a FAT partition on the disk Written the FreeBSD file system to a UFS partition on the disk So, when the Wandboard starts, it’ll look at disk offset 0x400, and find U-boot.  It’ll start U-boot, and we’ll have to configure U-boot to find the FreeBSD kernel and start it.   This is done with uEnv.txt.  When U-boot starts it’ll look for eEnv.txt on the first partition in the system, which in this case is the FAT partition we wrote the kernel to.  Inside that file it’ll find: uenvcmd=fatload mmc 0:1 12000000 kernel.bin;go 12000000; These instructions tell U-Boot to load the file “kernel.bin” from the FAT partition at “mmc 0:1”.  The kernel will be loaded at memory address 12000000 and then control will be passed to 12000000.  The memory address 0x12000000 was specified in the kernel configuration for Freescale.  Check here. Once the kernel starts, it will look for the FreeBSD filesystem.  If you look at the kernel source here, you will find that the Wandboard kernel looks for its filesystem at “ufs:mmcsd0s2a\”.  That is “the UFS file system on the 2nd slice of the first mmc disk”.  Specifically: # U-Boot stuff lives on slice 1, FreeBSD on slice 2. options ROOTDEVNAME=\”ufs:mmcsd0s2a\” One important aspect of this boot strategy is the fact that it doesn’t use loader(8).  This means that any configuration that would have been set up in /boot/loader.conf doesn’t get set up.  A notable disadvantage is accepting the terms for the urtwn driver, and loading kernel modules.

FreeBSD-11 on Wandboard

Adding Wandboard support to Crochet-FreeBSD has been a project of mine for a while.  After this fascinating kernel fix, I can now build a working image and boot FreeBSD-Current.   Here is an example boot log, and here’s an image file. The output of dmesg, with a USB wireless adapter in the board, is: KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 r261638: Sat Feb 8 13:46:47 MST 2014 tom@bernice:/storage/home/tom/crochet/crochet-freebsd/work/obj/arm.armv6/storage/home/tom/crochet/src/FreeBSDHead/head/sys/WANDBOARD-QUAD arm FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 Preloaded elf kernel “kernel” at 0xc24e0c6c. CPU: Cortex A9-r2 rev 10 (Cortex-A core) Supported features: ARM_ISA THUMB2 JAZELLE THUMBEE ARMv4 Security_Ext WB disabled EABT branch prediction enabled LoUU:2 LoC:1 LoUIS:2 Cache level 1: 32KB/32B 4-way data cache WB Read-Alloc Write-Alloc 32KB/32B 4-way instruction cache Read-Alloc real memory = 2147483648 (2048 MB) Physical memory chunk(s): 0x10000000 – 0x11ffffff, 32768 KBytes (8192 pages) 0x12620000 – 0x8d77ffff, 2016640 KBytes (504160 pages) avail memory = 2095054848 (1998 MB) Static device mappings: 0x00a00000 – 0x00afffff mapped at VA 0xffe00000 0x02000000 – 0x020fffff mapped at VA 0xffd00000 0x02100000 – 0x021fffff mapped at VA 0xffc00000 null: <null device, zero device> openfirm: <Open Firmware control device> mem: <memory> Falling back to <Software, Yarrow> random adaptor random: <Software, Yarrow> initialized nfslock: pseudo-device ofwbus0: <Open Firmware Device Tree> ofwbus0: <cpus> type unknown (no driver attached) simplebus0: <Flattened device tree simple bus> on ofwbus0 gic0: <ARM Generic Interrupt Controller> mem 0xa01000-0xa01fff,0xa00100-0xa001ff on simplebus0 gic0: pn 0x390, arch 0x1, rev 0x2, implementer 0x43b sc->nirqs 160 l2cache0: <PL310 L2 cache controller> mem 0xa02000-0xa02fff irq 124 on simplebus0 l2cache0: Part number: 0x3, release: 0x7 l2cache0: L2 Cache: 1024KB/32B 16 ways l2cache0: L2 Cache enabled l2cache0: Early BRESP response: disabled l2cache0: Instruction prefetch: disabled l2cache0: Data prefetch: disabled l2cache0: Non-secure interrupt control: disabled l2cache0: Non-secure lockdown: disabled l2cache0: Share override: disabled l2cache0: Double linefil: disabled l2cache0: Instruction prefetch: disabled l2cache0: Data prefetch: disabled l2cache0: Double linefill on WRAP request: disabled l2cache0: Prefetch drop: disabled l2cache0: Incr double Linefill: disabled l2cache0: Not same ID on exclusive sequence: disabled l2cache0: Prefetch offset: 0 simplebus1: <Flattened device tree simple bus> mem 0x2000000-0x20fffff on simplebus0 ccm0: <Freescale i.MX6 Clock Control Module> mem 0x20c4000-0x20c7fff irq 119,120 on simplebus1 imx6_anatop0: <Freescale i.MX6 Analog PLLs and Power> mem 0x20c8000-0x20c8fff on simplebus1 imx_gpt0: <Freescale i.MX GPT timer> mem 0x2098000-0x209bfff irq 87 on simplebus1 imx_gpt0: Running on 11000KHz clock, base freq 66000000Hz CR=0x0000027d, PR=0x00000005 Event timer “i.MXxxx GPT Eventtimer” frequency 11000000 Hz quality 1000 Timecounter “i.MX GPT Timecounter” frequency 11000000 Hz quality 1000 uart0: <imx_uart> mem 0x2020000-0x2023fff irq 58 on simplebus1 uart0: console (115200,n,8,1) uart0: fast interrupt simplebus1: <serial@021e8000> mem 0x21e8000-0x21ebfff irq 59 type unknown (no driver attached) simplebus1: <serial@021ec000> mem 0x21ec000-0x21effff irq 60 type unknown (no driver attached) simplebus1: <serial@021f0000> mem 0x21f0000-0x21f3fff irq 61 type unknown (no driver attached) simplebus1: <serial@021f4000> mem 0x21f4000-0x21f7fff irq 62 type unknown (no driver attached) usbphy0: <Freescale i.MX6 USB PHY> mem 0x20c9000-0x20c9fff irq 44 on simplebus1 usbphy1: <Freescale i.MX6 USB PHY> mem 0x20ca000-0x20cafff irq 45 on simplebus1 simplebus2: <Flattened device tree simple bus> mem 0x2100000-0x21fffff on simplebus0 ffec0: <Freescale Gigabit Ethernet Controller> mem 0x2188000-0x218bfff irq 150,151 on simplebus2 ffec0: MAC address 00:1f:7b:b4:06:7f: ffec0: PHY preamble disabled miibus0: <MII bus> on ffec0 atphy0: <Atheros F1 10/100/1000 PHY> PHY 1 on miibus0 atphy0: OUI 0x00c82e, model 0x0007, rev. 4 atphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto ffec0: bpf attached ffec0: Ethernet address: 00:1f:7b:b4:06:7f ehci0: <Freescale i.MX integrated USB controller> mem 0x2184000-0x21841ff irq 75 on simplebus2 ehci0: [GIANT-LOCKED] usbus0: EHCI version 1.0 usbus0 on ehci0 ehci0: usbpf: Attached ehci1: <Freescale i.MX integrated USB controller> mem 0x2184200-0x21843ff irq 72 on simplebus2 ehci1: [GIANT-LOCKED] usbus1: EHCI version 1.0 usbus1 on ehci1 ehci1: usbpf: Attached simplebus2: <usb@02184400> mem 0x2184400-0x21845ff irq 73 type unknown (no driver attached) simplebus2: <usb@02184600> mem 0x2184600-0x21847ff irq 74 type unknown (no driver attached) simplebus2: <usbmisc@02184800> mem 0x2184800-0x21849ff type unknown (no driver attached) simplebus2: <usdhc@02190000> mem 0x2190000-0x2193fff irq 54 type unknown (no driver attached) sdhci_imx0: <Freescale uSDHC controller> mem 0x2194000-0x2197fff irq 55 on simplebus2 sdhci_imx0-slot0: 200MHz HS 4bits 3.3V 3.0V PIO sdhci_imx0-slot0: ============== REGISTER DUMP ============== sdhci_imx0-slot0: Sys addr: 0x00000000 | Version: 0x00000002 sdhci_imx0-slot0: Blk size: 0x00000000 | Blk cnt: 0x00000001 sdhci_imx0-slot0: Argument: 0x00000000 | Trn mode: 0x00000000 sdhci_imx0-slot0: Present: 0xff888088 | Host ctl: 0x00000000 sdhci_imx0-slot0: Power: 0x0000000d | Blk gap: 0x00000080 sdhci_imx0-slot0: Wake-up: 0x00000008 | Clock: 0x00000002 sdhci_imx0-slot0: Timeout: 0x00000080 | Int stat: 0x00000000 sdhci_imx0-slot0: Int enab: 0x017f00fb | Sig enab: 0x017f00fb sdhci_imx0-slot0: AC12 err: 0x00000000 | Slot int: 0x00000000 sdhci_imx0-slot0: Caps: 0x0377c800 | Max curr: 0x80000000 sdhci_imx0-slot0: =========================================== sdhci_imx1: <Freescale uSDHC controller> mem 0x2198000-0x219bfff irq 56 on simplebus2 sdhci_imx1-slot0: 200MHz HS 4bits 3.3V 3.0V PIO sdhci_imx1-slot0: ============== REGISTER DUMP ============== sdhci_imx1-slot0: Sys addr: 0x00000000 | Version: 0x00000002 sdhci_imx1-slot0: Blk size: 0x00000200 | Blk cnt: 0x00000001 sdhci_imx1-slot0: Argument: 0x000095df | Trn mode: 0x00000011 sdhci_imx1-slot0: Present: 0xff8d8088 | Host ctl: 0x00000002 sdhci_imx1-slot0: Power: 0x0000000d | Blk gap: 0x00000080 sdhci_imx1-slot0: Wake-up: 0x00000000 | Clock: 0x00000002 sdhci_imx1-slot0: Timeout: 0x0000008b | Int stat: 0x00000000 sdhci_imx1-slot0: Int enab: 0x017f00fb | Sig enab: 0x017f00fb sdhci_imx1-slot0: AC12 err: 0x00000000 | Slot int: 0x00000000 sdhci_imx1-slot0: Caps: 0x0377c800 | Max curr: 0x80000011 sdhci_imx1-slot0: =========================================== mmc0: <MMC/SD bus> on sdhci_imx1 simplebus2: <usdhc@0219c000> mem 0x219c000-0x219ffff irq 57 type unknown (no driver attached) Timecounters tick every 4.000 msec vlan: initialized, using hash tables with chaining tcp_init: net.inet.tcp.tcbhashsize auto tuned to 32768 lo0: bpf attached sdhci_imx1-slot0: Divider 250 for freq 400000 (max 200000000) mmc0: Probing bus usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 mmc0: SD 2.0 interface conditions: OK mmc0: SD probe: OK (OCR: 0x00ff8000) mmc0: Current OCR: 0x00ff8000 ugen0.1: <Freescale> at usbus0 uhub0: <Freescale EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 ugen1.1: <Freescale> at usbus1 uhub1: <Freescale EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 mmc0: Probing cards mmc0: New card detected (CID 02544d534131364710279259a100bc00) mmc0: New card detected (CSD 400e00325b59000075077f800a400000) mmc0: Card at relative address 0x1234 added: mmc0: card: SDHC SA16G 1.0 SN 663902625 MFG 12/2011 by 2 TM mmc0: bus: 4bit, 50MHz, high speed timing mmc0: memory: 30679040 blocks, erase sector 8192 blocks mmc0: setting transfer rate to 50.000MHz (high speed timing) sdhci_imx1-slot0: Divider 2 for freq 50000000 (max 200000000) mmcsd0: 16GB <SDHC SA16G 1.0 SN 663902625 MFG 12/2011 by 2 TM> at mmc0 50.0MHz/4bit/65535-block GEOM: new disk mmcsd0 mmc0: setting bus width to 4 bits GEOM_PART: partition 1 is not aligned on 4194304 bytes GEOM_PART: partition 2 is not aligned on 4194304 bytes GEOM_PART: partition 1 is not aligned on 4194304 bytes GEOM_PART: partition 1 is not aligned on 4194304 bytes random: unblocking device. uhub0: 1 port with 1 removable, self powered uhub1: 1 port with 1 removable, self powered Spurious interrupt detected [0x000003ff] ugen1.2: <vendor 0x0bda> at usbus1 Trying to mount root from ufs:mmcsd0s2a []… WARNING: / was not properly dismounted WARNING: /: mount pending error: blocks 0 files 5 warning: no time-of-day clock registered, system time will not be set accurately start_init: trying /sbin/init GEOM_PART: partition 1 is not aligned on 4194304 bytes wlan: <802.11 Link Layer> urtwn0: <vendor 0x0bda product 0x8176, class 0/0, rev 2.00/2.00, addr 2> on usbus1 urtwn0: MAC/BB RTL8188CUS, RF 6052 1T1R urtwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps urtwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps