Time Machine backups using FreeBSD ZFS

In this blog article, I described a way to use Netatalk3 to do Time Machine backups on FreeBSD.  This approach worked well, but it had some problems:

  • The Time Machine backups are in every user's home dir.  That's messy and there is the potential that they'll accidentally delete the backup.
  • If I put the backups on a ZFS disk, I can compress them
  • I would like the potential to use ZFS snapshots down the road
  • I would prefer to have all the backups in one directory, rather than scattered across user profiles

So, here is a new, better recipe.  Note that this recipe will only work with Netatalk 3.1.2 or better.  The current FreeBSD port is version 3.1.3, so that helps.  Firstly, as in the other recipe, the first step is to install netatalk3, and nss_mdns

pkg install netatalk3
pkg install nss_mdns

avahi needs mDNS, so that needs to be configured in /etc/nsswitch.conf.  Ensure that this line exists:

hosts: files dns mdns

Next create a ZFS dataset for the backups.  my zpool is /storage, and the Time Machine backups will be in /storage/timemachine.  Notice that I've turned on compression.

zfs create storage/timemachine
zfs set compression=gzip storage/timemachine
zfs get all storage/timemachine
zfs list

I want to grant the ability to do time machine backups to a FreeBSD group, so I'll make that, and add the users.  This group will be referred to in afp.conf.

pw groupadd timemachine
pw groupmod timemachine -m tom
pw groupshow timemachine

After that, we need to create user directories, one for each time machine user.

mkdir storage/timemachine/tom
chown tom:timemachine /storage/timemachine/tom
chmod 700 /storage/timemachine/tom
chmod 777 /storage/timemachine

So, now that I have a compressed ZFS dataset to store the backups on, backup directories created, and a FreeBSD group created, I can create afp.conf.  I've chosen to limit Time Machine to 500GB space for each user.

;
; Netatalk 3.x configuration file
;

[Global]
vol preset = default_for_all_vol
log file = /var/log/netatalk.log
log level = default:warn 
hosts allow = 192.168.77.0/24
mimic model = TimeCapsule6,116

[default_for_all_vol]
file perm = 0640
directory perm = 0750
cnid scheme = dbd

[Homes]
basedir regex = /storage/home
#500 GB (units of MB)
vol size limit = 512000

[TimeMachine]
time machine = yes
path=/storage/timemachine/$u
valid users = @timemachine
#500 GB (units of MB)
vol size limit = 512000

You should notice that the path of the [TimeMachine] share is set to

path=/storage/timemachine/$u

This means that for each logged in user, $u is substituted wth the name of the user. So, when Time Machine logs in as "tom", the data is stored at /storage/timemachine/tom.

Only members of the group "timemachine" will be able to use Time Machine.

The rest of the steps are exactly as in the previous blog entry.

You'll need to start dbus, avahi, and netatalk, like this:

/usr/local/etc/rc.d/dbus onestart
/usr/local/etc/rc.d/avahi-daemon onestart
/usr/local/etc/rc.d/netatalk onestart

The next step takes place on your OS X client machine.  On each host that will perform backups, enable Time Machine to see non-TM volumes:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Then, mount your user's share using afp://.  This will make the share visible to TimeMachine.

After this, you should be able to see your Netatalk shares in Time Machine, and perform backups

TimeMachine backups on FreeBSD-10

I usually work on a Macbook, and I use keyless ssh with a cron job to do nightly backups.  It works very well.  However, I decided to try backups with Time Machine, using my FreeBSD-10 server as the host.

The first step is to install netatalk3, and nss_mdns

pkg install netatalk3
pkg install nss_mdns

avahi needs mDNS, so that needs to be configured in /etc/nsswitch.conf.  Ensure that this line exists:

hosts: files dns mdns

Finally, configure netatalk.  My configuration in /usr/local/etc/afp.conf looks like

[Global]
vol preset = default_for_all_vol
log file = /var/log/netatalk.log
hosts allow = 192.168.77.0/24
mimic model = TimeCapsule6,116

[default_for_all_vol]
file perm = 0640
directory perm = 0750
cnid scheme = dbd

[Homes]
basedir regex = /storage/home
time machine = yes

You'll need to start dbus, avahi, and netatalk, like this:

/usr/local/etc/rc.d/dbus onestart
/usr/local/etc/rc.d/avahi-daemon onestart
/usr/local/etc/rc.d/netatalk onestart

The next step takes place on your OS X client machine.  On each host that will perform backups, enable Time Machine to see non-TM volumes:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Then, mount your user's share using afp://.  This will make the share visible to TimeMachine.

After this, you should be able to see your Netatalk shares in Time Machine, and perform backups

Building U-boot 2014 on FreeBSD

I'm upgrading the U-boot version used for Wandboard by Crochet-BSD.    The first step is to upgrade the Device Tree Compiler (DTC); there are instructions here.

Once you have the U-boot 2014 source, the build is trivial:

gmake SED=gsed HOSTCC=cc CROSS_COMPILE=armv6-freebsd- wandboard_quad_config
gmake SED=gsed HOSTCC=cc CROSS_COMPILE=armv6-freebsd-

 

Building u-boot for Chromebook

I got interested in building Crochet images for Chromebook.  The first step, naturally is to build U-boot for Chromebook.

In order to build U-boot, you'll need to have already installed

  • gmake
  • gcc

You can get these from the ports tree.  You'll also need xdev, the FreeBSD cross-compiler toolkit.

I started with the latest U-boot 2014 source, which I got here.  The latest U-boot requires DTC 1.4.  I'm on FreeBSD 10, which has a much older DTC.

To build the latest DTC:

git clone git://git.kernel.org/pub/scm/utils/dtc/dtc.git
cd dtc
gmake CC=gcc
cp dtc /usr/bin

Once we have DTC 1.4, we can build U-boot.

wget ftp://ftp.denx.de/pub/u-boot/u-boot-2014.07.tar.bz2
tar zxvf u-boot-2014.07.tar.bz2
cd u-boot-2014.07
gmake SED=gsed HOSTCC=cc CROSS_COMPILE=armv6-freebsd- snow_config
gmake SED=gsed HOSTCC=cc CROSS_COMPILE=armv6-freebsd-

The U-boot binary image is "u-boot.bin". To burn the U-boot binary to a SD card:

sudo dd if=u-boot.bin of=/dev/rdisk1

The next challenge is the file system. The file system on the Chromebook looks like this:

       start        size    part  contents
           0           1          PMBR (Boot GUID: E780FED6-7991-B94B-B622-FA7541BE9FBB)
           1           1          Pri GPT header
           2          32          Pri GPT table
     8671232    21295104       1  Label: "STATE"
                                  Type: Linux data
                                  UUID: D5488177-EB19-3243-8D7C-D6E4F01DB5B3
       20480       32768       2  Label: "KERN-A"
                                  Type: ChromeOS kernel
                                  UUID: 47715F23-7DD6-5545-93DC-3807D1CD4AFD
                                  Attr: priority=0 tries=0 successful=0
     4476928     4194304       3  Label: "ROOT-A"
                                  Type: ChromeOS rootfs
                                  UUID: BE1064FF-EC28-E64B-B3A1-A5C176D57C0F
       53248       32768       4  Label: "KERN-B"
                                  Type: ChromeOS kernel
                                  UUID: 5D200D96-1F52-804C-83A7-AD8081E8ACD4
                                  Attr: priority=1 tries=0 successful=1
      282624     4194304       5  Label: "ROOT-B"
                                  Type: ChromeOS rootfs
                                  UUID: D0795A2A-1D0D-4140-9F57-F87AEF399C44
       16448           1       6  Label: "KERN-C"
                                  Type: ChromeOS kernel
                                  UUID: E8A703B8-E0A5-964D-9508-D88DE336D003
                                  Attr: priority=0 tries=15 successful=0
       16449           1       7  Label: "ROOT-C"
                                  Type: ChromeOS rootfs
                                  UUID: 7A947ABA-E723-8440-9A07-0EC394C85195
       86016       32768       8  Label: "OEM"
                                  Type: Linux data
                                  UUID: 173B3C97-CDCF-F849-99E7-7614C6C0F9FD
       16450           1       9  Label: "reserved"
                                  Type: ChromeOS reserved
                                  UUID: CD83D43A-4FC8-4849-B0DD-368E9C3819BA
       16451           1      10  Label: "reserved"
                                  Type: ChromeOS reserved
                                  UUID: 344D3149-A5C2-B44B-A65C-2DFD59758FC5
          64       16384      11  Label: "RWFW"
                                  Type: ChromeOS firmware
                                  UUID: D9B79232-94AD-A143-9C49-F947BFA2491E
      249856       32768      12  Label: "EFI-SYSTEM"
                                  Type: EFI System Partition
                                  UUID: E780FED6-7991-B94B-B622-FA7541BE9FBB
    30777311          32          Sec GPT table
    30777343           1          Sec GPT header

There is a good description of the Chromium OS file system here.  The size parameter is in sectors, which are usually 512 bytes each.  So, for example the state partition is 10GB in size.  KERN-A is 16MB in size, and ROOT-A is 2GB.

The partition ids for ChromeOS kernel and rootfs, are "fe3a2a5d-4f32-41a7-b725-accc3285a309" and "3cb8e202-3b7e-47dd-8a3c-7ff2a13cfcec" respectively.  The partition ids for ChromeOS are documented here.

 

Cross-compiling NetBSD on FreeBSD-10

I was curious to see how difficult it would be to compile NetBSD on FreeBSD 10.  Once you have the source, there is a build.sh script which is very helpful.  The build is as simple as:

./build.sh -m i386 tools
./build.sh -m i386 kernel=GENERIC release

In my case I have both clang and gcc installed.  I believe the cross-compile tools were built with clang.  The resulting gcc cross-compiler "i486--netbsdelf-gcc" was used to build the NetBSD source.

The generated binaries appear in obj/releasedir/i386/binary/

Enabling i2c on FreeBSD Wandboard

FreeBSD-11 has support for i2c on Wandboard.

The working boot image is here, and the boot log is here.

In order to enable it, there were three steps

  • Enable the kernel options
  • Update the files.imx6 file
  • Create the DTS mappings

The kerne config for Wandboard-Quad is here, however, most of the kernel options for Wandboard Quad are in the IMX6 config. The kernel options to enable are:

device fsliic # Freescale i2c/iic
device iic # iic protocol
device iicbus # iic bus

The files files.imx6 contains the mappings between the devices in the conf file, and the files to compile for that device.  In order to compile in the fsliic device for Freescale, I need to enable the mapping for it, so that /sys/arm/freescale/imx/i2c.c is compiled in.

arm/freescale/imx/i2c.c optional fsliic

Finally, I need to update the DTS files.  The file wandboard-quad.dts contains the device mappings for wandboard quad. However, similar to the kernel configuration, all the real mappings are in imx6.dtsi.  There are three i2c controllers on the wandboard quad, so I added these mappings:

i2c@021a0000 {

#address-cells = <1>;

#size-cells = <0>;

compatible = "fsl,imx-i2c";

reg = <0x021a0000 0x4000>;

interrupt-parent = <&gic>; interrupts = <68>;

};

i2c@021a4000 {

#address-cells = <1>;

#size-cells = <0>;

compatible = "fsl,imx-i2c";

reg = <0x021a4000 0x4000>;

interrupt-parent = <&gic>; interrupts = <69>;

};

i2c@021a8000 {

#address-cells = <1>;

#size-cells = <0>;

compatible = "fsl,imx-i2c";

reg = <0x021a8000 0x4000>;

interrupt-parent = <&gic>; interrupts = <70>;

};

The memory locations (0x21a000, 0x21a4000, 0x21a8000) as well as the interrupts (68,69,70) come from the IMX6 documentation.  This line

compatible = "fsl,imx-i2c";

is the line that the i2c.c driver uses in probe() to determine that this hardware is a compatible Freescale IMX i2c device.

Once the kernel is compiled and started it will detect the 3 iic devices.  The relevant lines from the boot log are:

iichb0: <I2C bus controller> mem 0x21a0000-0x21a3fff irq 68 on simplebus2
iicbus0: <OFW I2C bus> on iichb0
iic0: <I2C generic I/O> on iicbus0
iichb1: <I2C bus controller> mem 0x21a4000-0x21a7fff irq 69 on simplebus2
iicbus1: <OFW I2C bus> on iichb1
iic1: <I2C generic I/O> on iicbus1
iichb2: <I2C bus controller> mem 0x21a8000-0x21abfff irq 70 on simplebus2
iicbus2: <OFW I2C bus> on iichb2
iic2: <I2C generic I/O> on iicbus2

You should have 3 iic devices in the /dev/ directory

crw------- 1 root wheel 0x20 Jul 6 01:23 iic0
crw------- 1 root wheel 0x21 Jul 6 01:23 iic1
crw------- 1 root wheel 0x22 Jul 6 01:23 iic2

You can check which devices are specified in the dts file, using ofwdump.

root@wandboard:/dev # ofwdump -a
Node 0x38:
Node 0xa8: cpus
Node 0xd4: cpu@0
Node 0x190: aliases
Node 0x1bc: soc@00000000
Node 0x230: generic-interrupt-controller@00a00100
Node 0x2cc: mp_tmr0@00a00200
Node 0x348: l2-cache@00a02000
Node 0x3d0: aips@02000000
Node 0x458: ccm@020c4000
Node 0x4b4: anatop@020c8000
Node 0x520: timer@02098000
Node 0x594: gpio@0209c000
Node 0x668: gpio@020a0000
Node 0x71c: gpio@020a4000
Node 0x7f0: gpio@020a8000
Node 0x8a4: gpio@020ac000
Node 0x958: gpio@020b0000
Node 0xa0c: gpio@020b4000
Node 0xac0: serial@02020000
Node 0xb4c: serial@021e8000
Node 0xbdc: serial@021ec000
Node 0xc6c: serial@021f0000
Node 0xcfc: serial@021f4000
Node 0xd8c: usbphy@020c9000
Node 0xe2c: usbphy@020ca000
Node 0xed0: aips@02100000
Node 0xf58: ethernet@02188000
Node 0xfec: usb@02184000
Node 0x1088: usb@02184200
Node 0x1124: usb@02184400
Node 0x11b4: usb@02184600
Node 0x1244: usbmisc@02184800
Node 0x12c4: usdhc@02190000
Node 0x1368: usdhc@02194000
Node 0x1404: usdhc@02198000
Node 0x14a8: usdhc@0219c000
Node 0x1538: i2c@021a0000
Node 0x15d0: i2c@021a4000
Node 0x1668: i2c@021a8000
Node 0x1700: ocotp@021bc000
Node 0x1750: memory
Node 0x1774: chosen

You can also check that the iic modules are compiled into the kernel, using kldstat

root@wandboard:/usr/home/tom # kldstat -v | grep iic
131 i2c/iicbus
13 iichb/iicbus
12 iicbus/iic
55 iichb/ofw_iicbus
54 iicbb/ofw_iicbus

Once you have a device connected to the wandboard, you can use the i2c command to access the devices.