Greetings!
As the topic suggests we’re going to play with Xen and set up a FreeBSD DomU inside a Linux Dom0.

First some important information:

Background

We’re going to be running a PV (para-virtualised) installation of 32-bit FreeBSD. Unfortunately my older hardware doesn’t support the HVM (hardware virtual machine) CPU extensions, and at the time of writing the stability of 64-bit FreeBSD as a DomU is not certain.

Credit

A lot of credit for this goes to the following posts. I was able to piece together bits of what I needed to make this process work from all of the hard work put in by these guys 🙂
http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/GrowFS
http://forums.freebsd.org/showthread.php?t=10268
http://tuxhelp.org/debian/xen/xen_server_configuration#preparing_a_pv_domu_for_a_posix_os_eg_freebsd

Pre-requisites

You will need an existing FreeBSD install to start with. This is needed to build the XEN kernel for FreeBSD, and the template disk image from which we’ll make our final OS. The default kernel does not contain the drivers needed to run as a Xen guest. This is very easy to solve: download VirtualBox and the latest FreeBSD DVD ISO. Do a basic install with the kernel and OS sources.

You’ll also need an existing Xen Dom0 installation with LVM partitioning. I’m using Debian 5 (Lenny) as my Xen host with Xen 3.2. It seems to be working quite well. There are many good instructions online on how to set this up.

Setting up the whole world

Log in to your VirtualBox (or other) FreeBSD installation.
We’re going to create a disk image, compile the OS and then put the files onto the image.
root# cd /usr/src
/usr/src# truncate -s 1G /tmp/freebsd.img
/usr/src# mdconfig -f freebsd.img

This will tell you which md device was set up. We’ll assume here it was md0.
/usr/src# fdisk -BI /dev/md0
/usr/src# bsdlabel -wB md0s1
/usr/src# newfs -U md0s1a
/usr/src# mount /dev/md0s1a /mnt
/usr/src# make buildworld && \
make buildkernel KERNCONF=XEN
/usr/src# make DESTDIR=/mnt installworld && \
make DESTDIR=/mnt installkernel KERNCONF=XEN && \
make DESTDIR=/mnt distribution

Edit /mnt/etc/ttys and add this line:
xc0 "/usr/libexec/getty Pc" vt100 on secure

Edit /mnt/etc/fstab and add this line:
/dev/ad0s1a / ufs rw 0 0

Unmount the image, compress it, and copy it to your destination:
# umount /mnt
# mdconfig -d -u 0
# bzip2 -v9 /tmp/freebsd.img
# scp /tmp/freebsd.img.bz2 \
[email protected]:/tmp/freebsd.img.bz2

You will also need to copy the kernel image to your Dom0 host. Xen will use this to boot your upcoming installation:
scp /usr/obj/usr/srcsys/XEN/kernel \
[email protected]:/tmp/freebsd-kernel

Configure your Dom0

There are quite a few (simple) steps required here. We’re going to set up your LVM partition, install two copies of the FreeBSD OS:

1. A maintenance VM which is used to resize your filesystems
2. Your main FreeBSD VM

Both of these are created from the VM template image you created in the previous step.

Create your LVM partitions

My LVM physical group is named xen-vol. Replace this with the name of your PG. The first partition will be for the the maintenance VM, the second will be for the main DomU:
root@dom0# lvcreate -L1000 -n freebsdmaint.example.com xen-vol
root@dom0# lvcreate -L110000 -n freebsd.example.com xen-vol

Now copy the template into the partitions:
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/freebsdmaint.example.com \
bs=1M
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/dom0-host.example.com \
bs=1M

Configure it up

Here are two example configurations. The first one is for the maintenance VM, the second is for the main FreeBSD VM. Edit them to have the right file paths. You’ll notice they both have a kernel parameter. This points to the kernel file you copied over earlier in this process. Make sure that file is somewhere safe – it will be required to boot your VMs. I keep mine under /xen. I also found that, as of FreeBSD 8.2-RC1, booting with 2 virtual CPUs fails, so I set this to 1:

freebsdmaint.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsdmaint.example.com,hda,w', 'phy:/dev/xen-vol/freebsd-dom0.example.com,hdb,w' ]
name = 'freebsdmaint.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

freebsd-dom0.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsd-dom0.example.com,hda,w' ]
name = 'freebsd-dom0.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

Now start up your maintenance VM:
xm create -c freebsdmaint.example.conf.cfg

The process of resizing

Now let’s start with the process of resizing. We’ll be resizing three things on our main freebsd partition:

• The partition (with fdisk)
• The slice (with bsdlabel)
• The filesystem (with growfs)

The partition for the main FreeBSD instance will be exposed here as /dev/ad1s1a

Resize partition in fdisk

Start by editing the disk in fdisk like this:
# fdisk -u /dev/ad1
******* Working on device /dev/ad1 *******
parameters extracted from in-core disklabel are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Do you want to change our idea of what BIOS thinks ? [n] n
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 2088387 (1019 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 129/ head 254/ sector 63
Do you want to change it? [n] y


Answer ‘y’ to the first two questions. You will then be asked:
Supply a decimal value for "size" [2088387]

For me, the correct answer here was calculated as:
Size in MB * 2048

It’s likely you will get a warning such as this:
fdisk: WARNING: partition does not end on a cylinder boundary
fdisk: WARNING: this may confuse the BIOS or some operating systems
Correct this automatically? [n] y
Choose ‘y’, then choose the default answers to everything, and write out the changes:
Should we write new partition table? [n] y

Resize the slice in bsdlabel

Fire up bsdlabel with:
# bsdlabel -e /dev/ad1s1

Your editor will pop up and look like this:
# size offset fstype [fsize bsize bps/cpg]
a: 2088351 16 unused 0 0
c: 2088387 0 unused 0 0 # "raw" part, don't edit

Ignore the warning to not edit the “raw” part – I and others have received errors when you grow the slice but not the raw part. The warning likely refers to shrinking the a: partition.

There are two important things to note here:

1. The size of the raw part will be the same as the size you set the partition to in the previous fdisk step.
2. The size of the a: partition will also be the same as the size of the partition in the previous step MINUS 16.

My updated label looks like:
# size offset fstype [fsize bsize bps/cpg]
a: 225279416 16 unused 0 0
c: 225279432 0 unused 0 0 # "raw" part, don't edit

Let’s growfs this thing!

You’ve made it this far, we’re into the home stretch!
Let’s use growfs(8) to increase the size of our filesystem:
# growfs /dev/ad1s1a
We strongly recommend you to make a backup before growing the Filesystem
Did you backup your data (Yes/No) ? Yes
[ lots of superblock numbers outputted ]

Let it all up and running

Now you can shutdown and maintenance VM, and start up your FreeBSD instance.

WARNING! ACHTUNG!
It is very important that you do not run both VMs at the same time in their current state – both will try to mount the filesystem read-write! Xen may give you a warning if you try and do this, but you certainly should not depend on it doing so (and indeed, it may not). The safest thing to do, once you shut down the maintenance VM, is to edit its configuration and remove the mention of the main VM’s disk.

That’s all! I hope people find this useful. Most of the credit for this rightly goes to Ray White (University of Waterloo), Adrian Chadd (developer) and Aprogas (FreeBSD user). I just put it all together in a way that worked for me 🙂