Geeky

The road to DevOps.

Creating an LVM-backed FreeBSD DomU in a Linux Dom0

Greetings!
As the topic suggests we’re going to play with Xen and set up a FreeBSD DomU inside a Linux Dom0.

First some important information:

Background

We’re going to be running a PV (para-virtualised) installation of 32-bit FreeBSD. Unfortunately my older hardware doesn’t support the HVM (hardware virtual machine) CPU extensions, and at the time of writing the stability of 64-bit FreeBSD as a DomU is not certain.

Credit

A lot of credit for this goes to the following posts. I was able to piece together bits of what I needed to make this process work from all of the hard work put in by these guys ๐Ÿ™‚
http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/GrowFS
http://forums.freebsd.org/showthread.php?t=10268
http://tuxhelp.org/debian/xen/xen_server_configuration#preparing_a_pv_domu_for_a_posix_os_eg_freebsd

Pre-requisites

You will need an existing FreeBSD install to start with. This is needed to build the XEN kernel for FreeBSD, and the template disk image from which we’ll make our final OS. The default kernel does not contain the drivers needed to run as a Xen guest. This is very easy to solve: download VirtualBox and the latest FreeBSD DVD ISO. Do a basic install with the kernel and OS sources.

You’ll also need an existing Xen Dom0 installation with LVM partitioning. I’m using Debian 5 (Lenny) as my Xen host with Xen 3.2. It seems to be working quite well. There are many good instructions online on how to set this up.

Setting up the whole world

Log in to your VirtualBox (or other) FreeBSD installation.
We’re going to create a disk image, compile the OS and then put the files onto the image.
root# cd /usr/src
/usr/src# truncate -s 1G /tmp/freebsd.img
/usr/src# mdconfig -f freebsd.img

This will tell you which md device was set up. We’ll assume here it was md0.
/usr/src# fdisk -BI /dev/md0
/usr/src# bsdlabel -wB md0s1
/usr/src# newfs -U md0s1a
/usr/src# mount /dev/md0s1a /mnt
/usr/src# make buildworld && \
make buildkernel KERNCONF=XEN
/usr/src# make DESTDIR=/mnt installworld && \
make DESTDIR=/mnt installkernel KERNCONF=XEN && \
make DESTDIR=/mnt distribution

Edit /mnt/etc/ttys and add this line:
xc0 "/usr/libexec/getty Pc" vt100 on secure

Edit /mnt/etc/fstab and add this line:
/dev/ad0s1a / ufs rw 0 0

Unmount the image, compress it, and copy it to your destination:
# umount /mnt
# mdconfig -d -u 0
# bzip2 -v9 /tmp/freebsd.img
# scp /tmp/freebsd.img.bz2 \
[email protected]:/tmp/freebsd.img.bz2

You will also need to copy the kernel image to your Dom0 host. Xen will use this to boot your upcoming installation:
scp /usr/obj/usr/srcsys/XEN/kernel \
[email protected]:/tmp/freebsd-kernel

Configure your Dom0

There are quite a few (simple) steps required here. We’re going to set up your LVM partition, install two copies of the FreeBSD OS:

  1. A maintenance VM which is used to resize your filesystems
  2. Your main FreeBSD VM

Both of these are created from the VM template image you created in the previous step.

Create your LVM partitions

My LVM physical group is named xen-vol. Replace this with the name of your PG. The first partition will be for the the maintenance VM, the second will be for the main DomU:
root@dom0# lvcreate -L1000 -n freebsdmaint.example.com xen-vol
root@dom0# lvcreate -L110000 -n freebsd.example.com xen-vol

Now copy the template into the partitions:
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/freebsdmaint.example.com \
bs=1M
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/dom0-host.example.com \
bs=1M

Configure it up

Here are two example configurations. The first one is for the maintenance VM, the second is for the main FreeBSD VM. Edit them to have the right file paths. You’ll notice they both have a kernel parameter. This points to the kernel file you copied over earlier in this process. Make sure that file is somewhere safe – it will be required to boot your VMs. I keep mine under /xen. I also found that, as of FreeBSD 8.2-RC1, booting with 2 virtual CPUs fails, so I set this to 1:

freebsdmaint.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsdmaint.example.com,hda,w', 'phy:/dev/xen-vol/freebsd-dom0.example.com,hdb,w' ]
name = 'freebsdmaint.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

freebsd-dom0.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsd-dom0.example.com,hda,w' ]
name = 'freebsd-dom0.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

Now start up your maintenance VM:
xm create -c freebsdmaint.example.conf.cfg

The process of resizing

Now let’s start with the process of resizing. We’ll be resizing three things on our main freebsd partition:

  • The partition (with fdisk)
  • The slice (with bsdlabel)
  • The filesystem (with growfs)

The partition for the main FreeBSD instance will be exposed here as /dev/ad1s1a

Resize partition in fdisk

Start by editing the disk in fdisk like this:
# fdisk -u /dev/ad1
******* Working on device /dev/ad1 *******
parameters extracted from in-core disklabel are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Do you want to change our idea of what BIOS thinks ? [n] n
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 2088387 (1019 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 129/ head 254/ sector 63
Do you want to change it? [n] y

Answer ‘y’ to the first two questions. You will then be asked:
Supply a decimal value for "size" [2088387]

For me, the correct answer here was calculated as:
Size in MB * 2048

It’s likely you will get a warning such as this:
fdisk: WARNING: partition does not end on a cylinder boundary
fdisk: WARNING: this may confuse the BIOS or some operating systems
Correct this automatically? [n] y

Choose ‘y’, then choose the default answers to everything, and write out the changes:
Should we write new partition table? [n] y

Resize the slice in bsdlabel

Fire up bsdlabel with:
# bsdlabel -e /dev/ad1s1

Your editor will pop up and look like this:
# size offset fstype [fsize bsize bps/cpg]
a: 2088351 16 unused 0 0
c: 2088387 0 unused 0 0 # "raw" part, don't edit

Ignore the warning to not edit the “raw” part – I and others have received errors when you grow the slice but not the raw part. The warning likely refers to shrinking the a: partition.

There are two important things to note here:

  1. The size of the raw part will be the same as the size you set the partition to in the previous fdisk step.
  2. The size of the a: partition will also be the same as the size of the partition in the previous step MINUS 16.

My updated label looks like:
# size offset fstype [fsize bsize bps/cpg]
a: 225279416 16 unused 0 0
c: 225279432 0 unused 0 0 # "raw" part, don't edit

Let’s growfs this thing!

You’ve made it this far, we’re into the home stretch!
Let’s use growfs(8) to increase the size of our filesystem:
# growfs /dev/ad1s1a
We strongly recommend you to make a backup before growing the Filesystem
Did you backup your data (Yes/No) ? Yes
[ lots of superblock numbers outputted ]

Let it all up and running

Now you can shutdown and maintenance VM, and start up your FreeBSD instance.

WARNING! ACHTUNG!
It is very important that you do not run both VMs at the same time in their current state – both will try to mount the filesystem read-write! Xen may give you a warning if you try and do this, but you certainly should not depend on it doing so (and indeed, it may not). The safest thing to do, once you shut down the maintenance VM, is to edit its configuration and remove the mention of the main VM’s disk.

That’s all! I hope people find this useful. Most of the credit for this rightly goes to Ray White (University of Waterloo), Adrian Chadd (developer) and Aprogas (FreeBSD user). I just put it all together in a way that worked for me ๐Ÿ™‚

Tags:

Comments

17 responses to “Creating an LVM-backed FreeBSD DomU in a Linux Dom0”

  1. […] This post was mentioned on Twitter by avleen, avleen. avleen said: Just wrote a huge step-by-step post on setting up a FreeBSD DomU in a Linux Dom0 with LVM partitions ๐Ÿ™‚ http://bit.ly/dVhfFe Try it out! […]

  2. haad Avatar

    Hi,

    Nice article, have you tried to use NetBSD dom0 ? NetBSD in -current have LVM same as linux has, with BSD system style and stability.

    NetBSD uses port of linux LVM tools on top of BSD licensed device-mapper driver. More you can find here [1] and [2].

    [1] http://netbsd.org/docs/guide/en/chap-lvm.html
    [2] http://netbsd.org/ports/xen/howto.html

    1. Av Avatar

      Hi Haad! Thanks for the reply ๐Ÿ™‚

      I haven’t looked at NetBSD’s options in the last ~2 years, they’re on my list to revisit. I’ll definitely take a look!
      My goal was to promote BSD as a reasonable alternative in the VM space, and show that you can easily get a working system. If we can use NetBSD as the Host OS that is even better as it covers both sides of the equation!

  3. haad Avatar

    You can use NetBSD as host option too but I’m not sure if you are interested in that ๐Ÿ™‚

    1. Av Avatar

      Absolutely! I’ll check it out ๐Ÿ˜€

  4. Ricardo J. Barberis Avatar
    Ricardo J. Barberis

    Nice article, one nitpick:

    “You will also need to copy the kernel image to your Dom0 host. Xen will use this to boot your upcoming installation:
    scp /usr/obj/usr/srcsys/XEN/kernel \
    [email protected]:/tmp/freebsd.img.bz2″

    Please, change the destination file in this scp before someone overwrites his/her freebsd.img ๐Ÿ™‚

    Cheers.

    1. Av Avatar

      Nice catch! Thanks Ricardo! I’ve updated it ๐Ÿ™‚

  5. […] the topic suggests we’re going to play with Xen and set up a FreeBSD DomU inside a Linux Dom0. Hello there! If you are new here, you might want to subscribe to the RSS feed for updates on this […]

  6. Martin Avatar
    Martin

    Many thanks for this blog post, I’m trying to work through it but I cannot locate “/usr/obj/usr/srcsys/XEN/kernel”.

    Can you offer any suggestions?

    1. Martin Avatar
      Martin

      Figured it out.

      I re-ran “buildkernel KERNCONF=XEN”

  7. […] this article: Creating an LVM-backed FreeBSD DomU in a Linux Dom0 ร‚ยซ Geeky This entry was posted in Uncategorized and tagged dom0, filesystems, install-two, lvm, […]

  8. Robert Avatar
    Robert

    make distribution DESTDIR=/mnt doesn’t succeed with releng_9, mergemaster -iF -D /mnt does the trick ๐Ÿ˜‰ Not sure if the first is just me or my fault…

    1. Av Avatar

      Oh it’s all probably changed! I’ll have to build a 9 box soon as test this out!

  9. nico Avatar
    nico

    Nice guide, one question though : How do you upgrade your freebsd install ? Let’s say in a couple of mounth you want to go from your freebsd9.0 domU to a freebsd9.1 domU ?

    1. Av Avatar

      A great question! Unfortunately I haven’t tried it yet so I have no idea ๐Ÿ™‚

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.