Warning: I recently went through this process, and took notes, but a few pieces here are from memory. If things don’t work as expected, please let me know and I’d be happy to help.

Background

I’ve been running Debian for a long time – probably steadily about 6 years now. Before that I was a BSD fan, so many things about Debian appealed to me.

Over time my needs have changed. Much of my work involves using CentOS, and I quite like the ease of managing it. Debian is still nice, but it always had a few things I wasn’t a fan of, such as split (conf.d style) configurations. No disrespect to Debian, it’s a solid distro, the style of distribution I’m drifting to is purely personal preference.

My existing systems are all Debian VMs which live in a Debian Xen Dom0.

Objectives

Given our Debian Dom0, we want to install a CentOS 6 DomU.
The DomU must have XFS on the filesystem, and load it’s own recent kernel rather than the one supplied by the Debian Dom0.

Problems and solutions

• Installing a CentOS (or RedHat, or Fedora) client is non-trivial. Debian has debootstrap which bootstraps the initial OS on the filesystem. We will use mock for this.
• We need a newer kernel. CentOS 6 doesn’t come with Xen enabled kernels since RedHat pulled Xen support from their kernels, so we need a 3rd party kernel, or to build our own.
• We need to load the new kernel. In order to do this we need to use pygrub which comes with Xen.
• pygrub doesn’t understand reading XFS filesystems, so we need a separate /boot partition which it can read the grub menu.

The right tools for the job

To make this work, we need the following tools:

• pygrub (comes with Xen)
• mock (apt-get install mock)

Xen and LVM and filesystems

This particular setup uses LVM to create filesystems for each virtual machine. Many moons ago I found this to be more performant and cause fewer issues than file-backed filesystems. However that isn’t the case these days. My examples may show LVM partitions being used but this isn’t necessary.

Additionally, I wanted to start using XFS on the new system. After 6 hours of trying to make it work, I realised that pygrub doesn’t yet support reading XFS filesystems.
If you want to use XFS on your root filesystem, you’ll need an ext2 or ext3 /boot partition which pygrub will boot from.

Instructions

Basic VM initialisation

Create two filesystems which will be used by the new VM. My LVM volume group is called xen-vol. Yours will probably be called something else. If you want to use file-backed filesystems, you can do that instead at this point. Here’s the LVM way:

$ lvcreate -n newvm-boot -L 200M xen-vol
$ lvcreate -n newvm-root -L 100G xen-vol
mkfs.ext2 /dev/xen-vol/newvm-boot
mkfs.xfs /dev/xen-vol/newvm-root

Configuring Mock

mock is the tool we use to create the bare VM. Normally it is used to create a chroot environment for tasks like building packages. However it works really well for creating a basic OS layout which can then be booted.

By default it doesn’t come with a configuration file for CentOS 6, so create the file /etc/mock/centos-6-i386.cfg and populate it with this:

[base]
name=base
baseurl=http://mirror.centos.org/centos/6/os/i386/
gpgcheck=0

[update]
name=updates
baseurl=http://mirror.centos.org/centos/6/updates/i386/
gpgcheck=0

[buildsys]
name=buildsys
baseurl=http://dev.centos.org/centos/buildsys/6/
gpgcheck=0

"""

config_opts['macros']['%dist']=".el6.centos"
config_opts['macros']['%centos_ver']="6"
config_opts['macros']['%rhel']="6"

Populating the VM

Now we’ll mount and populate the VM using mock.

mkdir -p /mnt/newvm/boot
mount /dev/xen-vol/newvm-root /mnt/newvm
mount /dev/xen-vol/newvm-boot /mnt/newvm/boot
mock --init -r centos-6-i386 /mnt/newvm

mock will now set up a mostly complete chroot.

Configure the VM

For each of the following steps, you will need to be chrooted into /mnt/newvm first:

chroot /mnt/newvm /bin/bash

Install and configure Grub

Grub is the bootloader we use, so install that:

yum install -y grub

Now you’ll need a /boot/grub/grub.conf file. Create it and make it look like this:

default=0
timeout=2

title CentOS 6.3 i686
  root (hd1,0)
  kernel /vmlinuz-3.6.7-1.el6xen.i686 ro root=/dev/xvdb crashkernel=auto noquiet rdshell
  initrd /initramfs-3.6.7-1.el6xen.i686.img

Now make a new /boot/grub/device.map file which looks like this:

(hd0)   /dev/xvda
(hd1)   /dev/xvdb

The astute readers will notice that we just told Grub to boot from a kernel which doesn’t exist! Let’s do that next.

Install the new kernel

One of the beautiful things about Linux is the way people love to help each other. In our case, someone has built the kernel we need to run CentOS 6 as a Xen DomU client. To install it do:

yum install http://au1.mirror.crc.id.au/repo/kernel-xen-release-6-4.noarch.rpm
yum install kernel-xen


Final configurations

You’ll need an /etc/fstab file, like this:

/dev/xvda           /boot               ext3    noatime         1 1
/dev/xvdb           /                   xfs     noatime         1 1
tmpfs               /dev/shm            tmpfs   defaults        0 0
devpts              /dev/pts            devpts  gid=5,mode=620  0 0
sysfs               /sys                sysfs   defaults        0 0
proc                /proc               proc    defaults        0 0

And set the root password:

$ passwd root

Xen configuration

Leave the chroot and return to the Dom0 shell.
You will need a Xen configuration file for this VM, which should look something like this:

bootloader = '/usr/bin/pygrub'
extra      = 'console=hvc0 xencons=tty'
memory     = '256'
vcpus      = 2
name       = "newvm.example.com"
hostname   = "newvm.example.com"
vif        = [ 'ip=<ip address>' ]
netmask    = '<netmask>'
gateway    = '<gateway address>'
ip         = '<ip address>'
broadcast  = '<broadcast address>'
disk       = [ 'phy:xen-vol/newvm-boot,xvda,w', 'phy:xen-vol/newvm,xvdb,w' ]

Boot the VM

Umount the partitions andstart the new VM:

umount /mnt/newvm/boot
umount /mnt/newvm
xm create -c