We've obtained much of the lab equipment that we need to get started, so I've been working on getting it all installed and configured. The first thing I wanted to set up was a Xen server, which could then host virtual servers providing various essential services (DNS, DHCP, .deb/.rpm caching, VPN access, lab wiki, console server, syslog/Splunk, MRTG, Nagios, etc.) for the rest of the lab.
I've used Xen on Debian Linux as my primary server environment for several years, and been reasonably happy with it. The latest release of Xen (version 3.3.0) includes some new features that I think might be useful to us (such as optional full hardware virtualization, so that you can run guest operating systems without needed to modify their kernels, as is required for the older "paravirtualization" model that has historically been Xen's basis), but the latest version that's available in the Debian archives is Xen 3.0.3. I've been hearing a lot about Ubuntu Linux, which is a Debian derivative, and when I noticed that Xen 3.3 was available for Ubuntu, I decided to give Ubuntu a try.
It took a little work and a few bugfixes to get Xen 3.3 fully installed and configured on the latest Ubuntu release (8.10, aka "Intrepid Ibex"), but it seems to be working just fine now. Here are the steps I went through to get it working, in case anybody else might be interested.
1: Install Ubuntu 8.10
First, you need to download an Ubuntu 8.10 installation image, burn it onto a CD-R, and boot your server from that. There are several flavors of installation image to choose from; server versus desktop, GUI versus text-based installer, 32-bit versus 64-bit, and so forth. I chose the appropriate image for a 32-bit server (since that's what I have) with the text-based "alternate installer" (since that's reportedly just like Debian's installer, and I hadn't bothered to hook a mouse up to my server).
When you boot from the installation CD, you're presented with an installation startup screen. There are some option menus across the bottom of the screen; one of them (F4, I think, but I'm not sure) has an option "Install a command-line system", and that's what I chose. You could probably do all this with the standard Ubuntu GUI system as well, but I didn't see the point in devoting the resources (disk space, complexity, my time to figure out and use the GUI, etc.) to a GUI for a system that's going to be running headless in a rack most of the time.
1.1: Disk Partitioning and Layout
The next part of the installation process is disk partitioning and layout. In this particular server, I have two 320GB SATA drives. I decided that I wanted to run them in an RAID1 configuration, with the bulk of the space devoted to LVM for maximum flexibility in allocating disk space to virtual guests. My standard RAID1/LVM setup consists of partitioning each disk identically:
||Linux RAID auto (0xFD), bootable
||for md0, which will be /boot
||Linux swap (0x82)
||for swap space
||Linux RAID auto (0xFD)
||for md1, which will be / (root filesystem
||(rest of disk)
||Linux RAID auto (0xFD)
||for md2, which will be the LVM physical volume (PV)
Then, you combine the physical partitions from the two disks into RAID1 ("MD") partitions:
||sda1 and sdb1
||sda3 and sdb3
||sda4 and sdb4
||LVM physical volume
(Note that sda2 and sdb2 do NOT get merged into a RAID partition; the kernel will use both of them directly for swap space, which does not need to be RAID-protected.)
Finally, you set up an LVM volume group (I call mine "vg0") with RAID1 partition md2 as its sole physical volume (PV). This is the volume group that you'll allocate logical volumes (LVs) out of for disk and swap space for Xen guest hosts.
1.2: Install Kernel and Software
At this point, the Ubuntu installer will walk you through installing the basic operating system kernel and software packages onto the partitions that you just laid out. When it's done, the system will reboot from the newly-installed disks, and you'll have your basic system up and running.
2: Configuration and Additional Installation
2.1: Log in and set root password
You can now log in to the system on the console using the username and password that you specified during the Ubuntu installation process. Once logged in, you can do "sudo -i" to become root, which you'll need to be for the rest of these steps.
It's apparently not standard on Ubuntu to be able to log in as root, but I like to be able to, so I immediately set a root password:
dom0# password root
2.2: Set up network interfaces
The Ubuntu installer prompts you for information that it uses to set up the first network interface (eth0), which it apparently assumes gets its address via DHCP. If you have additional interfaces, or want to use static addresses rather than DHCP, you need to edit /etc/network/interfaces:
dom0# vi /etc/network/interfaces
... (see 'man interfaces' for details, as well as the comments and examples within the file)
dom0# ifdown eth0 ; ifup eth0 ; ifup eth1
(The "ifdown" and "ifup" commands shutdown and restart the eth0 and eth1 interfaces, so that the changes you just made in /etc/network/interfaces take effect.)
2.3: Configure DNS
You probably need to edit /etc/resolv.conf to tweak your DNS client configuration:
dom0# vi /etc/resolv.conf
2.4: Ensure everything is up to date
It's likely (almost certain, in fact) that there are updates available for some of the packages that the installer installed from the CD-R. So, as soon as you've got your network working, you should bring everything up to date:
dom0# aptitude update
dom0# aptitude safe-upgrade
2.5: Install and configure NTP
I like having an accurate clock on my systems, so the next step for me is to install and configure an NTP server on the system:
dom0# vi /etc/ntp.conf
I usually take advantage of the NTP Pool Project to find a set of 4 or so NTP servers to peer with, so I put the following in my /etc/ntp.conf file:
If your ISP offers an NTP server (see if you get a response to "ping ntp.your-isp-domain.net"), you might want to add that to the list as well, since it's probably topologically "closer" to you (and thus will give you more reliable time) than the random servers you get assigned from the NTP Pool Project.
Once you're done editing your /etc/ntp.conf file, you need to restart the NTP daemon so that your configuration changes take effect:
dom0# /etc/init.d/ntp restart
2.6: Install and configure SSH
I want to be able to access and manage my server remotely via SSH, so I need to install and configure that, then restart it so that my config changes take effect:
dom0# aptitude install openssh-server
dom0# vi /etc/ssh/sshd_config
dom0# /etc/init.d/ssh restart
About all I usually change in the /etc/ssh/sshd_config file is the "Port" specification, which controls which TCP port the SSH server listens for incoming connections on. I like to move my SSH server from the standard port (22) to some other port (which I memorize). This won't stop a determined attacker, but it keeps the riff-raff from continuously rattling the doorknob.
2.7: Install Xen packages
Now we're ready to install the various Xen-related packages. Ubuntu makes this very easy by defining a special "ubuntu-xen-server" pseudo-package which brings in everything you need:
dom0# aptitude install ubuntu-xen-server
2.8: Install fixed Xen kernel
Well, almost everything you need... It turns out that there is apparently a bug in the standard Ubuntu 8.10 kernels, which keeps networking from working for Xen guest hosts. That's kind of important, but there's a patched kernel available. So, we need to get curl, and then use that to get the patched kernel, then install the patched kernel:
dom0# aptitude install curl
dom0# cd /root
dom0# curl -O http://www.il.is.s.u-tokyo.ac.jp/%7Ehiranotaka/linux-image-2.6.24-16-xen...
dom0# dpkg -i linux-image-2.6.24-16-xen_2.6.24-16.30zng1_i386.deb
(this will leave a copy of the installation package in root's home directory, /root, in case you need to do it again or something)
2.9: Disable TLS Libraries
Xen has trouble with the TLS libraries, so you need to simply disable them:
dom0# mv /lib/tls /lib/tls.disabled
2.10: Reboot into your new Xen kernel
At this point, you're ready to reboot into your new Xen kernel, and start creating guest virtual hosts
2.11: Install xen-tools
This is optional, but I find the xen-tools package very useful for creating Xen guest installations:
dom0# aptitude install xen-tools
dom0# vi /etc/xen-tools.conf
Among other changes in the /etc/xen-tools.conf file, you'll want to uncomment the "lvm =" line and set it to "lvm = vg0", so that disk and swap space for your guest hosts are created out of the LVM space that you set up during partitioning.
2.12: Fix xendomains problems
2.12.1: Fixing /etc/init.d/xendomains
There is a silly bug in the /etc/init.d/xendomains script, which is used to start and stop Xen guest hosts when the dom0 server starts and stops, which will keep it from working if you don't fix it. Here's a patch:
--- /etc/init.d/xendomains.orig 2008-10-06 11:08:55.000000000 -0700
+++ /etc/init.d/xendomains 2008-12-29 18:48:23.000000000 -0800
@@ -183,7 +183,7 @@
name=`echo "$1" | cut -d\ -f1`
- rest=`echo "$1" | cut cut -d\ -f2-`
+ rest=`echo "$1" | cut -d\ -f2-`
read id mem cpu vcpu state tm < <(echo "$rest")
(just edit the file, search for "cut cut", and change it to simply "cut".)
2.12.2: Disable save/resume on dom0 reboot
With Xen, you should be able to "save" a running guest domain's state, and then "resume" that guest at some later time (even if the underlying dom0 host reboots in the mean time). This is handy, as it means that you can do a quick reboot of your dom0 server without having to reboot your guests; they'll simply "freeze" for a minute or so while the dom0 server reboots, then continue as they were. This "save before a dom0 shutdown, and resume after a dom0 boot" behavior has worked great in past versions of Xen, and is the default behavior of the "xendomains" script.
Unfortunately, with this version of Xen (3.3.0 on Ubuntu 8.10), it doesn't quite work... The "resumed" domains have problems, and break in subtle ways (processes start getting hung in "defunct" state rather than being harvested by their parents, for example). So, unfortunately, it's safer for xendomains to shut down the guest domains when the dom0 shuts down, and boot them when the dom0 boots. To do this, you edit the /etc/default/xendomains file and set the XENDOMAINS_SAVE= value to blank (nothing).
dom0# vi /etc/default/xendomains
... set XENDOMAINS_SAVE= to ""
This is by no means everything you'll want to do to customize your own installation of Xen 3.3 on Ubuntu 8.10 (Intrepid Ibex), but it should be enough to get you started, and to get you past a few key hurdles (needing the alternate kernel, and fixing the xendomains bugs and problems). Good luck!
Next time, I'll share some notes on how to set up a Xen dom0 that has multiple network interfaces such that guest hosts can access one or more of those networks according to their Xen configuration file, and how your Xen dom0 can do NAT for all its guest hosts.