Install using vmbuilder on Ubuntu
vmbuilder is a command line tool for creating virtual machines on Ubuntu. It can only create Ubuntu installations, contrary to
virt-manager and the collection of tools that use
libvirt. This is one way to use scripts instead of GUI for installation, although virt-manager can be used after for additional support.
The purpose of this document is to describe how to easily get a working Foswiki site by using vmbuilder. I have not found any duplicates of this effort and hope this is the right place for this document. Vmbuilder is quite well documented so this document is more to try to make a complete and easy process for getting a new server and Foswiki site up and running, ready for test or whatever you desire to do with it.
This is under construction. Anyone is invited to join. Comments at the bottom.
Testing is done on Utuntu 10.04, 10.10 and 11.04. Help wanted testing. There are unofficial deb files and repository for Debian based distros if that would be preferable.
This process generates a new virtual machine in a few minutes. On my laptop it takes about 3 minutes. And it does not require CD-ROM, ISO's or other stuff than your existing Ubuntu installation.
The vm's generated can be managed with the
virt-manager GUI, added automatically. The vm will be a clean current stable Foswiki and dedicated to that task. It will be a Ubuntu LTS server and you can of course do whatever modifications you wish to both the install described here and after the server is created. It will be a normal Ubuntu server.
Hopefully this will end up with at least two bash scripts, one for setting up the build machine, your local Ubuntu install, and another for quickly getting a ready Foswiki site ready.
Build machine
You need to install KVM and some packages to get started. Here's a sample command for a typical setup:
apt-get install kvm qemu-kvm bridge-utils build-essential virtinst virt-manager apt-cacher subversion
See
vmbuilder and JeOS.
Add yourself to the group
libvirtd
. (Log out and back in necessary?).
usermod $(whoami) -a -G libvirtd
Basics of vmbuilder
Sample command. The \ escapes newline so this can be copy/pasted into a bash command prompt:
vmbuilder kvm ubuntu \
--mem=1024 \
--cpus=1 \
--hostname=vm01 \
--dest=vm01 \
--mirror=http://192.168.122.1:3142/ubuntu \
--user=wikiadmin \
--name=administrator \
--pass=secret \
--rootpass=verysecret \
--libvirt=qemu:///system \
--rootsize=2048 \
--swapsize=512 \
--overwrite \
--tmpfs -
When the command has run successfully start the vm via virt-manager or
virsh --connect qemu:///system start vm01
Remember to start apt-cacher first if you want caching:
#apt-cacher
Run in background or have it autostart if you need. I only start it manually when needed.
The DEST parameter will create a directory with given name and put files there. Disk files will default go to
/var/lib/libvirt/images
or
/etc/libvirt/
and also the config file used for virt-manager.
Virtual machines can be for KVM, VMWare (vmw6, vmserver), Xen. For now this is focused on
KVM and adding to
virt-manager. If you want to skip virt-manager and not use any GUI, just skip that parameter and vmbuilder will create a file called
run.sh
in the directory with the virtual machine. In that file you see how to start the vm with just the disk file and parameters.
KVM does not need the xml config file, you can start a virtual machine by specifying the disk file and appropriate parameters.
#vmbuilder kvm ubuntu --help
Usage: vmbuilder hypervisor distro [options]
Options:
-h, --help show this help message and exit
--version Show version information
Build options:
--debug Show debug information
-v, --verbose Show progress information
-q, --quiet Silent operation
-o, --overwrite Configuration file
-c CONFIG, --config=CONFIG
Configuration file
--templates=DIR Prepend DIR to template search path.
-d DESTDIR, --destdir=DESTDIR
Destination directory
--only-chroot Only build the chroot. Don't install it on disk images
or anything.
--chroot-dir=CHROOT_DIR
Build the chroot in directory.
--existing-chroot=EXISTING_CHROOT
Use existing chroot.
-t DIR, --tmp=DIR Use TMP as temporary working space for image
generation. Defaults to $TMPDIR if it is defined or
/tmp otherwise. [default: /tmp]
--tmpfs=SIZE Use a tmpfs as the working directory, specifying its
size or "-" to use tmpfs default (suid,dev,size=1G).
Disk:
--rootsize=SIZE Size (in MB) of the root filesystem [default: 4096]
--optsize=SIZE Size (in MB) of the /opt filesystem. If not set, no
/opt filesystem will be added.
--swapsize=SIZE Size (in MB) of the swap partition [default: 1024]
--raw=PATH Specify a file (or block device) to use as first disk
image (can be specified multiple times).
--part=PATH Specify a partition table in PATH. Each line of
partfile should specify (root first): mountpoint
size one per line, separated by space, where size is
in megabytes. You can have up to 4 virtual disks, a
new disk starts on a line containing only '---'. ie:
root 2000 /boot 512 swap 1000 ---
/var 8000 /var/log 2000
Other options:
--ssh-key=PATH Add PATH to root's ~/.ssh/authorized_keys (WARNING:
this has strong security implications).
--ssh-user-key=SSH_USER_KEY
Add PATH to the user's ~/.ssh/authorized_keys.
--manifest=PATH If passed, a manifest will be written to PATH
Post install actions:
--copy=FILE Read 'source dest' lines from FILE, copying source
files from host to dest in the guest's file system.
--execscript=SCRIPT, --exec=SCRIPT
Run SCRIPT after distro installation finishes. Script
will be called with the guest's chroot as first
argument, so you can use 'chroot $1 <cmd>' to run code
in the virtual machine.
Settings for the initial user:
--user=USER Username of initial user [default: ubuntu]
--name=NAME Full name of initial user [default: Ubuntu]
--pass=PASS Password of initial user [default: ubuntu]
--rootpass=ROOTPASS
Initial root password (WARNING: this has strong
security implications).
--uid=UID Initial UID value.
--gid=GID Initial GID value.
--lock-user Lock the initial user [default: none]
Package options:
--addpkg=PKG Install PKG into the guest (can be specified multiple
times).
--removepkg=PKG Remove PKG from the guest (can be specified multiple
times)
--seedfile=SEEDFILE
Seed the debconf database with the contents of this
seed file before installing packages
General OS options:
-a ARCH, --arch=ARCH
Specify the target architecture. Valid options: amd64
i386 lpia (defaults to host arch)
--hostname=HOSTNAME
Set NAME as the hostname of the guest. Default:
ubuntu. Also uses this name as the VM name.
Scripts:
--firstboot=PATH Specify a script that will be copied into the guest
and executed the first time the machine boots. This
script must not be interactive.
--firstlogin=PATH Specify a script that will be copied into the guest
and will be executed the first time the user logs in.
This script can be interactive.
Installation options:
--suite=SUITE Suite to install. Valid options: dapper gutsy hardy
intrepid jaunty karmic lucid maverick natty [default:
lucid]
--flavour=FLAVOUR, --kernel-flavour=FLAVOUR
Kernel flavour to use. Default and valid options
depend on architecture and suite
--variant=VARIANT Passed to debootstrap --variant flag; use minbase,
buildd, or fakechroot.
--iso=PATH Use an iso image as the source for installation of
file. Full path to the iso must be provided. If
--mirror is also provided, it will be used in the
final sources.list of the vm. This requires suite and
kernel parameter to match what is available on the
iso, obviously.
--mirror=URL Use Ubuntu mirror at URL instead of the default, which
is http://archive.ubuntu.com/ubuntu for official
arches and http://ports.ubuntu.com/ubuntu-ports
otherwise
--proxy=URL Use proxy at URL for cached packages
--install-mirror=URL
Use Ubuntu mirror at URL for the installation only.
Apt's sources.list will still use default or URL set
by --mirror
--security-mirror=URL
Use Ubuntu security mirror at URL instead of the
default, which is http://security.ubuntu.com/ubuntu
for official arches and http://ports.ubuntu.com
/ubuntu-ports otherwise.
--install-security-mirror=URL
Use the security mirror at URL for installation only.
Apt's sources.list will still use default or URL set
by --security-mirror
--components=COMPS A comma seperated list of distro components to include
(e.g. main,universe).
--ppa=PPA Add ppa belonging to PPA to the vm's sources.list.
--lang=LANG Set the locale to LANG [default: nb_NO.UTF-8]
--timezone=TZ Set the timezone to TZ in the vm. [default: UTC]
Network:
--domain=DOMAIN Set DOMAIN as the domain name of the guest [default:
defaultdomain].
Network:
--ip=ADDRESS IP address in dotted form [default: dhcp].
--mac=MAC MAC address of the guest [default: random].
--mask=VALUE IP mask in dotted form [default: based on ip setting].
Ignored if ip is not specified.
--net=ADDRESS IP net address in dotted form [default: based on ip
setting]. Ignored if ip is not specified.
--bcast=VALUE IP broadcast in dotted form [default: based on ip
setting]. Ignored if ip is not specified.
--gw=ADDRESS Gateway (router) address in dotted form [default:
based on ip setting (first valid address in the
network)]. Ignored if ip is not specified.
--dns=ADDRESS DNS address in dotted form [default: based on ip
setting (first valid address in the network)] Ignored
if ip is not specified.
libvirt integration:
--libvirt=URI Add VM to given URI
--bridge=BRIDGE Set up bridged network connected to BRIDGE.
--network=NETWORK Set up a network connection to virtual network
NETWORK.
VM settings:
-m MEM, --mem=MEM Assign MEM megabytes of memory to the guest vm.
[default: 128]
--cpus=CPUS Assign NUM cpus to the guest vm. [default: 1]
Caching for faster creation of vm's
Your build machine can cache packages for the process to avoid fetching from the internet thus speeding up the process. This is optional and if you have a decent internet connection it would not matter much.
Apt-cacher seem to work well for this purpose. Start apt-cacher before doing the vmbuilder and point to the cache with the
mirror
or
install-mirror
parameter. Apt-cacher uses port 3142.
You can import into cache existing packages to save initial time:
/usr/share/apt-cacher/apt-cacher-import.pl /var/cache/apt/archives/
There are other proxy tools, like apt-proxy, use any if you want this caching.
Customisation of vm (virtual machine)
You will want to have some custom settings on the vm. Set root passwd, add packages etc.
Networking
We use default and get NAT networking. You can interact from your local Ubuntu installation and you virtual machines. See the doc for doing a brigde instead. If you want to route traffic you can use virt-manager with a GUI or do command line with
brctl
.
Disk and partition layout
The parameters are rootsize, swapsize and optsize. Or you can have a simple text file like this:
root 5000
swap 1000
---
/var 10000
The triple dash separator line makes /var be a separate disk file, /dev/sdb, while root (/) and swap partitions are on /dev/sda.
Add and remove packages
Add and remove packages as needed. Here is what works for me:
--addpkg unattended-upgrades \
--addpkg acpid \
--addpkg ssh \
--addpkg vim \
--addpkg rcs \
--addpkg sendmail \
--addpkg apache2 \
--addpkg libcgi-pm-perl \
--addpkg libapache-session-perl \
--addpkg libapache-htpasswd-perl \
--addpkg wget \
--addpkg subversion \
--addpkg libnet-ldap-perl \
--addpkg libhtml-entities-numbered-perl \
--addpkg libhtml-parser-perl \
--addpkg libhtml-tree-perl \
--addpkg libwww-perl \
--addpkg libauthen-sasl-perl \
--addpkg libi18n-charset-perl \
--addpkg libunicode-maputf8-perl \
--addpkg acpid
Getting Foswiki and setting up a site
We need to create a script file that will be copied into the vm and ran at first boot. The parameter
--firstboot scriptfile --firstlogin scriptfile
will do this.
#!/bin/bash
# firstlogin.sh
ip=$(ifconfig eth0 | awk '/inet addr/ { print $2 }' | sed 's/.*://')
echo now visit http://$ip/foswiki/bin/configure
#!/bin/bash
# firstboot.sh
current=Foswiki-1.1.2.tgz
dlurl=http://sourceforge.net/projects/foswiki/files/foswiki/1.1.2
www=/var/www/foswiki
wget $dlurl/$current
# no verbose on untar, takes too long
tar xzf $current
mkdir /var/www/foswiki
cp -a ${current%.tgz}/* /var/www/foswiki/
cd $www
chown -R www-data:www-data $www
htpasswd -bc $www/data/.htpasswd admin secret
chown www-data:www-data $www/data/.htpasswd
cd $www/bin
cp LocalLib.cfg.txt LocalLib.cfg
chown www-data:www-data LocalLib.cfg
sed 's|absolute/path/to/your|var/www/foswiki|' -i LocalLib.cfg
sed 's|\(php_admin_flag\)|#\1|' $www/foswiki_httpd_conf.txt > $www/foswiki.conf
sed 's|JohnDoe|admin|' -i $www/foswiki.conf
cp $www/foswiki.conf /etc/apache2/sites-available/foswiki
a2ensite foswiki
/etc/init.d/apache2 reload
If you need to set a proxy for internet connections, it's in /etc/subversion/servers.
We'll stick with the tgz for now.
We can check out current from subversion and do pseudo-install. That would maybe work well, how about mixing linked installed plugins and updating then from configure?
svn co http://svn.foswiki.org/branches/Release01x01@HEAD foswiki
mv foswiki /var/www/foswiki
# etc.. would perhaps be more suitable for dev/test
The checkout can be modified to check out Tags for Foswiki 1.1.3, i.e. the versjon from .tgz file. That would also have known issued and bugs. @HEAD will most likely be stable and have fixes you want. You can also point to trunk for testing bleeding edge.
STATUS:
On my machine:
- Virtual machines are created, from copy/paste text from here:
- Copy top section (sample) and packages section into a new text file (create.sh), and add the two parameters --firstboot firstboot.sh and --firstlogin firstlogin.sh (need full path to script files, use $(pwd)/firstboot.sh )
- create the firstboot.sh and firstlogin.sh by copying the two above sections into each file
- Then you can run the create.sh
- vm is created
- start vm, =virsh --connect qemu:///system start vm01
- firstboot.sh runs at boot
- login on console, firstlogin.sh runs and gives URL to configure (run via ssh also first time)
- To find ip of vm without logging in to it first, look in /var/log/syslog. There you find DHCPACK and dnsmasq-dhcp entries.
- No virsh or other command line to find vm ip?
- Doing save in configure LocalSite.cfg is created and the wiki is ready to go
TODO:
- probably something missing, should already hav DBI, LDAP etc...
- how about that logging....
- Three files attached to this topic, works here.
- Make Foswiki 1.1.3 (RC).
Other notes and links
A side note on cloning/imaging virtual machines:
- grub use disk uuid in the configuration, meaning new disk get a different uuid, and that has to be handled (parameter in /etc/default/grub: #GRUB_DISABLE_LINUX_UUID=true, update grub config, /etc/fstab)
- /etc/udev/rules.d/70-persistent-net.rules ties eth0 to the MAC address when booted for the first time. A clone will therefore create and use eth1 (eth0 binds to the original vm's eth0).
These issues are avoided when using vmbuilder as it is a new install.