Geeky The road to DevOps.

So you want to be a Systems Administrator?

Recently I wrote about the shortage of entry level Operations people. These are the people who run the millions of computers that make everything in our daily lives possible. Without them, there would be no large working computer systems, anywhere.

Imagine a world where air traffic controllers don't have computers, where you need to talk to an operator any time you want to make a phone call, and where you have no email, no instant messaging, and no websites. We've certainly come to the point where it's hard to imaging a world without websites.

The role of managing these and other computer systems is held in Operations groups. An Operations group can include a variety of people, from the desktop support team for a company, to the people who design and build large scale computing infrastructures, to the senior management that help guide the company's IT decisions. It is often a large and under recognized group which has recently been getting a lot more attention for its importance in organizations.

We'll be examining one specific role in the group, the systems administrator, and what you should do to become one.

What exactly is a systems administrator?

The role of the systems administrator (or sysadmin) is quite diverse and encompasses a variety of specialties (in much the same way as doctors graduate and specialize in surgery, pediatrics, internal medicine, and so on). At the high level, the systems administrator is responsible for building and maintaining the computer systems for an organization. When a system crashes, they fix it. When it under-performs, they tune it, and when it gets old they upgrade it.
You might be thinking that it sounds like a glorified member of a home tech support company, and to a degree you're right. In the same way that you might ask someone to come fix your PC at home, the systems administrator fixes computer that run the point of sale system at your grocery store, the scheduling system for your doctor's office, flight reservations system at airlines, and so on.

Well that doesn't sound very exciting!

It's certainly not for everyone! Those of us who have chosen to be systems administrators enjoy it greatly. We get to sit behind the scenes, making sure millions of people every day can order a coffee, read a newspaper, and buy cute hand knitted kittens. Our satisfaction comes from ensuring our users can get to the information they need without interruption, and from building elegant networks of computers to achieve some pretty amazing things.

That sounds better! Ok, you've got me - I want to be a sysadmin!

Great! I have good news, and bad news for you.
The bad news is that there is no short cut to becoming a sysadmin. This is not something you can go to school to learn (yet). Being a sysadmin is much more a vocation or a trade. Most people are self-taught, or they take on apprenticeship and work their way up. Most people (and certainly most successful sysadmins I know) started learning about the work for fun, and eventually grew it into a career, or saw other sysadmins around them and decided they wanted to get involved.

Other successful sysadmins initially take the development path, working as software engineers for several years while they build up their operations skill sets "on the job".

The good news is that the work can be incredibly fulfilling and motivating, and lead to a vast number of different opportunities which can take a whole lifetime to explore and enjoy.
Systems administration can also be a stepping stone to other careers such as software development, IT management, and other engineering careers.

Is that it?

Certainly not! The best news is we have our own day, on the last Friday of July every year. While some may argue otherwise, I believe there is good long term job security as a systems administrator - like the other critical roles, all companies need people to manage their computer systems. We're at an interesting point in the evolution of IT systems; computers are indispensable in almost every type of business, but the availability of skills had not kept up with demand. Computers are also still very specialist devices and will likely remain this way for the foreseeable future.

So tell me - what are my entry paths into this wonderful career?

You have a couple of options:
If you are the academic type, I would recommend pursuing a degree in computer science, computer engineering or even electrical engineering. Most sysadmins have at least a passing interest in one of these topics, whether they study them of not. The desire to take things apart and put them back together, to break and fix things, and to improve what already exists are strong personality traits in many sysadmins.

If you aren't really the academic type but still are keep on working hands on with computers, I would recommend finding entry level work with computers. Jobs such as desktop support, or the tech support helpdesk are both common first jobs for systems administrators.

In either case, spend your spare time browsing the internet for tutorials on common sysadmin tasks:

• Install a web server
• Read the entire http://www.ipprimer.com/ website - this is where I learnt how TCP/IP works and it's really easy to digest
• Buy a cheap old computer at a flea market or on craigslist (don't pay over $100!), and open it up - identify all the parts inside, take it apart, put it back together. Does it still work? Good!
• Google for articles on tweaking your computer to run faster.

• If you're running Windows, find out how defragmenting your hard drive works, browse the entries in the Event Viewer and find out what the all mean, and go over the list of Windows Services and learn about each one.
• If you run OS X or Linux, grab a copy of Think Unix by Jon Lasser
  (ISBN: 978-0789723765). It is a great introduction to how Unix-style operating systems work.

I spent the many of my early years reading popular computer magazines and browsing the help columns. When someone had a problem they wrote in about, it was a great opportunity to learn a little more about how things break and how they can be fixed. When you can identify the solution to those problems before reading the answer from the columnist, being a sysadmin is for you.

The Future of Operations

This week I was at Velocity Conf with most of the Etsy Ops team. We had people talking, people helping, and people having great fun all week long.

Towards the end of the week, during the DevOps Days meetings, I had a chat with John Allspaw and the other members of the team about the future of Operations as a craft and trade. Some interesting things came out of this.

...but first a little history lesson:

Back in the late 80's, through to about 2003, most people agree the best place to find strong operations talent was in ISPs. For years ISPs had been incubators for people who were hungry to learn, improve and excel at Operation roles such as Systems Administration.

At the end of the dot-com bubble, businesses realized they needed to tighten their belts in order to survive. One of the unintended consequences of this was the end of the great incubators we had come to know and love. Suddenly people entering the trade did not have such great places to go.

It took several years for companies centered around web technologies to grow into the main stream. Google, Amazon, Facebook, Yahoo, MySpace and many others were able to provide a new place for the raw talents of newcomers to be tempered.

Having crossed that rickety bridge we find a new problem is emerging: In the last ~10 years it has become more difficult to find junior operations talent.
I don't know what the cause of this is, but it is clearly a trend that needs more investigation and attention.

I feel that as a community, we need to step up and encourage more people to enter the operations trade.
It's certainly a hard sell; there are few schools which provide degrees in systems administration which isn't entirely surprising as Systems Administration is much more a craft and vocation than an academic pursuit.
We're also waiting for systems administration salaries to stabilize again after the recent global recession.

That said, the outlook today seems good. The jobs board at Velocity Conf was covered multiple layers deep in jobs ads. By the end of the conference, our little Etsy Careers sticker could hardly be seen.

Most systems administrators start out being self-taught, playing with Linux or Windows internals, learning how the system works.
I feel that we need to do more to encourage this activity, and expose more people to the crafts of Systems Administration and Operations, in much the same way that companies have done for Computer Science and Software Engineering careers over the last two decades.

There is a growing understanding of the importance of Operations from respected leaders in the IT industry, and research groups, but we need to make sure we don't ignore the foundations of the pyramid we've spent many years building. We need to keep bringing more people into the fold, for they are the ones who will continue the work.

Schools around the world welcome anyone who wants to talk to students about career options. I encourage anyone reading this to find a local high school and see if they would be willing to accept a talk with students on the benefits of careers in Operations.

The community needs to be nourished not just with knowledge, but also with talent, energy and ideas from new faces.

Creating an LVM-backed FreeBSD DomU in a Linux Dom0

Greetings!
As the topic suggests we're going to play with Xen and set up a FreeBSD DomU inside a Linux Dom0.

First some important information:

Background

We're going to be running a PV (para-virtualised) installation of 32-bit FreeBSD. Unfortunately my older hardware doesn't support the HVM (hardware virtual machine) CPU extensions, and at the time of writing the stability of 64-bit FreeBSD as a DomU is not certain.

Credit

A lot of credit for this goes to the following posts. I was able to piece together bits of what I needed to make this process work from all of the hard work put in by these guys
http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/GrowFS
http://forums.freebsd.org/showthread.php?t=10268
http://tuxhelp.org/debian/xen/xen_server_configuration#preparing_a_pv_domu_for_a_posix_os_eg_freebsd

Pre-requisites

You will need an existing FreeBSD install to start with. This is needed to build the XEN kernel for FreeBSD, and the template disk image from which we'll make our final OS. The default kernel does not contain the drivers needed to run as a Xen guest. This is very easy to solve: download VirtualBox and the latest FreeBSD DVD ISO. Do a basic install with the kernel and OS sources.

You'll also need an existing Xen Dom0 installation with LVM partitioning. I'm using Debian 5 (Lenny) as my Xen host with Xen 3.2. It seems to be working quite well. There are many good instructions online on how to set this up.

Setting up the whole world

Log in to your VirtualBox (or other) FreeBSD installation.
We're going to create a disk image, compile the OS and then put the files onto the image.
root# cd /usr/src
/usr/src# truncate -s 1G /tmp/freebsd.img
/usr/src# mdconfig -f freebsd.img

This will tell you which md device was set up. We'll assume here it was md0.
/usr/src# fdisk -BI /dev/md0
/usr/src# bsdlabel -wB md0s1
/usr/src# newfs -U md0s1a
/usr/src# mount /dev/md0s1a /mnt
/usr/src# make buildworld && \
make buildkernel KERNCONF=XEN
/usr/src# make DESTDIR=/mnt installworld && \
make DESTDIR=/mnt installkernel KERNCONF=XEN && \
make DESTDIR=/mnt distribution

Edit /mnt/etc/ttys and add this line:
xc0 "/usr/libexec/getty Pc" vt100 on secure

Edit /mnt/etc/fstab and add this line:
/dev/ad0s1a / ufs rw 0 0

Unmount the image, compress it, and copy it to your destination:
# umount /mnt
# mdconfig -d -u 0
# bzip2 -v9 /tmp/freebsd.img
# scp /tmp/freebsd.img.bz2 \
user@dom0-host.example.com:/tmp/freebsd.img.bz2

You will also need to copy the kernel image to your Dom0 host. Xen will use this to boot your upcoming installation:
scp /usr/obj/usr/srcsys/XEN/kernel \
user@dom0-host.example.com:/tmp/freebsd-kernel

Configure your Dom0

There are quite a few (simple) steps required here. We're going to set up your LVM partition, install two copies of the FreeBSD OS:

1. A maintenance VM which is used to resize your filesystems
2. Your main FreeBSD VM

Both of these are created from the VM template image you created in the previous step.

Create your LVM partitions

My LVM physical group is named xen-vol. Replace this with the name of your PG. The first partition will be for the the maintenance VM, the second will be for the main DomU:
root@dom0# lvcreate -L1000 -n freebsdmaint.example.com xen-vol
root@dom0# lvcreate -L110000 -n freebsd.example.com xen-vol

Now copy the template into the partitions:
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/freebsdmaint.example.com \
bs=1M
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/dom0-host.example.com \
bs=1M

Configure it up

Here are two example configurations. The first one is for the maintenance VM, the second is for the main FreeBSD VM. Edit them to have the right file paths. You'll notice they both have a kernel parameter. This points to the kernel file you copied over earlier in this process. Make sure that file is somewhere safe - it will be required to boot your VMs. I keep mine under /xen. I also found that, as of FreeBSD 8.2-RC1, booting with 2 virtual CPUs fails, so I set this to 1:

freebsdmaint.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsdmaint.example.com,hda,w', 'phy:/dev/xen-vol/freebsd-dom0.example.com,hdb,w' ]
name = 'freebsdmaint.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

freebsd-dom0.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsd-dom0.example.com,hda,w' ]
name = 'freebsd-dom0.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

Now start up your maintenance VM:
xm create -c freebsdmaint.example.conf.cfg

The process of resizing

Now let's start with the process of resizing. We'll be resizing three things on our main freebsd partition:

• The partition (with fdisk)
• The slice (with bsdlabel)
• The filesystem (with growfs)

The partition for the main FreeBSD instance will be exposed here as /dev/ad1s1a

Resize partition in fdisk

Start by editing the disk in fdisk like this:
# fdisk -u /dev/ad1
******* Working on device /dev/ad1 *******
parameters extracted from in-core disklabel are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Do you want to change our idea of what BIOS thinks ? [n] n
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 2088387 (1019 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 129/ head 254/ sector 63
Do you want to change it? [n] y


Answer 'y' to the first two questions. You will then be asked:
Supply a decimal value for "size" [2088387]

For me, the correct answer here was calculated as:
Size in MB * 2048

It's likely you will get a warning such as this:
fdisk: WARNING: partition does not end on a cylinder boundary
fdisk: WARNING: this may confuse the BIOS or some operating systems
Correct this automatically? [n] y
Choose 'y', then choose the default answers to everything, and write out the changes:
Should we write new partition table? [n] y

Resize the slice in bsdlabel

Fire up bsdlabel with:
# bsdlabel -e /dev/ad1s1

Your editor will pop up and look like this:
# size offset fstype [fsize bsize bps/cpg]
a: 2088351 16 unused 0 0
c: 2088387 0 unused 0 0 # "raw" part, don't edit

Ignore the warning to not edit the "raw" part - I and others have received errors when you grow the slice but not the raw part. The warning likely refers to shrinking the a: partition.

There are two important things to note here:

1. The size of the raw part will be the same as the size you set the partition to in the previous fdisk step.
2. The size of the a: partition will also be the same as the size of the partition in the previous step MINUS 16.

My updated label looks like:
# size offset fstype [fsize bsize bps/cpg]
a: 225279416 16 unused 0 0
c: 225279432 0 unused 0 0 # "raw" part, don't edit

Let's growfs this thing!

You've made it this far, we're into the home stretch!
Let's use growfs(8) to increase the size of our filesystem:
# growfs /dev/ad1s1a
We strongly recommend you to make a backup before growing the Filesystem
Did you backup your data (Yes/No) ? Yes
[ lots of superblock numbers outputted ]

Let it all up and running

Now you can shutdown and maintenance VM, and start up your FreeBSD instance.

WARNING! ACHTUNG!
It is very important that you do not run both VMs at the same time in their current state - both will try to mount the filesystem read-write! Xen may give you a warning if you try and do this, but you certainly should not depend on it doing so (and indeed, it may not). The safest thing to do, once you shut down the maintenance VM, is to edit its configuration and remove the mention of the main VM's disk.

That's all! I hope people find this useful. Most of the credit for this rightly goes to Ray White (University of Waterloo), Adrian Chadd (developer) and Aprogas (FreeBSD user). I just put it all together in a way that worked for me

Run commands against all of your Puppet or Chef hosts

Both Puppet and Chef are incredibly powerful tools, and they are great at allowing your servers to keep themselves up to date.

However, there comes a time in every sysadmin's life when they find there is a need to poll a large number of servers in their infrastructure. They're faced with two problems:

1. How do you get a list of all of your current servers?
2. How do you connect to them all quickly?

Both Puppet and Chef are able to solve these problems for us.

Generating your server list

Let's look at the first problem: How do you get a list of all of your current servers?
The answer can depend on where you typically store your list of servers:

• On a piece of paper
• In a text file
• In a spreadsheet
• In a database

The configuration management systems themselves also store a list of known servers internally, and getting at this list is very easy. If you accept that all of your servers run puppet or chef, then you can use the system to get at the list.

Chef:
HOSTS=$(knife status | awk '$4 ~ /example.com/{ print $4 }')
knife returns a nice display of the list of known servers and when they last ran chef-client. The fourth column is the FQDN of the server.

Puppet:
HOSTS=$(puppetca -la | awk '{ print $2 }')
puppetca's output is less pretty, simply showing which servers are known. The second column is the FQDN of the server.

It happens that over time, servers will be removed, renamed or be offline when you you do this work, so you want to make a list of servers which have an SSH daemon listening. Nagios is a great monitoring system, and it comes with a plugin called check_ssh, which tests to make sure you can SSH to a server. We can use this to generate a list of servers:

HOSTS_AVAIL=$( for host in ${HOSTS}; do check_ssh -t1 ${host} > /dev/null && echo ${host}; done )

Running your distributed commands

The second problem with have is: How do you connect to them all quickly?

Here we can drop back to some fun shell scripting. The following loop runs multiple ssh connections at the same time, up to some predefined limit. For this step, it is highly advised that you have a way to log in to your servers without having to type your password in each time. SSH keys are a great way to accomplish this.

MAX=5 # Don't run more than 5 SSH sessions at once
CMD="uptime" # The command you want to run
for server in ${HOSTS_AVAIL}; do
    while :; do
        if [ $(ps ax | grep -E "ssh.*${CMD}") -ge 5 ]; then
            sleep 1
        else
             break
        fi
    done
    ssh ${server} "${CMD}" >> ${server}.log 2>&1 &
done

This might look complicated, but in reality it's quite simple:

• Define the maximum number of processes to run, and the command to run on the remote servers
• For each server...
• If we have more than 5 processes running, sleep 1 second and check again
• Once there is a slot open for a process to run, execute it into the background, and set the output to ${server}.log and loop around to the beginning immediately

Alternative methods

Chef specific way:
Chef allows us to display hosts which are subscribed specific roles. We can use this connect to all of the chef clients quickly:

knife ssh "$( for host in
    $(knife status | awk '$4 ~ /example.com/{ print $4 }'); do
        /opt/local/libexec/nagios/check_ssh -t 1 ${host} > /dev/null && \
        echo ${host}; done)" \
    "ps ax | grep httpd" \
    -m -x<username> -P'<password>' -l debug

Creating an SVN mirror

If you haven't had the pleasure of working with a distributed version control system such as Mercurial or Git, you may find that it necessary to take some manual steps to set up a mirror of your repository.

For SVN, this process isn't well integrated into the system itself, but there are tools we can use to achieve this goal. An important thing to be aware of, is that unlike a distributed, you won't be able to merge changes from your mirror back into the original repo. You should make sure the only thing writing to the mirrored copy is the svnsync process.

Tthe documentation on the internet is pretty terse for this, here're the steps which worked for me.

I wanted to locally mirror a repo which is normally servered over https (although it'd work equally well for repos servers using svn or svn+ssh).

• Remote repo: https://svn.example.org/repos/project
• Local directory: /home/svn/project

Create an empty SVN repository locally:
svn create /home/svn/project

You need to create a hook script at /home/svn/project/hooks/pre-revprop-change which looks like:

#!/bin/sh

USER="$3"

exit 0

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may change revision properties" >&2
exit 1

Make the script executable:
chmod 755 /home/svn/project/hooks/pre-revprop-change

Now initialise the repo as a mirror:
svnsync init file:///home/svn/project https://svn.example.org/repos/project

Finally, you can do the sync:
svnsync sync file:///home/svn/project

I have that last command run from cron every minute to keep things up to date:
/etc/cron.d/svnsync
# Sync the SVN mirror every minute
* * * * * root /usr/bin/svnsync sync file:///home/svn/project

Resizing a Solaris partition

This was originally worked out when I had to resize a Solaris 10 partition under VMware, but it applies equally to native Solaris installs too!
WARNING: This only really works, if you have one partition. This should be true if you're using VMware, and for the majority of Solaris installations. Partition, here, is in the true sense of the word: partitions as seen by fdisk, the BIOS and other disk management software, not the slices you mount as /usr, /home, and so on. Those are, errr, slices.
If you're doing this for a VMware install, power off the virtual machine, and use vmware-vdiskmanager to resize the image file: vmware-vdiskmanager -x 20Gb solaris10_vmware.vmdk.
If the partition you are resizing is the root partition, you need to boot up in "failsafe" mode, or boot from a Solaris CD/DVD and choose the "shell" option at the first menu. You'll be asked if you want to mount the hard drive install as /a. You do not. But you do want to remember the name of the disk (eg, c1t0d0p0).

* Dimensions:
*    512 bytes/sector
*     63 sectors/track
*    255 tracks/cylinder
*   2610 cylinders