Geeky The road to DevOps.

16Mar/1216

Tornado Alert App

Tornado Alert LogoI live two hours south of Joplin, MO.
Almost a year ago a tornado drove through the middle of this small town and destroyed everything in its path. I had only recently moved to this area and figured that while it was really bad, it may not be entirely uncommon. Then I did a bad thing: I felt bad, but I did nothing.
Tornado season ended and I watched as people slowly put their lives back together. To this day my wife sends me pictures of the animals the Humane Society is *still* finding as a result of the aftermath.

Tornado season started again rather suddenly a few weeks ago. This time, however, I wasn't prepared for what I saw - two children, one almost the same age as our youngest son, picked up and thrown hundreds of feet by a tornado. Their families didn't know a tornado was headed their way, and by the time they found out it was too late. Everyone in both families died that day.

With the incredible amount of technology at our disposal, I found it saddening that we weren't doing more to save lives. People still listen to the news, and wait many minutes before tornados are spotted before they hear about them. These precious minutes are far too long.

No more.

I present to you, Tornado Alert.
This simple app runs on almost any Android phone sold in the last 2 years.
The key thing that sets it apart from every other tornado alerting application I've seen is that Tornado Alert is almost entirely crowd sourced.

What does this mean?
It means that if you install the application, and you spot a tornado, you can press *one button* on the menu, and alert other people with the same app within 10 miles. Fast.

Fast, eh? How fast?
5-10 seconds. That fast. The National Weather Service and TV stations often takes upwards of 5 minutes to report a tornado. If you're listening to a live broadcast and waiting for confirmation of a tornado, it could be too late. These services, while invaluable and life saving in and of themselves, also require your TV or radio to still be on. Anyone who has lived through a tornado producing storm knows that electricity is often the first service to be lost.

How does Tornado Alert alert me?
When a new Tornado Warning or Tornado Watch is issued by the National Weather Service, we immediately send a notification to your phone.
If you see a tornado in your area, you simple press the "Tornado Near Me!" button, and a notification is sent to all other users in the area.
You won't get alerted more than once every 5 minutes for the same thing - the last thing you need when you're trying to get to safety is to be bugged by your phone.

Oh yeah? I bet you charge a lot for this. There's an iPhone app that will alert me and costs $10!
Zero - That's exactly how much you will pay, now and forever. The app is and will remain completely free to you. I have only one motivation for this - I don't want anyone else to die from something completely preventable. Children need their parents, and communities need families. Let's help each other stay alive.
In future we will look for other ways to help support the service, either through sponsorship or ads, or something. There will also be a version of the software you can pay for if you want to donate to running the service, but there is no obligation to do so.

The future
The application is in its first release. I feel it is in a good state to get out and start being immediately useful. We do however have a number of features which we'll be working into future versions:

  • Integration with social media: When you spot a tornado, we'll post on your Facebook wall and twitter, so your friends and family know what is happening.
  • Alerts via phone calls: Currently we can only use notification sounds on your phone to alert you. In future we'll add the ability for you to receive a phone call too.
  • Take pictures of the tornado and share them with others.
  • Differentiated between high-priority and low-priority alerts. Currently we assume everything is high priority.
  • Any other options people ask for 🙂
  • Versions for iPhones and Blackberry devices (help is appreciated on these)

Please, take a look at the application, try it out. It's very light weight, and hopefully it can help save a life this tornado season.
You can find the application in the Android Market / Google Play called "Tornado Alert" or go: HERE.

Filed under: Tech 16 Comments
8Mar/125

Slow shell open in OSX Lion

Do you find that opening a new tab in Terminal, or iTerm takes a long time? Maybe 5 to 10 seconds, instead of being nearly instant?

If so, you may want to take a look at Apple's syslog replacement called ASL.

ASL stores its logs on OSX Lion, under /var/log/asl/.
When Terminal or iTerm open, by default the execute:

login -fp <username>

This (for whatever reason) seems to read all of the log files under that directory. If like me you have 300Mb of rotated log files in there since your last reboot, opening a new shell can take quite a while.

I don't know what the impact of deleting the files is, but given that they're log files I deleted the rotated ones anyway. So far, so good! Shells open quickly again!

7Feb/127

Significantly increase your 4G Android’s battery life

Sounds like either a lie, or spam, right? Not so!
In the last week I have managed to triple the battery life of my Droid Bionic, with one Android setting and one simple app.

A word of warning: This method will work best if you are in areas where there are WiFi connections available. If you find yourself on the road or traveling around a lot, you may not have as much of a gain.

We'll take a look a the major causes of battery drain in cell phones and then look at how we can eliminate them.

Why your battery life sucks
To understand why we can get better battery life, you have to understand why your battery life is so terrible to begin with:

  • Signal strength
    At home where I spend most of my time as a remote-employee, my signal strength is fairly weak. Under normal circumstances my phone can maintain 1-2 bars. This is pretty terrible. If you can maintain 4-5 bars of signal strength for most of your day your battery life gains may not be as impressive as mine. As your signal gets weaker, your phone has to use more power to maintain a good connection to the network.
  • 2G, 3G, 4G, WiFi oh my!
    • 2G is the technology by which all cell phones make digital phone calls. These protocols have been around since 1991 and continue to work well for the most basic phone functions. Enhancements to 2G such as EDGE and 1xRTT allowed data transfer speeds up to 115kbit/sec
    • 3G is a more recent technology which started gaining use in 2001/2002. 3G provides higher data transfer speeds, while using more power. There are many variations on how 3G is implemented, but data transfer speeds are required to be at least 2Mbit/sec for stationary users.
    • 4G LTE is the latest technology, offering significantly more bandwidth than before. 4G offers at least 100Mbit/sec for fast-moving users, and 1Gbit/sec for stationary or slow moving users. Of course this comes at the cost of significantly more power use.
    • WiFi, while not a mobile network, is the other option we have for data transfer. While in theory it is slower than 4G, in practice it can be at least as fast or faster than the current generation of 4G and uses far less power. The reduced power is due to the limited range at which WiFi networks can operate. While mobile networks have to be able to work over many miles, WiFi networks are far more limited and usually don't extend far beyond 100-200ft (40-70m) in the best case.
  • Dual-mode and circuit-switched fallback
    There's a problem with 4G. Phone calls are made over 2G and 3G networks, but can't be made over 4G yet. Why not? Well, 4G is not widely available yet, and until a week ago there was no technology available to hand off calls between 4G and 3G/2G networks. So networks have worked around this in two ways:

    • Dual-mode operation is used by CDMA networks such as Verizon and Sprint. In this mode, your phone maintains connections to BOTH 2G and 4G networks at the same time. Yikes! This double-connection to the network is one of the primary reasons people on these networks have such problems with battery life.
    • Circuit Switched Fallback (CSFB) is a similar trick done on GSM networks such as AT&T and T-Mobile. When a call comes in, the phone is kicked off the 4G network and switches to 2G/3G for the duration of the call. Not very elegant, an this switching causes more power drain. If you take a lot of short calls, this can drain your battery even faster.

Extending that battery life
Ok, that's all well and good, but how can we use this knowledge to extend the life of our battery?

In my case, and in the cases of the small (and probably biased) group of people I've spoken to, we found our phone use fell mostly into this pattern:

  • Heavy use at home or at work
  • Lighter use while outdoors or commuting

Whereas 4G uses quite a lot of power, it is also very handy to have for some quick browsing and navigation on the move. Our goal is to reduce dependence on 4G and take advantage of available WiFi networks around us. Ideally, we want to have the WiFi network connected all the time. On an android phone, this can be achieved by going to your Settings -> Wireless & network -> Wi-Fi settings. From here, tap the Menu button and choose Advanced Settings, and change your Wi-Fi Sleep Policy to "Never". This will cause your Wi-Fi connection to stay on permanently.

Now some of you many have already noticed that being connected to a WiFi network causes your phone to lose power FASTER not SLOWER!

Very true! There is one more crucial step.
Open the Android Market, and search for an app called DataSwitch by TomatoX.
DataSwitch is a wonderful application, which allows you to completely disable and enable your 3G and 4G connection.

As you have your phone using your available WiFi connections, you have no need for a data connection to your mobile network. Each time you run the DataSwitch app, it will enable or disable your mobile data connection.

Final results
Using this method I was able to extend the life of my battery from a typical life of 15 hours on 4G, to 45 hours. Quite a gain!

Filed under: Tech 7 Comments
13Jul/1127

So you want to be a Systems Administrator?

Recently I wrote about the shortage of entry level Operations people. These are the people who run the millions of computers that make everything in our daily lives possible. Without them, there would be no large working computer systems, anywhere.

Imagine a world where air traffic controllers don't have computers, where you need to talk to an operator any time you want to make a phone call, and where you have no email, no instant messaging, and no websites. We've certainly come to the point where it's hard to imaging a world without websites.

The role of managing these and other computer systems is held in Operations groups. An Operations group can include a variety of people, from the desktop support team for a company, to the people who design and build large scale computing infrastructures, to the senior management that help guide the company's IT decisions. It is often a large and under recognized group which has recently been getting a lot more attention for its importance in organizations.

We'll be examining one specific role in the group, the systems administrator, and what you should do to become one.

What exactly is a systems administrator?

The role of the systems administrator (or sysadmin) is quite diverse and encompasses a variety of specialties (in much the same way as doctors graduate and specialize in surgery, pediatrics, internal medicine, and so on). At the high level, the systems administrator is responsible for building and maintaining the computer systems for an organization. When a system crashes, they fix it. When it under-performs, they tune it, and when it gets old they upgrade it.
You might be thinking that it sounds like a glorified member of a home tech support company, and to a degree you're right. In the same way that you might ask someone to come fix your PC at home, the systems administrator fixes computer that run the point of sale system at your grocery store, the scheduling system for your doctor's office, flight reservations system at airlines, and so on.

Well that doesn't sound very exciting!

It's certainly not for everyone! Those of us who have chosen to be systems administrators enjoy it greatly. We get to sit behind the scenes, making sure millions of people every day can order a coffee, read a newspaper, and buy cute hand knitted kittens. Our satisfaction comes from ensuring our users can get to the information they need without interruption, and from building elegant networks of computers to achieve some pretty amazing things.

That sounds better! Ok, you've got me - I want to be a sysadmin!

Great! I have good news, and bad news for you.
The bad news is that there is no short cut to becoming a sysadmin. This is not something you can go to school to learn (yet). Being a sysadmin is much more a vocation or a trade. Most people are self-taught, or they take on apprenticeship and work their way up. Most people (and certainly most successful sysadmins I know) started learning about the work for fun, and eventually grew it into a career, or saw other sysadmins around them and decided they wanted to get involved.

Other successful sysadmins initially take the development path, working as software engineers for several years while they build up their operations skill sets "on the job".

The good news is that the work can be incredibly fulfilling and motivating, and lead to a vast number of different opportunities which can take a whole lifetime to explore and enjoy.
Systems administration can also be a stepping stone to other careers such as software development, IT management, and other engineering careers.

Is that it?

Certainly not! The best news is we have our own day, on the last Friday of July every year. While some may argue otherwise, I believe there is good long term job security as a systems administrator - like the other critical roles, all companies need people to manage their computer systems. We're at an interesting point in the evolution of IT systems; computers are indispensable in almost every type of business, but the availability of skills had not kept up with demand. Computers are also still very specialist devices and will likely remain this way for the foreseeable future.

So tell me - what are my entry paths into this wonderful career?

You have a couple of options:
If you are the academic type, I would recommend pursuing a degree in computer science, computer engineering or even electrical engineering. Most sysadmins have at least a passing interest in one of these topics, whether they study them of not. The desire to take things apart and put them back together, to break and fix things, and to improve what already exists are strong personality traits in many sysadmins.

If you aren't really the academic type but still are keep on working hands on with computers, I would recommend finding entry level work with computers. Jobs such as desktop support, or the tech support helpdesk are both common first jobs for systems administrators.

In either case, spend your spare time browsing the internet for tutorials on common sysadmin tasks:

  • Install a web server
  • Read the entire http://www.ipprimer.com/ website - this is where I learnt how TCP/IP works and it's really easy to digest
  • Buy a cheap old computer at a flea market or on craigslist (don't pay over $100!), and open it up - identify all the parts inside, take it apart, put it back together. Does it still work? Good!
  • Google for articles on tweaking your computer to run faster.
  • If you're running Windows, find out how defragmenting your hard drive works, browse the entries in the Event Viewer and find out what the all mean, and go over the list of Windows Services and learn about each one.
  • If you run OS X or Linux, grab a copy of Think Unix by Jon Lasser
    (ISBN: 978-0789723765). It is a great introduction to how Unix-style operating systems work.

I spent the many of my early years reading popular computer magazines and browsing the help columns. When someone had a problem they wrote in about, it was a great opportunity to learn a little more about how things break and how they can be fixed. When you can identify the solution to those problems before reading the answer from the columnist, being a sysadmin is for you.

Filed under: Tech 27 Comments
19Jun/111

The Future of Operations

This week I was at Velocity Conf with most of the Etsy Ops team. We had people talking, people helping, and people having great fun all week long.

Towards the end of the week, during the DevOps Days meetings, I had a chat with John Allspaw and the other members of the team about the future of Operations as a craft and trade. Some interesting things came out of this.

...but first a little history lesson:

Back in the late 80's, through to about 2003, most people agree the best place to find strong operations talent was in ISPs. For years ISPs had been incubators for people who were hungry to learn, improve and excel at Operation roles such as Systems Administration.

At the end of the dot-com bubble, businesses realized they needed to tighten their belts in order to survive. One of the unintended consequences of this was the end of the great incubators we had come to know and love. Suddenly people entering the trade did not have such great places to go.

It took several years for companies centered around web technologies to grow into the main stream. Google, Amazon, Facebook, Yahoo, MySpace and many others were able to provide a new place for the raw talents of newcomers to be tempered.

Having crossed that rickety bridge we find a new problem is emerging: In the last ~10 years it has become more difficult to find junior operations talent.
I don't know what the cause of this is, but it is clearly a trend that needs more investigation and attention.

I feel that as a community, we need to step up and encourage more people to enter the operations trade.
It's certainly a hard sell; there are few schools which provide degrees in systems administration which isn't entirely surprising as Systems Administration is much more a craft and vocation than an academic pursuit.
We're also waiting for systems administration salaries to stabilize again after the recent global recession.

That said, the outlook today seems good. The jobs board at Velocity Conf was covered multiple layers deep in jobs ads. By the end of the conference, our little Etsy Careers sticker could hardly be seen.

Most systems administrators start out being self-taught, playing with Linux or Windows internals, learning how the system works.
I feel that we need to do more to encourage this activity, and expose more people to the crafts of Systems Administration and Operations, in much the same way that companies have done for Computer Science and Software Engineering careers over the last two decades.

There is a growing understanding of the importance of Operations from respected leaders in the IT industry, and research groups, but we need to make sure we don't ignore the foundations of the pyramid we've spent many years building. We need to keep bringing more people into the fold, for they are the ones who will continue the work.

Schools around the world welcome anyone who wants to talk to students about career options. I encourage anyone reading this to find a local high school and see if they would be willing to accept a talk with students on the benefits of careers in Operations.

The community needs to be nourished not just with knowledge, but also with talent, energy and ideas from new faces.

Filed under: Tech 1 Comment
27Dec/1012

Creating an LVM-backed FreeBSD DomU in a Linux Dom0

Greetings!
As the topic suggests we're going to play with Xen and set up a FreeBSD DomU inside a Linux Dom0.

First some important information:

Background

We're going to be running a PV (para-virtualised) installation of 32-bit FreeBSD. Unfortunately my older hardware doesn't support the HVM (hardware virtual machine) CPU extensions, and at the time of writing the stability of 64-bit FreeBSD as a DomU is not certain.

Credit

A lot of credit for this goes to the following posts. I was able to piece together bits of what I needed to make this process work from all of the hard work put in by these guys 🙂
http://www.freebsd.uwaterloo.ca/twiki/bin/view/Freebsd/GrowFS
http://forums.freebsd.org/showthread.php?t=10268
http://tuxhelp.org/debian/xen/xen_server_configuration#preparing_a_pv_domu_for_a_posix_os_eg_freebsd

Pre-requisites

You will need an existing FreeBSD install to start with. This is needed to build the XEN kernel for FreeBSD, and the template disk image from which we'll make our final OS. The default kernel does not contain the drivers needed to run as a Xen guest. This is very easy to solve: download VirtualBox and the latest FreeBSD DVD ISO. Do a basic install with the kernel and OS sources.

You'll also need an existing Xen Dom0 installation with LVM partitioning. I'm using Debian 5 (Lenny) as my Xen host with Xen 3.2. It seems to be working quite well. There are many good instructions online on how to set this up.

Setting up the whole world

Log in to your VirtualBox (or other) FreeBSD installation.
We're going to create a disk image, compile the OS and then put the files onto the image.
root# cd /usr/src
/usr/src# truncate -s 1G /tmp/freebsd.img
/usr/src# mdconfig -f freebsd.img

This will tell you which md device was set up. We'll assume here it was md0.
/usr/src# fdisk -BI /dev/md0
/usr/src# bsdlabel -wB md0s1
/usr/src# newfs -U md0s1a
/usr/src# mount /dev/md0s1a /mnt
/usr/src# make buildworld && \
make buildkernel KERNCONF=XEN
/usr/src# make DESTDIR=/mnt installworld && \
make DESTDIR=/mnt installkernel KERNCONF=XEN && \
make DESTDIR=/mnt distribution

Edit /mnt/etc/ttys and add this line:
xc0 "/usr/libexec/getty Pc" vt100 on secure

Edit /mnt/etc/fstab and add this line:
/dev/ad0s1a / ufs rw 0 0

Unmount the image, compress it, and copy it to your destination:
# umount /mnt
# mdconfig -d -u 0
# bzip2 -v9 /tmp/freebsd.img
# scp /tmp/freebsd.img.bz2 \
user@dom0-host.example.com:/tmp/freebsd.img.bz2

You will also need to copy the kernel image to your Dom0 host. Xen will use this to boot your upcoming installation:
scp /usr/obj/usr/srcsys/XEN/kernel \
user@dom0-host.example.com:/tmp/freebsd-kernel

Configure your Dom0

There are quite a few (simple) steps required here. We're going to set up your LVM partition, install two copies of the FreeBSD OS:

  1. A maintenance VM which is used to resize your filesystems
  2. Your main FreeBSD VM

Both of these are created from the VM template image you created in the previous step.

Create your LVM partitions

My LVM physical group is named xen-vol. Replace this with the name of your PG. The first partition will be for the the maintenance VM, the second will be for the main DomU:
root@dom0# lvcreate -L1000 -n freebsdmaint.example.com xen-vol
root@dom0# lvcreate -L110000 -n freebsd.example.com xen-vol

Now copy the template into the partitions:
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/freebsdmaint.example.com \
bs=1M
root@dom0# dd if=freebsd.img \
of=/dev/xen-vol/dom0-host.example.com \
bs=1M

Configure it up

Here are two example configurations. The first one is for the maintenance VM, the second is for the main FreeBSD VM. Edit them to have the right file paths. You'll notice they both have a kernel parameter. This points to the kernel file you copied over earlier in this process. Make sure that file is somewhere safe - it will be required to boot your VMs. I keep mine under /xen. I also found that, as of FreeBSD 8.2-RC1, booting with 2 virtual CPUs fails, so I set this to 1:

freebsdmaint.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsdmaint.example.com,hda,w', 'phy:/dev/xen-vol/freebsd-dom0.example.com,hdb,w' ]
name = 'freebsdmaint.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

freebsd-dom0.example.conf.cfg
kernel = "/xen/kernels/freebsd_8.2-RC1_kernel"
vcpus = '1'
memory = '64'
disk = [ 'phy:/dev/xen-vol/freebsd-dom0.example.com,hda,w' ]
name = 'freebsd-dom0.example.com'
vif = [ 'bridge=eth0,mac=00:16:3E:62:DB:03' ]
extra = 'xencons=tty1'
extra = "boot_verbose"
extra += ",boot_single"
extra += ",kern.hz=100"
extra += ",vfs.root.mountfrom=ufs:/dev/ad0s1a"

Now start up your maintenance VM:
xm create -c freebsdmaint.example.conf.cfg

The process of resizing

Now let's start with the process of resizing. We'll be resizing three things on our main freebsd partition:

  • The partition (with fdisk)
  • The slice (with bsdlabel)
  • The filesystem (with growfs)

The partition for the main FreeBSD instance will be exposed here as /dev/ad1s1a

Resize partition in fdisk

Start by editing the disk in fdisk like this:
# fdisk -u /dev/ad1
******* Working on device /dev/ad1 *******
parameters extracted from in-core disklabel are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=14023 heads=255 sectors/track=63 (16065 blks/cyl)

Do you want to change our idea of what BIOS thinks ? [n] n
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 2088387 (1019 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 129/ head 254/ sector 63
Do you want to change it? [n] y

Answer 'y' to the first two questions. You will then be asked:
Supply a decimal value for "size" [2088387]

For me, the correct answer here was calculated as:
Size in MB * 2048

It's likely you will get a warning such as this:
fdisk: WARNING: partition does not end on a cylinder boundary
fdisk: WARNING: this may confuse the BIOS or some operating systems
Correct this automatically? [n] y

Choose 'y', then choose the default answers to everything, and write out the changes:
Should we write new partition table? [n] y

Resize the slice in bsdlabel

Fire up bsdlabel with:
# bsdlabel -e /dev/ad1s1

Your editor will pop up and look like this:
# size offset fstype [fsize bsize bps/cpg]
a: 2088351 16 unused 0 0
c: 2088387 0 unused 0 0 # "raw" part, don't edit

Ignore the warning to not edit the "raw" part - I and others have received errors when you grow the slice but not the raw part. The warning likely refers to shrinking the a: partition.

There are two important things to note here:

  1. The size of the raw part will be the same as the size you set the partition to in the previous fdisk step.
  2. The size of the a: partition will also be the same as the size of the partition in the previous step MINUS 16.

My updated label looks like:
# size offset fstype [fsize bsize bps/cpg]
a: 225279416 16 unused 0 0
c: 225279432 0 unused 0 0 # "raw" part, don't edit

Let's growfs this thing!

You've made it this far, we're into the home stretch!
Let's use growfs(8) to increase the size of our filesystem:
# growfs /dev/ad1s1a
We strongly recommend you to make a backup before growing the Filesystem
Did you backup your data (Yes/No) ? Yes
[ lots of superblock numbers outputted ]

Let it all up and running

Now you can shutdown and maintenance VM, and start up your FreeBSD instance.

WARNING! ACHTUNG!
It is very important that you do not run both VMs at the same time in their current state - both will try to mount the filesystem read-write! Xen may give you a warning if you try and do this, but you certainly should not depend on it doing so (and indeed, it may not). The safest thing to do, once you shut down the maintenance VM, is to edit its configuration and remove the mention of the main VM's disk.

That's all! I hope people find this useful. Most of the credit for this rightly goes to Ray White (University of Waterloo), Adrian Chadd (developer) and Aprogas (FreeBSD user). I just put it all together in a way that worked for me 🙂

Filed under: Tech 12 Comments
22Oct/100

Run commands against all of your Puppet or Chef hosts

Both Puppet and Chef are incredibly powerful tools, and they are great at allowing your servers to keep themselves up to date.

However, there comes a time in every sysadmin's life when they find there is a need to poll a large number of servers in their infrastructure. They're faced with two problems:

  1. How do you get a list of all of your current servers?
  2. How do you connect to them all quickly?

Both Puppet and Chef are able to solve these problems for us.

Generating your server list

Let's look at the first problem: How do you get a list of all of your current servers?
The answer can depend on where you typically store your list of servers:

  • On a piece of paper
  • In a text file
  • In a spreadsheet
  • In a database

The configuration management systems themselves also store a list of known servers internally, and getting at this list is very easy. If you accept that all of your servers run puppet or chef, then you can use the system to get at the list.

Chef:
HOSTS=$(knife status | awk '$4 ~ /example.com/{ print $4 }')
knife returns a nice display of the list of known servers and when they last ran chef-client. The fourth column is the FQDN of the server.

Puppet:
HOSTS=$(puppetca -la | awk '{ print $2 }')
puppetca's output is less pretty, simply showing which servers are known. The second column is the FQDN of the server.

It happens that over time, servers will be removed, renamed or be offline when you you do this work, so you want to make a list of servers which have an SSH daemon listening. Nagios is a great monitoring system, and it comes with a plugin called check_ssh, which tests to make sure you can SSH to a server. We can use this to generate a list of servers:

HOSTS_AVAIL=$( for host in ${HOSTS}; do check_ssh -t1 ${host} > /dev/null && echo ${host}; done )

Running your distributed commands

The second problem with have is: How do you connect to them all quickly?

Here we can drop back to some fun shell scripting. The following loop runs multiple ssh connections at the same time, up to some predefined limit. For this step, it is highly advised that you have a way to log in to your servers without having to type your password in each time. SSH keys are a great way to accomplish this.

MAX=5 # Don't run more than 5 SSH sessions at once
CMD="uptime" # The command you want to run
for server in ${HOSTS_AVAIL}; do
    while :; do
        if [ $(ps ax | grep -E "ssh.*${CMD}") -ge 5 ]; then
            sleep 1
        else
             break
        fi
    done
    ssh ${server} "${CMD}" >> ${server}.log 2>&1 &
done

This might look complicated, but in reality it's quite simple:

  • Define the maximum number of processes to run, and the command to run on the remote servers
  • For each server...
  • If we have more than 5 processes running, sleep 1 second and check again
  • Once there is a slot open for a process to run, execute it into the background, and set the output to ${server}.log and loop around to the beginning immediately

Alternative methods

Chef specific way:
Chef allows us to display hosts which are subscribed specific roles. We can use this connect to all of the chef clients quickly:

knife ssh "$( for host in
    $(knife status | awk '$4 ~ /example.com/{ print $4 }'); do
        /opt/local/libexec/nagios/check_ssh -t 1 ${host} > /dev/null && \
        echo ${host}; done)" \
    "ps ax | grep httpd" \
    -m -x<username> -P'<password>' -l debug
Filed under: Tech No Comments
14Oct/100

Creating an SVN mirror

If you haven't had the pleasure of working with a distributed version control system such as Mercurial or Git, you may find that it necessary to take some manual steps to set up a mirror of your repository.

For SVN, this process isn't well integrated into the system itself, but there are tools we can use to achieve this goal. An important thing to be aware of, is that unlike a distributed, you won't be able to merge changes from your mirror back into the original repo. You should make sure the only thing writing to the mirrored copy is the svnsync process.

Tthe documentation on the internet is pretty terse for this, here're the steps which worked for me.

I wanted to locally mirror a repo which is normally servered over https (although it'd work equally well for repos servers using svn or svn+ssh).

  • Remote repo: https://svn.example.org/repos/project
  • Local directory: /home/svn/project

Create an empty SVN repository locally:
svn create /home/svn/project

You need to create a hook script at /home/svn/project/hooks/pre-revprop-change which looks like:

#!/bin/sh

USER="$3"

exit 0

if [ "$USER" = "syncuser" ]; then exit 0; fi

echo "Only the syncuser user may change revision properties" >&2
exit 1

Make the script executable:
chmod 755 /home/svn/project/hooks/pre-revprop-change

Now initialise the repo as a mirror:
svnsync init file:///home/svn/project https://svn.example.org/repos/project

Finally, you can do the sync:
svnsync sync file:///home/svn/project

I have that last command run from cron every minute to keep things up to date:
/etc/cron.d/svnsync
# Sync the SVN mirror every minute
* * * * * root /usr/bin/svnsync sync file:///home/svn/project

Filed under: Tech No Comments
14Oct/105

Resizing a Solaris partition

This was originally worked out when I had to resize a Solaris 10 partition under VMware, but it applies equally to native Solaris installs too!
WARNING: This only really works, if you have one partition. This should be true if you're using VMware, and for the majority of Solaris installations. Partition, here, is in the true sense of the word: partitions as seen by fdisk, the BIOS and other disk management software, not the slices you mount as /usr, /home, and so on. Those are, errr, slices.
If you're doing this for a VMware install, power off the virtual machine, and use vmware-vdiskmanager to resize the image file: vmware-vdiskmanager -x 20Gb solaris10_vmware.vmdk.
If the partition you are resizing is the root partition, you need to boot up in "failsafe" mode, or boot from a Solaris CD/DVD and choose the "shell" option at the first menu. You'll be asked if you want to mount the hard drive install as /a. You do not. But you do want to remember the name of the disk (eg, c1t0d0p0).

Write out the current partition table to disk and edit the file:
# fdisk -W ptbl.tmp && vi ptbl.tmp

At the top of the file, you'll see the actual current disk geometry, which should look somethign like this:

* Dimensions:
*    512 bytes/sector
*     63 sectors/track
*    255 tracks/cylinder
*   2610 cylinders

The sectors, tracks and cylinders numbers are important - write these down somewhere, and skip to the end of the file.

You'll see a row of numbers, with words over them like Id, Act, Bhead etc. The two numbers you need to replace, correspond with Ecyl and Numsect.

Ecyl is the ending cylinder on the disk for this partition. The value for it, is the cylinders number that you wrote down in the previous step, MINUS ONE. The minus one is very important. While the total number of cylinders in my example is 2610, the ending cylinder number is actually 2609, because the numbering of cylinders on disk starts at zero, not one. This is the only time you need to worry about this.
You also need to change Numsect, which is the number of sectors on the disk. The value for this, is calculated as: sectors/track x tracks/cylinder x cylinders.
In our example, this is: 63 x 255 x 2610 = 41929650.

Save the file, and exit.

Now update the fdisk partition table on disk:
# fdisk -S ptbl.tmp -I /dev/rdsk/c1t0d0p0
Replace the partition name, with that of your disk. You'll be presented with a little menu. The table at the top should indicate that 100% of the disk space is now used by the partition(s). Choose option 5, to save and exit fdisk.

Finally, we get to resize the actual partitions! Reboot into multiuser mode.

Look in /etc/vfstab to find the device that is mounted, to the slice you want to grow. In my case, this was the root slice mounted from /dev/dsk/c1t0d0s0.
We need to turn this into a metadevice (think: software raid), to be able to grow it. Don't worry, this is perfectly (mostly) safe, and won't hurt your performance (much, as far as I know!).

metainit -f d10 1 1 c1t0d0s0. This creates a metadevice named d10, from the given slice name. The -f causes this to be forced, which is needed if the slice is currently mounted.
Run metastat. This will give you the name of the new metadevice you created. For me, it was called d0.
If you are working on resizing the root filesystem, you need to run metaroot /dev/md/dsk/d0. This will update /etc/vfstab with the correct mount information. Otherwise, you need to update /etc/vfstab manually.

Filed under: Tech 5 Comments
19Oct/070

Dynamically loading modules in Python

Have you wanted to load modules, when you didn't know their name?

I recently came across this problem while refactoring GoogleBot (my IRC bot). I wanted to place loadable modules in a directory, and have them all loaded, without having to know what they are named.

A good fellow called Issac at work helped me with this:

import imp
googlebot_mod_dir = os.path.abspath('modules')
sys.path.append(googlebot_mod_dir)
for module_file in os.listdir(googlebot_mod_dir):
module_name, ext = os.path.splitext(module_file)
if ext == '.py':
module_location = imp.find_module(module_name)
module = imp.load_module(module_name, *module_location)
globals()[module_name] = module

Works like a charm!

Filed under: Tech No Comments