My current method for tracking/gitifying changes in foriegn VCSes

In vmbuilder.git we deal with a lot of upstream branches on launchpad being tracked with bazzar. So, I have to convert those revisions to git commits. This is how I handled importing a new branch from launchpad that branched from Bzed's r344 today:

#grab the revision

bzr branch lp:~ckaenzig/+junk/lenny

cd vmbuilder

#go to the point this revision branches from - note that this is a tag and not a branch

git checkout bzed_vmbuilder_debian-r344

#make a new branch for the underlying bzr branch

git checkout -b ckaenzig_junk_lenny

#clear out the current git working area, to reflect any RMs in the bzr revision

git rm * -r

#clear out any untracked files, such as .pyc and .py~

rm * -rf

#copy the revision over to the git repo, and stage it

cp ../lenny/* . -r

git add * -r

#make a new commit on this branch, which should now reflect the underlying bzr revision.

git commit -m "New bzr branch lp:~ckaenzig/+junk/lenny"

#tag the commit by it's underlying bzr revision number for easy reference

git tag ckaenzig_junk_lenny-r346 -a

This process generally works, but it requires careful attention and is easy to screw up. For example, I have to do a second rm to remove files that aren't being tracked by git (.pyc, .py~ files) , otherwise I end up committing junk that might be from another branch entirely when I stage the new revision. Not to mention that I may miss a revision this way. Say I have a branch A in bzr that I've manually imported this way for revisions r100, r123, and I wanna import branch B which breaks off from branch A at r112, I have to either put the new git branch at r100 (and have slightly inaccurate commit history) or start moving commits around, which I've never tried and probably doesn't push well. Fortunately VMbuilder doesn't get that many commits per day so I usually am able to keep on top of 'em.

Is there an easier way to convert bzr revisions into git commits? I'm horribly unfamiliar with bzr, so I haven't really investigated automating this process.

.euca//GIFT

Over the weekend Stephen Moeller contributed a bunch of patches to the Vmbuilder repository for Virtualbox support as well as fixing up the Debian packaging. You can examine them at the usual place(1), but be forewarned that I haven't tested them just yet. I'll get these off to Bzed tomorrow once I can be bothered to go downstairs and hook up an ethernet cable for Debian. And, of course, the usual machinations of pulling in changes from the vmbuilder trunk and bzed debian branch.

(1) git://git.debian.org/git/pkg-escience/vmbuilder.git

P.S. A little background - my Dell Studio 1535, while otherwise being a good machine, is cursed with a Broadcom Wifi card. I would rather not install non-free drivers on my Debian install after having a few bad experiences using the NVIDIA drivers on Debian on an older machine. So instead I go downstairs and physically connect the Lappy-486 to the home network.

yay mass commit bombs!

More work on VMbuilder - after fixing how the Debian plugin finds Xen kernels I got it to build a Xen image. (The Debian plugin was looking for images the way Ubuntu versions them, not the way Debian versions them) Now I need to figure out building EC2 images and such.

And send bzed a patched distro.py.

Epic merge fail

(seven less than symbols since blogger can't handle them)

#import sys

#import os

#sys.path += [os.getcwd()]

import VMBuilder

=======

class Suite(object):

def __init__(self, vm):

self.vm = vm

self.isodir = '/media/vmbuilder_inst_image'

self.iso_mounted = False

(seven greater than symbols since blogger can't handle them)vmbuilder_trunk:VMBuilder/plugins/ubuntu/suite.py

This happened to come from my attempt to merge VMbuilder trunk 0.11 with the current master; Git thought suite.py was moved to debian-vm-builder and tried to merge it.

Oh and yes that also means VMbuilder trunk 0.11 is now in alioth. I can't believe I missed it. It includes all the EC2 stuff without the install_[hardy,intrepid,jaunty] cruft as well.

So, to-do list:

• Get Debian plugins working with EC2 VMbuilder. The debian plugin branch on Launchpad hasn't been updated since June 6th; maybe bzed maintains another repo?
• Try building the package and installing it. Side note: They removed packaging info from the trunk midway between 0.10 and 0.11, but I kept the /debian folder anyway in vmbuilder.git.
• Speaking of which during the merge I noticed the default sudoers template explicitly references an "ubuntu" group, something that should be changed eventually.
• Look into the vmbuilder-gui branch sometime.

fs weirdness

So I was in the middle of refreshing my vmbuiler sources when I got an out-of-disk error in the middle of the bzr pull. Fine, so I fired up Filelight, excluded /media and checked to see how much was being used... 5GBs. On a 19GB partition. Feeling this to be a bit odd I figured I'd rather extend the partition another gigabyte. I fire up the Ubuntu partiton, load gparted, and.... it reports the Debian partition as being 5GBs used. Odd. So I extend the partition back a gig, which takes a great deal of time. That's done, so I remount the new debian, fixup grub, restart, and get a kernel panic.

It is at this point that I remember that I had hibernated Debian instaed of shutting it down, and then I moved the filesystem out from under it. So I reboot again, it works. First thing I do after login is check "df"... and the drive reports 5GBs used.

I'm not a master at kernel internals but I have at least part of an inkling that the missing 14 gigabytes of space had something to do with hibernating and restoring the machine multiple times.

In other news it looks like there is a branch vmbuilder-gui, which I'll have to look into sometime.

VMbuilder delay

I said I expected to have a working 'beta' of vmbuilder that could make AMIs of Debian by Friday. Well I don't think I can get it to you in time, because of the way that vmbuilder works. You see, the core VMbuilder has a nice system to support future distros sanely. So I could easily extend it to create VMs for pretty much anything that uses the Linux kernel.

But... EC2 has very specific pre-install/post-install actions that are hard-coded to three different Ubuntu distributions. So I need to talk to the EC2 plugin guys about how to refactor the thing to allow the distro plugin to offer specific EC2 actions. I _could_ just add preinstall/install_etch/lenny functions, but that would be a copout. I suspect this is the reason why the EC2 plugin isn't in VMbuilder trunk.

And as a final kink, it uses ec2-tools to bundle and upload the image, and a python library called Boto to register it in the VM deploy code. So I need to get patches in Boto - or find someone who's trying to get patches in it - to allow it to connect to Eucalyptus.

On the other hand, I found someone who's already written Debian plugins that work with regular (non-EC2/Euca) VMs. It only supports Etch, but Lenny shouldn't be that different from Etch.

So let's just say when I said Friday on the SOC coordination mailing list, it was in Valve Time.

Previous attempts at vmbuilder-debian

Eric Hammond informed me of a previously-existing branch of vmbuilder which was adding Debian support. It currently has only etch support, but I decided to work off that.

I've merged it into the existing Git which is currently based off the Vmbuilder trunk. There's not much difference between the two anyway: the trunk has packaging that the branch doesn't, and the branch has a Debian plugin and frontend which the trunk doesn't.

Now to set it up and see if it works.

Stop clobbering my bootloader!

During this past weekend I tried to get Windows XP, Windows 7, OSX, Debian, and Ubuntu all on one machine. Due to some mishaps, driver oddities with Windows 7, and other things, I had to install certain operating systems multiple times. Now, the problem is, each operating system has it's own peculiar way of loading. Linux uses GRUB, Windows XP has NTLDR, 7 has the Windows Boot Manager, etc. Most of these operating systems also decide it would be helpful for the end-user that when his computer reboots, it can actually load the new OS. That's great, but I already have a bootloader.

I have a partition at /dev/sda5 which contains a GRUB installation. That's supposed to handle every OS's own method of booting, including the Linux installations which have their own GRUBs on their own partitions. But when you decide to overwrite the MBR, I don't get your nice and shiny OS. A good number of OSes use an MBR that won't boot an extended partition anyway. So instead of seeing your bright and shiny new OS, I see "NTLDR is missing". Or the ever-so-helpful "boot0: error". I'm -lucky- if I even get an OS at all, much less one that can load the grub shell so I can fix what you broke.

I understand having 3 GRUB installations plus a whole bunch of other bootloaders isn't exactly a standard use case. Most people don't go beyond dual-boot. But please, for the love of all that is holy, please just put a little checkmark somewhere,"Do not replace MBR". Ubuntu and Debian are the only OSes that actually give you that option in some form.

Or better yet, let's all just standardize on GRUB. Sure, it's not perfect, but it's as flexible as you can get.