David Wendt's GSoC Updates Blog

Pencils down

2011-08-22T14:09:00.000-07:00

Today is the GSoC pencils down date. I've created and pushed two new git branches, gsoc2011-gsttranscodebin and gsoc2011-gupnpdlna-integration. These have been rebased upon the latest master commit.

The work I have done this summer boils down to three things:

A new GStreamer plugin called "gsttranscodebin", which takes a single profile argument and transcodes it's sink pad to conform to the profile and outputs it on it's source pad. This is available at https://github.com/kmeisthax/gst-transcode.
Patches to Rygel to use gsttranscodebin to encode things. They are available here: https://github.com/kmeisthax/rygel/tree/gsoc2011-gsttranscodebin. It's the "gsoc2011-gsttranscodebin" branch.
Patches to Rygel to use GUPnP-DLNA to retrieve encoding profiles, obsoleting the current fixed-transcoder classes. They are available here: https://github.com/kmeisthax/rygel/tree/gsoc2011-gupnpdlna-integration. It's the "gsoc2011-gupnpdlna-integration" branch.

I split the Rygel work into two pieces - one involving GUPnP-DLNA and one involving GstTranscodeBin. Since I inevitably need to get GstTranscodeBin into Gstreamer eventually, I need to get in touch with some of the Gstreamer maintainers, including Edward Hervey (creator of decodebin2/encodebin, recommended to me by Zeeshan) - I want to know what they think of the element and if it can be added to Gstreamer eventually.

Asfor the GUPnP-DLNA work, it's mostly done, but there's kind of an issue - now that we have many more encoding profiles you'll also need many more encoder plugins installed, since we don't yet check if we can actually encode a particular profile. I believe the original fixed-transcoders had the same issue, but I've kind of exacerbated the problem. Additionally, MP3 encoding using the GUPnP-DLNA profile breaks the existing transcoding logic as well as GstTranscodeBin (since it's based on the existing transcoding logic). I'm not yet sure if this is an issue with the profiles I get from GUPnP, or how the transcode logic works. I'm going to post this to the appropriate mailing lists later in the week, hopefully when I have that last issue worked out.

Progress report, Aug 01 to 08

2011-08-08T20:27:00.000-07:00

Last week, I...

...finished up Rygel.DLNATranscoder
...patched Rygel.TranscodeManager to populate the Transcoder list with profiles from GuPNP-DLnNA
...removed Rygel.MP3Transcoder, Rygel.WMVTranscoder, Rygel.MP2TSTranscoder, Rygel.L16Transcoder
...begun testing

This week, I plan to...

...continue testing
...package up my changes into a patch
...celebrate

+David

2011-07-09T13:45:00.000-07:00

Got Google+, so you can follow me here. Feel free to annoy me with questions while I finish up this next piece of work on Rygel.

(Hint: It involves dynamically grabbing gupnp-dlna's profiles instead of the builtin transcoder classes)

My apologies but if you commented I had no clue

2011-07-01T23:13:00.000-07:00

I forgot to enable e-mailed comment notifications until just now, so if I ignored your comments in the past it's not because I'm lazy, it's because the system never told me about them.

GstTranscodeBin is (hopefully) done

2011-07-01T21:38:00.001-07:00

I now have transcodebin doing actual transcoding as part of rygel, so I think it's time to actually publicize some git repositories.

If you're wondering what I did over the past few weeks, aside from the annoying delays which kept me from my chair at times... I moved out all the transcoding logic from Rygel into a separeate gstreamer plugin called gst-transcode. The plugin can be spawned as "transcodebin" from the element factory; it provides a sink and a src pad. You also have to set the "profile" property with a GstEncodingProfile just like encodebin. Anything coming in the sink pad gets autoplugged and transcoded, and output on the src pad.

Transcodebin is available on this github repo:
https://github.com/kmeisthax/gst-transcode

There's also a test application in tests called gst-transcodetest which grabs a DLNA profile by name from gupnp-dlna and tries to transcode a file into the target format. Compilation of the test application is disabled by default; since it's not very useful, adds a build-dependancy on gupnp-dlna, and I plan to remove it eventually. If you want to compile it look in the Makefile.am and configure.ac files to see what needs to be uncommented.

It also has debian packaging which I added back before I spent two days switching from Ubuntu to Fedora in search of newer packages, then to Fedora Rawhide to get the bleeding edge. That ordeal is immortalized on my twitter.

Likewise I patched Rygel to use transcodebin; it removes about 50 lines of code from rygel-transcoder.vala. I've also got a github for that here:
https://github.com/kmeisthax/rygel

Master is just a mirror of the gnome git; my patch is in branch gsoc2011-transcoding. I've tested this on a Sony Bluray player, specifically playing an OCRemix album in OGG format live-transcoded to LPCM through Rygel and transcodebin.

There's a known bug; if I do OGG->LPCM through the gst-transcodetest program, autoplugging fails. OGG->MP3 works, and so does Rygel's LPCM transcoder. Since I next plan to replace Rygel's transcoders with a single generic transcoder fed by gupnp-dlna's profiles, I expect this bug to rear it's ugly head when said work is done.

Project info, IRC reachability, contact info

2011-05-28T18:33:00.000-07:00

For people interested in the project, the public page is available here. I'm making some enhancements to Gstreamer's DLNA support as well as moving some transcoding stuff out of rygel and into core. It's based off of a project request on the wiki.

There are multiple ways to contact me, including:

E-Mail! Just decode the following anti-spam mail string:

dcrkid COMMERCIAL-AT yahoo SMALL-LITTLE-DOT-ON-THE-BOTTOM com

Twitter! Follow @Libertardian for spurt-of-the-moment political and gaming rants

IRC! I'm registered on freenode as dwendt and typically lurk in #gstreamer, #gsoc, #ubuntu, and/or #Reddit

Reddi --- Actually, bad idea for me to be on Reddit during this project. Very bad idea. Forget I mentioned it.

Updated for Summer of Code 2011

2011-05-28T18:01:00.000-07:00

Alrighty, it's GSoC 2011 time! I'm reviving the blog I used in my GSoC2009 project and should have used for my GSoC2010 project but didn't. This time, however, my project involves the GStreamer people rather than Debian - so hopefully my posts aren't being automatically routed to Planet Debian this time.

apple, jailbreaking, and debconf in manhattan?!

2009-07-29T19:08:00.000-07:00

Well, while I tangle around with annoying conflicts between Xen and NetworkManager I might as well post something semi-related to computing. Today it's Apple's anti-jailbreaking plea. Right off the bat they point to their end user license "agreement" and claim that people who bought an iPhone or iPod Touch somehow agreed to this mystical EULA that's somehow signed at point of sale. Nevermind the fact that EULAs are non-negotiable, involve no transfer of title, money, or services, and frequently never even get shown to the user.

But enough about that. Apparantly DebConf 10 is going to be in New York City. Right next to where I live. That's great. I didn't go to DebConf this year, despite Arthur Liu recommending it to all the GSoCers. Mostly due to paranoia over airline security and US customs searching my laptop.

Debian is apparantly going to actually have release deadlines, albeit every other year as opposed to twice a year like Ubuntu. Still, it's nice to see a commitment to releasing. My first experience with Debian was a while back, when all the worthwhile packages were still in testing.

Hopefully, tomorrow I can get Xen working again. Xend won't start, something about an 'invalid dom0 state'. I suspect it has something to do with the fact that NetworkManager manually configures peth0 when it shouldn't, requiring me to fix up the routing tables. Worst comes to worse I'll have to try and run Xen and euca on my desktop, which is just barely compatible with Debian.

One of the things I'd like to do for VMbuilder is have it automatically package the Linux kernel and ramdisk into Eucalyptus, if the user opts for it. You can already upload new kernels and ramdisks with Eucalyptus/euca2ools provided your cert has the appropriate rights on the cloud. see [0]

[0] http://open.eucalyptus.com/wiki/EucalyptusImageManagement_v1.5.2

lenny vmbuilder with xen, debconf, I discover key visualization

2009-07-27T08:58:00.000-07:00

Well, the current VMBuilder available on the git archive[0] now supports building Lenny on vanilla Xen. We're very close to building AMIs, I can almost taste it. (The reason why you haven't heard about this until today is that accessing Blogger is super-finnicky when not logged in for absolutely no reason at all. My apologies for the lack of updates on this blog.) Additionally we're starting to actually put documentation and stuff on the wiki [1,2,3] about the GSoC project and euca2ools.

The next step is to look at a few packages called 'ec2-init' and 'ec2-modules'. They only exist in Ubuntu right now, and I purposefully disabled VMBuilder from installing them on Debian so far since Debian doesn't have them. That, and I believe I should pick out a suitable AKI/ARI combo sometime (the AKI info in the Debian plugin is just a placeholder right now). This shouldn't take too long. I would like to get the VMBuilder changes in Debian soon, and hopefully get those changes in the VMBuilder trunk[4] too.

My project was part of the Debconf talk on the Debian GSOC projects which I believe was 2 or 3 days ago. I'd look up the presentation and watch it but right now I have my friend begging me to install Ubuntu for him so he doesn't have to use Vista anymore.

[0] git://git.debian.org/git/pkg-escience/vmbuilder.git
(that's a lotta gits)
[1] http://wiki.debian.org/VMBuilder
[2] http://wiki.debian.org/euca2ools
[3] http://wiki.debian.org/DavidWendt/GSOC09Project
[4] https://launchpad.net/vmbuilder

euca2ools, kmeisthax@Lappy-486.A, and other annoyances

2009-07-17T17:33:00.000-07:00

Well, it's finally out so I can tell you all about it: euca2ools, the fabled free software EC2 API client! Unlike a certain other company's tools, these don't have onerous usage restrictions. Yes, our VMBuilder[0] already supports using it as opposed to Amazon's tools, it defaults to Euca tools then Amazon's tools. (Block the usage of either using --ec2-no-amazon-tools or --ec2-no-euca-tools, keep in mind you still need one or the other to bundle or upload.)

And another thing... I recently took a look at the git repo [0], and noticed it's littered with commits by "David Wendt (kmeisthax@Lappy-486.A)"... please ignore the screwed-up e-mail address, I reinstalled Debian and forgot to change it to my normal address. (which is dcrkid (a) yahoo () com)

[0] git://git.debian.org/git/pkg-escience/vmbuilder.git

My current method for tracking/gitifying changes in foriegn VCSes

2009-06-24T13:48:00.000-07:00

In vmbuilder.git we deal with a lot of upstream branches on launchpad being tracked with bazzar. So, I have to convert those revisions to git commits. This is how I handled importing a new branch from launchpad that branched from Bzed's r344 today:

#grab the revision

bzr branch lp:~ckaenzig/+junk/lenny

cd vmbuilder

#go to the point this revision branches from - note that this is a tag and not a branch

git checkout bzed_vmbuilder_debian-r344

#make a new branch for the underlying bzr branch

git checkout -b ckaenzig_junk_lenny

#clear out the current git working area, to reflect any RMs in the bzr revision

git rm * -r

#clear out any untracked files, such as .pyc and .py~

rm * -rf

#copy the revision over to the git repo, and stage it

cp ../lenny/* . -r

git add * -r

#make a new commit on this branch, which should now reflect the underlying bzr revision.

git commit -m "New bzr branch lp:~ckaenzig/+junk/lenny"

#tag the commit by it's underlying bzr revision number for easy reference

git tag ckaenzig_junk_lenny-r346 -a

This process generally works, but it requires careful attention and is easy to screw up. For example, I have to do a second rm to remove files that aren't being tracked by git (.pyc, .py~ files) , otherwise I end up committing junk that might be from another branch entirely when I stage the new revision. Not to mention that I may miss a revision this way. Say I have a branch A in bzr that I've manually imported this way for revisions r100, r123, and I wanna import branch B which breaks off from branch A at r112, I have to either put the new git branch at r100 (and have slightly inaccurate commit history) or start moving commits around, which I've never tried and probably doesn't push well. Fortunately VMbuilder doesn't get that many commits per day so I usually am able to keep on top of 'em.

Is there an easier way to convert bzr revisions into git commits? I'm horribly unfamiliar with bzr, so I haven't really investigated automating this process.

.euca//GIFT

2009-06-21T21:35:00.000-07:00

Over the weekend Stephen Moeller contributed a bunch of patches to the Vmbuilder repository for Virtualbox support as well as fixing up the Debian packaging. You can examine them at the usual place(1), but be forewarned that I haven't tested them just yet. I'll get these off to Bzed tomorrow once I can be bothered to go downstairs and hook up an ethernet cable for Debian. And, of course, the usual machinations of pulling in changes from the vmbuilder trunk and bzed debian branch.

(1) git://git.debian.org/git/pkg-escience/vmbuilder.git

P.S. A little background - my Dell Studio 1535, while otherwise being a good machine, is cursed with a Broadcom Wifi card. I would rather not install non-free drivers on my Debian install after having a few bad experiences using the NVIDIA drivers on Debian on an older machine. So instead I go downstairs and physically connect the Lappy-486 to the home network.

yay mass commit bombs!

2009-06-18T15:14:00.000-07:00

More work on VMbuilder - after fixing how the Debian plugin finds Xen kernels I got it to build a Xen image. (The Debian plugin was looking for images the way Ubuntu versions them, not the way Debian versions them) Now I need to figure out building EC2 images and such.

And send bzed a patched distro.py.

Epic merge fail

2009-06-16T14:20:00.000-07:00

(seven less than symbols since blogger can't handle them)

#import sys

#import os

#sys.path += [os.getcwd()]

import VMBuilder

=======

class Suite(object):

def __init__(self, vm):

self.vm = vm

self.isodir = '/media/vmbuilder_inst_image'

self.iso_mounted = False

(seven greater than symbols since blogger can't handle them)vmbuilder_trunk:VMBuilder/plugins/ubuntu/suite.py

This happened to come from my attempt to merge VMbuilder trunk 0.11 with the current master; Git thought suite.py was moved to debian-vm-builder and tried to merge it.

Oh and yes that also means VMbuilder trunk 0.11 is now in alioth. I can't believe I missed it. It includes all the EC2 stuff without the install_[hardy,intrepid,jaunty] cruft as well.

So, to-do list:

Get Debian plugins working with EC2 VMbuilder. The debian plugin branch on Launchpad hasn't been updated since June 6th; maybe bzed maintains another repo?
Try building the package and installing it. Side note: They removed packaging info from the trunk midway between 0.10 and 0.11, but I kept the /debian folder anyway in vmbuilder.git.
Speaking of which during the merge I noticed the default sudoers template explicitly references an "ubuntu" group, something that should be changed eventually.
Look into the vmbuilder-gui branch sometime.

fs weirdness

2009-06-16T10:18:00.001-07:00

So I was in the middle of refreshing my vmbuiler sources when I got an out-of-disk error in the middle of the bzr pull. Fine, so I fired up Filelight, excluded /media and checked to see how much was being used... 5GBs. On a 19GB partition. Feeling this to be a bit odd I figured I'd rather extend the partition another gigabyte. I fire up the Ubuntu partiton, load gparted, and.... it reports the Debian partition as being 5GBs used. Odd. So I extend the partition back a gig, which takes a great deal of time. That's done, so I remount the new debian, fixup grub, restart, and get a kernel panic.

It is at this point that I remember that I had hibernated Debian instaed of shutting it down, and then I moved the filesystem out from under it. So I reboot again, it works. First thing I do after login is check "df"... and the drive reports 5GBs used.

I'm not a master at kernel internals but I have at least part of an inkling that the missing 14 gigabytes of space had something to do with hibernating and restoring the machine multiple times.

In other news it looks like there is a branch vmbuilder-gui, which I'll have to look into sometime.

VMbuilder delay

2009-06-11T19:31:00.000-07:00

I said I expected to have a working 'beta' of vmbuilder that could make AMIs of Debian by Friday. Well I don't think I can get it to you in time, because of the way that vmbuilder works. You see, the core VMbuilder has a nice system to support future distros sanely. So I could easily extend it to create VMs for pretty much anything that uses the Linux kernel.

But... EC2 has very specific pre-install/post-install actions that are hard-coded to three different Ubuntu distributions. So I need to talk to the EC2 plugin guys about how to refactor the thing to allow the distro plugin to offer specific EC2 actions. I _could_ just add preinstall/install_etch/lenny functions, but that would be a copout. I suspect this is the reason why the EC2 plugin isn't in VMbuilder trunk.

And as a final kink, it uses ec2-tools to bundle and upload the image, and a python library called Boto to register it in the VM deploy code. So I need to get patches in Boto - or find someone who's trying to get patches in it - to allow it to connect to Eucalyptus.

On the other hand, I found someone who's already written Debian plugins that work with regular (non-EC2/Euca) VMs. It only supports Etch, but Lenny shouldn't be that different from Etch.

So let's just say when I said Friday on the SOC coordination mailing list, it was in Valve Time.

Previous attempts at vmbuilder-debian

2009-06-09T13:27:00.000-07:00

Eric Hammond informed me of a previously-existing branch of vmbuilder which was adding Debian support. It currently has only etch support, but I decided to work off that.

I've merged it into the existing Git which is currently based off the Vmbuilder trunk. There's not much difference between the two anyway: the trunk has packaging that the branch doesn't, and the branch has a Debian plugin and frontend which the trunk doesn't.

Now to set it up and see if it works.

Stop clobbering my bootloader!

2009-06-01T21:39:00.000-07:00

During this past weekend I tried to get Windows XP, Windows 7, OSX, Debian, and Ubuntu all on one machine. Due to some mishaps, driver oddities with Windows 7, and other things, I had to install certain operating systems multiple times. Now, the problem is, each operating system has it's own peculiar way of loading. Linux uses GRUB, Windows XP has NTLDR, 7 has the Windows Boot Manager, etc. Most of these operating systems also decide it would be helpful for the end-user that when his computer reboots, it can actually load the new OS. That's great, but I already have a bootloader.

I have a partition at /dev/sda5 which contains a GRUB installation. That's supposed to handle every OS's own method of booting, including the Linux installations which have their own GRUBs on their own partitions. But when you decide to overwrite the MBR, I don't get your nice and shiny OS. A good number of OSes use an MBR that won't boot an extended partition anyway. So instead of seeing your bright and shiny new OS, I see "NTLDR is missing". Or the ever-so-helpful "boot0: error". I'm -lucky- if I even get an OS at all, much less one that can load the grub shell so I can fix what you broke.

I understand having 3 GRUB installations plus a whole bunch of other bootloaders isn't exactly a standard use case. Most people don't go beyond dual-boot. But please, for the love of all that is holy, please just put a little checkmark somewhere,"Do not replace MBR". Ubuntu and Debian are the only OSes that actually give you that option in some form.

Or better yet, let's all just standardize on GRUB. Sure, it's not perfect, but it's as flexible as you can get.

Bad journalism that's suprisingly related to my project

2009-05-29T18:50:00.001-07:00

These guys could do at least 10 minutes of research before writing an article. The AWS APIs are already 'open source', you're allowed to write free-software tools that use them. They're out in the open, with no NDA or anything attached. Eucalyptus also exists, which is a pretty darn good reimplementation of EC2. The only real impediment to an 'open' (you meant free, right?) cloud is that you can't use ec2-tools with it unless you want to fight off a pack of female Amazon warrior-lawyers. I hear their punches can leave a mark.

I'm too lazy to send them an e-mail, and the Slashdot comments are already calling 'em out anyway.

awstool.py

2009-05-27T12:52:00.000-07:00

Since I can't use a Eucalyptus cluster without breaking the usage restrictions on the default ec2-tools apps, I'm writing a set of Python scripts to do the same job. The APIs that the tools use are well-documented and don't have onerous hand-tying usage restrictions.

So that should be 'fully functional' by next week. By fully functional I mean being able to bundle an AMI, upload it to a cloud, and manage it's instances without crashing or screwing up. I'm probably not going to bother supporting stuff that Eucalyptus doesn't support, since the main focus of this is Eucalyptus and not Amazon EC2.

I'll also include a command for saving cloud settings, i.e. "awstool savecloud --ec2-cert --cert --url rightscale" which then lets you do something like "awstool upload-image --cloud rightscale debian-lenny.ami".

Also, pkg-escience has agreed to host the Eucalyptus integration project, so you can follow along at:

git://git.debian.org/git/pkg-escience/awstool.git

Ignore the first commit's e-mail address of 'kmeisthax@Lappy-486'; I forgot to set some config somewhere before I did the initial push.

I can't believe it did that

2009-05-10T23:19:00.000-07:00

Please excuse the double-post. I added the tag "eucalyptus-integration" to my previous post in order to ensure that I could separate the feed going to Debian's GSOC planet, and the public project feed. "To-debian"/"to-debian-gsoc" is for posts that appear on the Planet, and "eucalyptus-integration" is for posts relating to the integration work itself, and -that- feed will be going on the public project page on socghop.appspot.com.

For example, this is one of those things that ONLY appear on Planet Debian/GSOC, since I just double-posted accidentally.

It appears that Planet assumes that any change in a post is a new post that needs to be pushed to the front of the site. I'll refrain from editing old posts, then.

Sound Check

2009-05-01T21:08:00.001-07:00

Now that Arthur got my blog on the GSoC planet, I can finally post updates about Eucalyptus and Debian. Speaking of which, after installing Debian, that makes it, what, four OSs on my laptop now?

One thing that's really bugging me, is ec2-tools. To reiterate my point, the license says plain and simple you can only use it with Amazon EC2, yet Ubuntu's community documentation clearly recommends installing them and using them contrary to the license terms. I've heard the only reason why the ec2-tools software even has a license at all was at the behest of Canonical or something. Has anyone seen something that duplicates the ec2-tools functionality as a command line tool?

At any rate, I won't be updating this blog much until finals season ends and the GSoC project officially 'starts'. In completely off-topic thoughts, since when were the Jonas Brothers considered a high-trend Twitter topic? I thought little kids weren't allowed to have Twitter accounts.

Fr1st Ps0t

2009-04-27T08:56:00.000-07:00

Hello, my name is David Wendt, and I am a GSoC09'er. This year I will be working on integrating Eucalyptus/EC2 into Debian.

What does that mean?! Well... I want to have Eucalyptus in Debian, I want to be able to have the Debian maintainers release AMIs which will work with a Eucalyptus or EC2 cluster, and I want to package all the other useful commandline tools such as vmbuilder. However, Debian can't package ec2-tools since it's actually illegal to use them with anything other than EC2 according to the onerously DFSG-violating license agreement. So, depending on the effort required, I might end up making replacements for the ec2-tools. (I haven't even downloaded those things, to avoid violating the license.)

Some background, for laypersons:
Amazon EC2 is a commercial on-demand computing system which uses the Xen VM to rent out computing resources. Eucalyptus is an open-source cloud/grid computing system which currently emulates the EC2 interface. Vmbuilder is an Ubuntu app package which lets people build and modify their own VM images. ec2-tools is a set of command-line tools to interface with EC2, currently under an onerous licensing restriction.

Eucalyptus is currently not in any Debian repository (unstable, testing, or stable) and the only mention of it is on the prospective packages list with a report filed half a year ago, so I suspect this might take some work to get Eucalypus fully debianized. However, I am up to the challenge.

Sincerely, David.

P.S. This is not a troll. Also Ron Paul.