Ian Haken's Home

Snail-mail is (arguably) less buggy

Fri, 25 Jan 2008 10:57:22 UTC

If anyone tried to email me via ianhaken.com in the last three weeks, you probably met with little success. My contract for dedicated hosting ended at the beginning of the year, and for my small personal uses it wasn't worth the cost to renew it. Since I already had an always-on file server at home, I decided to move my hosting services to my own server. After some downtime associated with transfering domain registrars, I was able to get this much setup.

Since I'm just hosting on a cheap ISP connection, I of course don't get a static IP. Many free DNS services offer dynamic IP update clients which solve this problem, but of course you can't quite take advantage of your personal domain name. I took advantage of a tip I received to redirect all hostname lookups with a CNAME entry to my dynamic DNS domain name, which stays up-to-date with the free client. So in short, a little indirection was able to solve the dynamic IP problem.

What I failed to realize is the affect this would have on my MX records. While I had them properly configured for my domain name, I realized that mail was still not getting through. It took me longer than it should have to perform a manual MX lookup and figure out that my CNAME was being resolved and then MX records were being searched for this canonical name rather than ianhaken.com. So after adding identical MX records to my dynamic IP service all seems to be well again.

On a related note, I've also been trying to get exim working on my internal server. Since I do run a dynamic IP I can't run a full-blown mail-server, but I'll I really want to do is be able to send outbound messages anyway. I've been able to make this happen by configuring exim use gmail as a smarthost. Ideally, I would also like local mail to be delivered to my external mailbox. I figured this would do it:

echo myname@gmail.com > ~/.forward

and it seems to get half way there. Sending a message to myname@localhost results in the following lines in my log:

2008-01-06 13:50:46 1JBdOI-0005sm-4Q <= myname@localhost U=myname P=local S=388 2008-01-06 13:50:48 1JBdOI-0005sm-4Q => myname@gmail.com R=smarthost T=remote_smtp_smarthost H=gmail-smtp.l.google.com [209.85.199.109] X=TLS-1.0:RSA_3DES_EDE_CBC_SHA1:24 DN="C=US,ST=California,L=Mountain View,O=Google Inc,CN=smtp.gmail.com" 2008-01-06 13:50:48 1JBdOI-0005sm-4Q Completed

By the look of it, exim realizes it is supposed to be delivering out to the smarthost, but it's still sending to myname@localhost. At any rate, I'm definitely not seeing anything in my inbox on either end. Anyone know if I'm just not understanding the .forward file or is there something in the exim config I need to change? It's not particularly critical, but it would be nice to see error mail go to my inbox.

Trials and Tribulations of ARM Programming

Wed, 19 Dec 2007 12:32:36 UTC

I spent a rather long time today trying to figure out a rather obscure bug. I few months ago I bought a gp2x, an open portable gaming system (and media player, etc.). I've really enjoyed it so far. The open platform for development has lent itself it many fun apps, console emulators the most conspicuous I think, but it's also fun to experiment and develop with it. I recently ported an asteroids clone I wrote and the whole process was actually fairly simple. As such, I decided to take on a larger task of porting something like FreeCNC, which as far as I know, no one has done yet.

After some simple trouble with library version conflicts, I got the program compiling correctly. However, if wasn't reading files quite right. I spent a little while investigating and discovered the problem was in a function which essentially hashes filenames. In the end, it was this snippet of code which was giving the trouble:

char buffer[13]; ... calc = ROL(calc)+*(long *)(buffer+i)

In particular, this is trying to read 4 bytes from this character array as an integer. At first I though the trouble might be endian-ness, but both the gp2x (an ARM processor) and x86 use big-endian^*. I finally resorted to running gdb and sifting through assembly level instructions.

$r2 = 0xbffffd7c $r3 = 0x41434f4c 0x0000a310 <_ZN8MIXFiles7calc_idEPc+52>: str r3, [r2]

Here, the register $r2 points to buffer and $r3 is the value we're trying to save into the buffer. After this we would expect
*0xbffffd7c = 0x41434f4c
but instead see
*0xbffffd7c = 0x08db9841.
If I purposely set *0xbffffd7c = 0xffffffff and repeat this instruction we see
*0xbffffd7c = 0xffffff41.

Clearly the problem lies in this single str instruction, and was resident to the ARM architecture. So of course it was time to go to the processor documentation. Reading the fine print finally reveals what I was looking for.

A word store (STR) should generate a word aligned address. The word presented to the data bus is not affected if the address is not word aligned. That is, bit 31 of the register being stored always appears on data bus output 31.

Yep, since the buffer was an odd-sized array and stored on the stack, it started on an odd number in memory. Hence its address was not word aligned, and so as per the documentation, the store did not have the expected result. Here's a short program which shows what I'm talking about.

x86	ARM
`buf=BFBF3471, buf%4=1 Num: 4C4F4341 Buf+0: 4C4F4341 Buf+1: 4C4F4341 Buf+2: 4C4F4341 Buf+3: 4C4F4341`	`buf=BFFFFD75, buf%4=1 Num: 4C4F4341 Buf+0: 4F434100 Buf+1: 43410000 Buf+2: 41000000 Buf+3: 4C4F4341`

Perhaps the lesson to learn here is that when casting pointers, it's buyer-beware, and there are safer bitwise manipulations to use, though certainly less efficient.

^*Edit: Both x86 and the ARM processor in question are little-endian, not big-endian. Typo on my part, but notwithstanding the point is that they both had the same endian-ness, so it wasn't where the issue lay.

My Internet Map Reads: Here There Be Monsters

Tue, 13 Nov 2007 05:26:02 UTC

For anyone running a UNIX desktop/server and who is as slow as I am, I would suggest enforcing something like fail2ban on your machine. Otherwise you end up with bunch of guys in your internets, hacking all your boxes(1.7MB). I only just took a look at logs today, but apparently you should only venture out onto the internet prepared.

Someone pointed out to me that sshd_config also offers an AllowedUsers option, which is useful if you only want some users on SSH (you may have some users using others services such as Samba which you want totally restricted from SSH). Fail2ban more directly addresses my issue, which is some obvious brute force attempts (one of the IPs in that log had over 3000 login attempts). Fail2ban actually inserts a temporary DROP line into your iptables after a few failed login attempts, so I should hopefully only be seeing a few lines from each attacker instead of several thousand.

Long Time, No Update

Fri, 19 Oct 2007 21:12:14 UTC

So I've clearly neglected giving frequent updates, so I'll try to summarize everything that's been going on and maybe try to post more frequently in the future (I feel like I've said that many times before).

The summer ended very well, I think. My GSoC project vlosuts concluded with a fully working version, and so hopefully some guys over at Quality Assurance can be goaded into giving it a try. Once Lars and I find some time (it seems like both our lives have been quite hectic) we'll be submitting it to the unstable branch. I intend to continue supporting and developing it; in fact last night I just checked in the first update for VirtualBox support, as recommended by Jon in my comments. It does seem to work faster than QEMU and Innotek did a quality job (in my opinion at least) of packaging it. The biggest kink is that it does not support raw disk images, and so vlosuts gets slowed down by having to convert to VMDK and back, for which I am using qemu-img (and does anyone know why VBoxManage unregisterimage doesn't seem to be working for VMDKs?). If it's possible to do a loopback mount of VMDK (without VMware's utility) it would probably make more sense to use VMDK instead of a raw image to begin with, but I'm not familiar with any way to do this. At any rate, first version of VirtualBox is there for anyone who'd like to give it a shot.

Anyone not concerned with the personal going-ons of my life should stop reading right about now. Since my semester started up again at the end of August, I've pretty much hit the ground running. I'm taking 20 units (I think the average is around 15), one of which is a mathematics graduate course. I'm also doing a research project in computer science, working a part time job (which I try to do for about 10 hours a week), and getting my graduate school applications taken care of. For anyone who is curious, the GRE is a terrible thing. Graduate school and fellowship applications have been a particular headache, but fortunately I'm just about finished up with the essays. For the curious I'll be applying to mathematics departments with an emphasis in logic and foundations at the following universities (list subject to change frequently): MIT, Berkeley, Stanford, Carnegie Mellon, Urbana-Champaign, UMich.

On Power Outages and Package Building

Wed, 25 Jul 2007 18:48:23 UTC

I'm sure few would disagree with me when I say that power outages suck. But on the bright side it gives me a chance to test the resiliency of my server setup. I haven't bought a UPS yet, mostly because I'm waiting until my next bulk purchase of hardware.

When I logged onto my home system from the office ssh gave me a friendly notice that a new IP for this hostname had been added to the list of known hosts. This flagged my attention since although I have a dynamic IP with my ISP, they haven't given me a different one for months. But it's reassuring to know that the program for updating automatically the DNS entry is working. I quickly figured out that there was definitely something wrong when my screen sessions were gone and PIDs were small. Sure enough, w reports the system has only been up for an hour and a half. Good to know the Full On BIOS setting works as well. Really aside from losing screen sessions, there was no real notice of the power outage. Well, that and I had to reset my alarm clock when I got home.

But back to the topic subject of summer of code, vlosuts is just about out of alpha now. For anyone wanting to try it out, here's the set of commands that'll build and install the package.

cd /tmp svn co svn://svn.debian.org/vlosuts/vlosuts cd vlosuts debuild cd .. sudo dpkg -i vlosuts_0.1_*.deb sudo vlosuts upgrade --config=/usr/lib/vlosuts/default.cfg

I haven't added debootstrap as a dependency yet, so make sure you've got that. Also make sure you have qemu/kqemu/kvm installed (depending on what backend you want to use; default is QEMU). Notice that vlosuts needs to be run with superuser privileges. This is a consequence of needing to mount and bootstrap the image. The "upgrade" part of the command line instructs it to perform an upgrade test (as opposed to just building an image or creating a vlosuts mirror file of the system).
You'll probably want to copy that config file and change the mirror to something local. Also make sure you've modprobed kqemu (or else it could take far longer than necessary) and if you're running amd64, either change the target arch to amd64 or set the qemu path in the config to /usr/bin/qemu-system-x86_64 (as of writing, kqemu on amd64 only works with this binary, not qemu). man vlosuts can also provide some information.

Summer of Code Midterm Update

Tue, 10 Jul 2007 04:19:04 UTC

As we reach mid-summer, I think now is a good time to provide a more detailed update on my summer of code project, vlosuts. For anyone not familiar (or for anyone forgetful) the following is a brief summary of my project proposal. In order to assist in distribution upgrades, vlosuts (Virtual Live Operating System Upgrade Test Suite) aims at performing upgrades of live systems (between releases and/or repositories) while logging any errors encountered. By running on a live system (virtualized via QEMU/KVM/Xen/other) problems that may only be encountered with services upgrading can be identified early. The full proposal can be found at: http://code.google.com/soc/2007/debian/appinfo.html?csaid=24C7766EC4271F15

I am happy to say that vlosuts is successfully in alpha, with most of the desired features already implemented. Based on my initial project propasal timeline, I'm well on schedule.

Some of the features that were proposed included the ability to run the upgrade with packages selected by priority, from a list, and as mirrored from an existing system (packages can even have a specific version specified, though of course the repository used must have that version available). Additionally, when mirroring this package list, configuration files can optionally saved as well. This allows testers to (as nearly as possible) reproduce a specific system.
These options have all been implemented among others. I have also put in support for running with QEMU, KVM, and Xen backends.

For the remainder of the summer, my goal will be adding additional features and tackling a few bugs that have come up. In particular I'm trying to take advantage of QEMU's ability to emulate various architectures. However, the stability of user-mode emulation has been uneven for me (segmentation faults whenever any program is launched), which is required in order to cross bootstrap. Once this is worked out, one should be able to test upgrades on any of QEMU's supported architectures.

Additionally, vlosuts is not yet available as a package, which is clearly something that needs to be fixed soon. I'll be spending some time in the next couple weeks packaging it and making sure it sets up correctly.

I think that about sums up the current state of things. It's been a good experience thus far, and hopefully the rest of the summer continues to go well.

A vlosuts update

Tue, 26 Jun 2007 06:43:29 UTC

I'd say vlosuts is just about alpha at this point. It perfectly manages building an image and then upgrading (at least on the systems I've tested it on) and I haven't seen any major bugs. The main feature I'd like to get working is support for a Xen backend. This shouldn't be too difficult, it's just taking me a bit of effort to get Xen working. It would've been easy enough, but I just couldn't get the network configured. I followed guide after guide, and finally figured out the problem was that my wireless card didn't support bridging. Which explains why I'd never managed to get that to work in the past either. So now I'm reconfiguring my Xen setup with NAT, which should give me a system good enough for testing. After that I'll start fixing some of the less important bugs and adding some features here and there.

Foreign chrooting

Tue, 19 Jun 2007 06:12:00 UTC

As part of testing upgrades, I'm trying to build disk images for foreign architectures. debootstrap already has the first-stage second-stage framework for this, but of course you have to be able to boot a system in the target architecture to finish the second stage. I've tried a few ways of accomplishing this.

The first is extracting a kernel from a package and using qemu-system to boot that. Trouble there is that AFAIK there aren't prebuilt initrds and so the root image can't be mounted.

The second option occurred to me that I could use user mode emulation to do the necessary second-stage bootstrapping. You'd still need to be able to chroot into the system, though, and so I tried this out by statically building qemu binaries. This didn't seem to compile too cleanly, but after some searching I discovered that what I was trying to do had already been done over here. Seems like a pretty good solution. Only thing is if I go with this method, it would require those packages as prerequisities which aren't in the Debian repository.

If anyone has any additional suggestions on bootstrapping for a foreign architecture, let me know.

GSoC, vlosuts, and other acronyms

Tue, 29 May 2007 02:16:01 UTC

Summer of Code officially started today, and as such it was time to start hacking out some code for it. For the past couple of weeks I've been casually looking over an upgrade testing suite for Ubuntu (website, bzr repo). It's fairly sophisticated and does a lot of things that I'd like to bring to Debian. However, after considering it for awhile, I decided that I would prefer to start on the Debian version from scratch. This decision came about for a few reasons:

The Ubuntu version focuses on upgrading between releases, whereas I would like to possibly upgrade between entirely separate repositories. While this could fairly easily be added as a configuration parameter, it was still a point of consideration.
I would like to write the scripts for building the image as a seperate module. This would aid in the future goal of snapshotting and rebuilding systems based on a list of installed packages, versions, and config files.
Until recently the Ubuntu upgrade tester used chroot and had no virtualization option. In addition to having this, the aforementioned process of building the initial image independent of backend should make the resulting scripts easily pluggable into any virtualization option.

At any rate, I will probably end up reusing many pieces of the code from that tester.

Updates posted here as progress continues.

GSoC 2007: Project is Underway

Wed, 16 May 2007 01:38:45 UTC

I've gotten started on setup for my Google Summer of Code project (original abstract for anyone unfamiliar), originally titled "Automated Upgrade Testing Using QEMU." In thinking about the direction I'd like to take the project, I've created an Alioth project with the title "Virtual Live OS Upgrade Testing Suite" (short name vlosuts). As discussed briefly in the GSoC application comments, the system should have a plugable backend so that one could use QEMU, Xen, etc. as preferred, and so I dropped QEMU from the title in favor of the word Virtual.

The project itself is still in a planning phase, but Ubuntu has a similar package using chroot which may serve a good basis for what I'm doing. I will be making periodic updates of my progress via this blog, but status and mailing lists should also be available at the Alioth project: http://vlosuts.alioth.debian.org/.