Planet StreetGeek

First part
Second part

This is an ongoing saga, and if you don't know what this is part 3 of, the previous two parts are linked above.

Right, so we know where the file starts, we know where it ends, and we know that the file should be pretty much linear as the card was empty. Screw the FAT, let's just read this thing from the clusters.

#!/usr/bin/env python
import sys
numReservedSectors = 38
bytesPerSector = 512
sectorsPerCluster = 8
numFATs = 2
fat1length = 7771648

def clusterToBytes(cluster):
    firstDataSector = (numReservedSectors * bytesPerSector) + (numFATs * fat1length)
    address = (((cluster - 2) * sectorsPerCluster) * bytesPerSector) + firstDataSector
    return address

print clusterToBytes(0x0005CDF3)

Bring in some values. Define a function to convert clusters to bytes.
From the first bit, we know where it starts.

File found: 4621 clusters (18 MB), start 0005CDF3

So run the program and we get:
1573685248 which is 0x5DCC8800

From the second bit, we know that it is supposed to end at 0x05EED5800. Then we found that the data really ends at 0x0614B3600.

start = 0x05DCC8800
middle = 0x05EED5800
end = 0x0614B3600

size = end - start

f = open(sys.argv[1],'r')
f.seek(start)
a = f.read(size)
f.close()

f = open("recovered.avi",'w')
f.write(a)
f.close()

Running this results in many more frames and a bigger audio track.

Here's the last frame:

and here's the video:

Recovered Crash Video final version from gm on Vimeo.

This is a continuation of yesterday's post "Fun with FAT32, and part 2 in the process of getting that video back.

OK, so we have two promising looking files that don't play. Let's dive into the AVI file format.

Microsoft have a published spec for it here.

An AVI file is really a RIFF file. The basic structure of a RIFF file is a 4 byte identifier, followed by a 32-bit size. Then a length field, then a pad byte if the length isn't even. You then have a header with fields showing info like what codec etc etc. I'm not going to go into this in detail, because it's not actually important here (trust me). The one thing that is is that "DWORD", supposedly meaning "double word", a word being 32 bits, is not 64 bits. Microsoft compilers treat WORD and DWORD types as the same, or so I'm told. I'm just going to say "32 bit value" or "unsigned int", but even then, on some 64 bit platforms, ints are 64 bit, so be careful.

The code will be available in the SVN. Also excuse the hardcodedness of a lot of things in this program - AVI files from the same camera don't tend to differ. Fat32 volumes formatted in many different things do.

So I made a dodgy program to dump data from the headers and it would appear that our two mystery files are AVI files, but there's some stuff missing. The header stuff is in exactly the same position - my program just assumes this as I wasn't going to write a complete and proper RIFF implementation. So it likely won't work on AVI files not from this camera.



Playable one on the left, unplayable one on the right.

0x0, 0fps, 0 frames long. While the resolution is certainly known from the start, the number of frames quite reasonably isn't.

I attempted to drop reasonable values into the file, but to no avail - it still didn't play. In MPlayer, Quicktime or VLC. VLC offered to repair the AVI index, and promptly crashed upon doing so. Oh well.

The reason why previously I said that it didn't matter about the header is that I am just going to rip the data out of the AVI and deal with it elsewhere.

import sys
import block
import struct
import re

f = open(sys.argv[1],'r')
file = f.read()

Yes we do read the file in. What do you mean you don't have 500MB of RAM?

Now some calculations:
The RIFF chunk with the actual data in it is named "movi".

movicount = file.count("movi")
print "movi block count: %i" % movicount
moviloc = file.find("movi")
print "movi block location: %i" % moviloc

There should only be one instance of "movi" in the file. It's an outstanding coincidence if this happens in the video data. Or it's annoying if it happens in the header. We'll just hope it doesn't.

print "regexing..."
avidata = file[moviloc+4:]
avichunks = re.split("(00dc|01wb)",avidata)[1:]

chunktype = avichunks[0::2]
chunkdata = avichunks[1::2]

First, we get the entire file from 4 bytes after the "movi" chunk. We ignore it's size field - remember that this is a not properly finished file we are dealing with, and fields such as these may not be present yet.
Now for some regex. We need to use re.split because the string split method doesn't support splitting on multiple things, and loses what it was split by.
The way AVI works is that it interleaves audio and video data. This means that reading and writing can be done in one stream, greatly simplifying the recording and playback process. There are no seperate reads or seek operations to two seperate parts of the file (in the days AVI was new, we played fancy multimedia CD-ROMs. Seeking on CD-ROMs is horribly slow, but linear reads weren't so bad.). We have RIFF chunks with types indicating what they are.

The first two digits identify the track number. dc means compressed video, and wb means audio data. There are four such codes - read the earlier linked spec to find out about them.

You can have an overcomplicated regex here like "(\d\d)(dc|wb)(.*)" - I did originally, but it was actually matching movie data. There were no false matches with this one, and no tracks we are interested in besides the video and audio tracks.

Anyway, this results in a list like so:
["","00dc","block of video data","00dc","more video data","01wb","audio data"]
and so on.

Next, if you don't know, in Python, array indexes can have up to 3 values.

a[1] = item 1 of a (second item, arrays are indexed from 0)
a[:] = all of A (same as "a")
a[1:] = from item 1 onwards
a[1:3] = from 1 to 3
a[-1] = last value

The third number is "every n".

a[0::2] is every second value starting at 0, or every odd value.
a[1::2] is every even value.

a[::-1] is the array in reverse order.

Anyway, we drop the first value of the array because it's an empty string.
chunktype is the first element of every pair, chunkdata is the second.

Now let's do something with them:

streams = {}
streams["00dc"] = []
streams["01wb"] = []

for i in xrange(len(chunktype)):
    stype = chunktype[i]
    sdata = chunkdata[i]
    streams[stype] += [sdata]

We initialize the dictionary "streams" with two lists. Then we go through all the chunks, and place them at the end of the list they belong to.

Now let's do something with this dictionary.

for stream in streams.keys():
    f = open("streams/%s" % stream,'w')
    print "writing streams/%s" % stream
    for block in streams[stream]:
        f.write(block)
    f.close()

Right, let's do this.

Promising.

Now we know that the video is motion JPEG, and that the audio is uncompressed. Let's look at one of the known good movies:

I'll leave the video for later. Let's deal with the audio first. I'll use Audacity for this.

And what do we have?

It sounds like the radian... but what is that fast clicking in the background?
There seems to be some extra bytes in here. Ahh... we forgot about the size bytes after all the audio blocks!

Change the line where the blocks are written to this, and run again.

f.write(block[4:])

And no clicking. I can hear the Radian's motor running flat out and producing some very distorted sound, followed by it shutting off, and silence. Then it being run shortly... shouldn't there be some wind in here? I skip to the end, expecting the recording to end with a loud bang... it ends with me saying something after the motor winding down. This isn't the video - but why isn't it in the filesystem? I remember pulling the power too soon on a test run of my camera. That must be it. It must be the unidentified 18MB file, not the 70MB one. Or it could be neither of them. Hmm.

Let's run it on the other file.

It only produces 28 seconds of sound.

The first few spikes are the canopy being put on, followed by the motor starting at 9 seconds in, and the throttle being maxed out at 18 seconds. The video ends 8 seconds later, still at full throttle. This doesn't sound like a test. I'm not saying anything. But I can't hear wind noise in it. And I could have sworn it took longer than 18 seconds from video start to crash the plane. I check another video - and it seems that at full throttle, there isn't wind noise - it's not audible above the motor.

I guess there's one way to find out - let's mess with the video. VLC won't touch the 00dc file. MPlayer doesn't like it. Quicktime... wait what? A video!

No. A frame. Damnit.

But it's the video. I'm turning the camera on from the transmitter. I'm standing where I was standing when I launched the plane for it's disasterous flight. This is the video.

Now let's take a look at the data.

FF D8? That sounds oddly familiar. A quick Google search for "jpeg magic bytes" finds that it is the marker for the start of a JPEG file. FF D9 is supposed to be the end. I search for it. Not found. But FF D8 is in this file... a lot.

A search for Motion JPEG format finds this page from the Planning for the Library of Congress Collections. It doesn't say much about the video format itself, but it does say:

Avery Lee, writing in the rec.video.desktop newsgroup in 2001, commented that "MJPEG, or at least the MJPEG in AVIs having the MJPG fourcc, is restricted JPEG with a fixed -- and *omitted* -- Huffman table. The JPEG must be YCbCr colorspace, it must be 4:2:2, and it must use basic Huffman encoding, not arithmetic or progressive. . . . You can indeed extract the MJPEG frames and decode them with a regular JPEG decoder, but you have to prepend the DHT segment to them, or else the decoder won't have any idea how to decompress the data. The exact table necessary is given in the OpenDML spec."

Interesting.

I take a rough guess that I should split all the JPEGs out at FF D8.

#!/usr/bin/env python
import sys
import block
import struct
import re

f = open(sys.argv[1],'r')
file = f.read()
file = file.split("\xFF\xD8") # JPEG magic bytes

index = 0
for frame in file:
   &nbspf = open("frames/%04i.jpg" % index, 'w')
   &nbspf.write("\xFF\xD8" + frame)
   &nbspf.close()
   &nbspindex += 1

Not too complicated.

Hmm, they seem to have icons. It's obvious why. Finder must be using the standard Apple libraries which are probably descended off Quicktime. Which when Quicktime notices that the JPEG has no DHT segment, uses the one from it's Motion JPEG decoder. Brilliant. Except that all these pictures can be displayed in Safari. Wait, that's an Apple app. Chromium? Still works. Imagemagick even displays it!

I look at the Wikipedia article for JPEG. FF C4 is the marker for a Huffman table. And there's one there. I know what's going on here. The camera's chipset is feeding it JPEGs, and they're just bit-stuffing them into an AVI, at a rate of 30 of them every second. I guess none of the MJPEG decoders are picky.

So, after deleting 0000.jpg (it's the bit of data from the start), I import the image sequence into QuickTime Pro at 30fps. It looks a little bit fast. I'm not sure why at this point, but I combine the audio and it doesn't match up. So I bring both into Final Cut Pro. I note the duration of the audio track, and slow the video track down to match the length. The video matches the audio perfectly. And I see for the first time, the video. However, it runs out of frames in the air, pointed upwards.

Recovered crash video attempt 1 from gm on Vimeo.

Did the camera not finish writing to the card? I remember from the firmware upgrade that it has 8MB of DRAM in it. It's probably not storing anywhere near 8MB of data in the RAM, most likely a few JPEG frames until it has enough to write a block into the AVI. Besides, 8MB would only be a few seconds. It felt like longer from that point to where the plane came down.

I remember though, from the first step.

Traversing files...
Error: nextptr 00004000 is unallocated
Error: nextptr 0005E000 is unallocated

I wonder if it's writing the FAT one sector at a time. This would mean that the sectors are not actually marked as used, but that there is data there - after all it had to go somewhere. This was the last video of the day, and the card was not mounted prior to imaging, so if there is - it hasn't been overwritten. And since it's FAT, the allocation should be reasonably linear. Time to go back to stage 1 I guess. It will be obvious if the zeros are on a 512 byte boundary.

Convert endianness, and 00E00500 is present in the FAT.

It is. The next line is 0x00017CC00 = 1559552. 1559552 / 512 = 3046 exactly. That's a block boundary alright.
(You are a hardcore geek if you know 512 = 0x200 and recognised it was a multiple from looking at the hex offset value).

0x0005E000 = 385024.
Remembering from the program, the address of a cluster is:
firstDataSector = (numReservedSectors * bytesPerSector) + (numFATs * fat1length)
address = (((cluster - 2) * sectorsPerCluster) * bytesPerSector) + firstDataSector

So this gives us:
((385024 - 2) * 8 * 512) + (38 * 512) + (2 * 7771648)
1577050112 + 19456 + 15543296
= 1592612864
= 0x5EED5800

There's more data here, and this data is identical to the end of 33.avi.

Not much further down, there's a 00dc (AVI image data block), 4 size bytes, and a FF D8 (JPEG start of image marker if you've forgotten)!

The first lot of all zeros is at 0x0614B3600 = 1632318976

1632318976 (what we thought was the end) - 1592612864 (where the 0 blocks start) = 39706112 (of extra data).
39706112 / 1048576 = 37MB more, hopefully of the same file!

Now that we know where it is, we can dig it up, and hope it's in order, because there's no FAT linked list to tell us that it is.

Anyway, I'll continue this later, and hopefully find some results.

What's the worst part of crashing a RC glider? No, not the fact that I'll have to break out the epoxy... again.

TLDR of the situation is that when the glider hit the orange fencing, it went through one of the holes up to the wings. The brushless motor and front section, being heavy, decided that they'd rather keep going forward, and so tore the front bit off. The battery decided it'd also like to come along for the ride... and so power was lost to the onboard camera.

Which was recording video at the time. I'd like to see that video, and I know it's somewhere on the card. It has to be somewhere on the card. According to information that is part of the firmware upgrade, it has 8MB of DRAM, so it's definitely not staying in there. And it stops recording quickly.

But there is no filesystem entry for the file. TestDisk doesn't see it. PhotoRec for some reason doesn't even see all the videos that *were* on the card normally - I guess it's designed for photos.

So looks like I'm going to have to work out what's going on here, and write my own app to get the data back. Fat32 isn't that complicated. I imaged the card without mounting it, so I have a completely untouched copy of the card.

This page is quite handy. So is this one. I'm going to assume you have at least some grasp of Python here.

Let's dive right in:

#!/usr/bin/env python
import sys
import block

f = open(sys.argv[1],'r')
volumeid = f.read(512)

The "block" library I am using here is this one. It's not absolutely necessary but it does make things easier here.

Now for some definitions:

def getUInt8(data,offset):
    return block.Readview('B',block.Block(data[offset:offset+1]))[0]

def getUInt16(data,offset):
    return block.Readview('H',block.Block(data[offset:offset+2]))[0]

def getUInt32(data,offset):
    return block.Readview('I',block.Block(data[offset:offset+4]))[0]

def syswrite(text):
    sys.stdout.write(text)
    sys.stdout.flush()

Yes, we can do this without block, but it's easier this way. On a side note - I'm not sure what happens on a big-endian system.

Now let's pull some values out of the first sector. Note that this is the first sector of the partition, not the first sector of the MBR. I imaged the partition, not the card. So we don't have to worry about the MBR or a partition table.

OEMIdent = volumeid[0x03:0x0A]
bytesPerSector = getUInt16(volumeid,0x0B)
sectorsPerCluster = getUInt8(volumeid,0x0D)
numReservedSectors = getUInt16(volumeid,0x0E)
numFATs = getUInt8(volumeid,0x10)
numDirEnt = getUInt16(volumeid,0x11)
totalSectors16 = getUInt16(volumeid,0x13)
totalSectors32 = getUInt32(volumeid,0x20)
sectorsPerFAT = getUInt32(volumeid,0x24)
rootDirFirstCluster = getUInt32(volumeid,0x2C)
signature = getUInt16(volumeid,0x1FE)

OEMIdent is a text string that identifies what was used to format the disk. In this case it is "mkdosfs".
BytesPerSector is how many bytes/sector the disk media is. This is 512 on nearly every media - even 4k sector HDDs.
SectorsPerCluster is how many sectors in a cluster. Fat32 doesn't address sectors individually, it addresses "clusters" of sectors. In our case it is 8, resulting in 4KiB clusters.
numReservedSectors is essentially how many sectors into the disk the real fun starts. In our case it is 38. This is from the start of the disk and includes the first sector we are reading from now.
numFATs is the number of FATs on the disk. I'll get to exactly what a FAT is later, but there are often (read: nearly always two) sequential copies of one on the disk to guard against filesystem corruption from a dodgy disk. In our case it is two.
numDirEnt is the number of root directory entries.
totalSectors16 is the 16-bit count of sectors. Only problem is, though, that like 640k of RAM supposedly being enough for anyone, a 16-bit value has a maximum of 65535, allowing a maximum of 32MB of sectors. Not quite enough for modern forms of storage. This is zero here. On a side note, I wonder if the people who originally wrote this thing are surprised that it's still in use today?
TotalSectors32 is in the extended information area, an area previously unused by Fat12 and 16. This is a full 32-bit count of sectors, enough for 2TB of drive with 8k clusters, a limit that we have finally exceeded in a single drive. However, most people with drives this large are using ext3/4, HFS+ or NTFS on their HDDs by this stage.
sectorsPerFAT is the size of one of the FATs in the filesystem, in sectors.
rootDirFirstCluster is the where the root directory entry is on the disk. This is relative to the end of the FAT.
Finally, we have a signature which should be 0xAA55.

Now here's the fun stuff.
Fat32 is really Fat28. Only the lower 28 bits of the address are actually used. Go figure.
What's the address of the first cluster? 0? 1? No, it's 2. Clusters 0 and 1 don't exist. Why? You'll see later.

So let's find the FAT:

fat1addr = numReservedSectors*bytesPerSector
fat1length = sectorsPerFAT*bytesPerSector

f.seek(fat1addr)
fat1 = f.read(fat1length)

So what is the FAT? If the filesystem is named after it, it must be important right?
The FAT is an array of 12, 16 or 32 bit values, one per cluster. For Fat32:

• 0x00000000 means a free block.


• 0x00000001 is "reserved" (read: they haven't thought of something dodgy to do with it yet).


• 0x00000002 to 0x0FFFFFEF point to the next block in the file


• 0x0FFFFFF0 to 0x0FFFFFF6 are reserved


• 0x0FFFFFF7 means not to use this cluster as it has a bad sector in it


• 0x0FFFFFF8 to 0x0FFFFFFF mean that this is the last sector in the file, or the only sector

So essentially the FAT is a big pool of linked lists. And all kinds of fun stuff can happen, like:

• Pointing to a free block - not too bad, reference is usually overwritten anyway


• Two files pointing to the same block - cross-linked files - very bad. If this happens accidentally, the end of one file is lost, and replaced with the end of another. In addition, writing to the end of one will overwrite the other.


• Pointing to an invalid block - likely will produce an error

So let's read in the FAT.

fatptr = 0
usedclusters = {}

print "Finding non-zero clusters"
while (fatptr fat1length/4): # 256 now
    nextlink = getUInt32(fat1,fatptr*4)
    if (nextlink > 0): # if this block does something
        usedclusters[fatptr] = nextlink
    fatptr += 1
    if ((fatptr % 1024) == 0):
        syswrite(".")

print "\nFound %i used clusters = %iMB data\nTraversing files..." % (len(usedclusters),(len(usedclusters)*sectorsPerCluster*bytesPerSector)/1048576)

We save all the non-zero FAT entries into a dictionary, keyed by their position. The value is the next link.
Running this over my card image reveals that there is a different number of used sectors to the "used on disk" measurement... something's in the FAT but not linked to by the directory. Suspicious.

Now let's do something with the data...

files = []

while True:
    keys = usedclusters.keys()
    if (len(keys) > 0):
        nextptr = usedclusters.keys()[0];
    else:
        break
    newfile = []
    while True:
        thisptr = nextptr
        if not (usedclusters.has_key(nextptr)):
            print "Error: nextptr %.8X is unallocated" % nextptr
        break
    nextptr = usedclusters[nextptr]
    del usedclusters[thisptr]
    newfile.append(thisptr)
    if (nextptr >= 0x0FFFFFF8):
        break
    files.append(newfile)

print "%i files found" % len(files)

for i in files:
    print "File found: %i clusters (%i MB), start %.8X" % (len(i),len(i)*sectorsPerCluster*bytesPerSector/1048576,i[0])

Basically we pick the first item out of our set of keys. We are assuming the start of the file is before any part of it, but this may not be true. Then we retrieve the next sector from the FAT - essentially traversing the linked list until either the end, or until it links to a free sector. The second should never happen - but the camera did get it's power cut in the middle of recording.

Traversing files...
Error: nextptr 00004000 is unallocated
Error: nextptr 0005E000 is unallocated
34 files found
File found: 1 clusters (0 MB), start 00000000
File found: 1 clusters (0 MB), start 00000001
File found: 1 clusters (0 MB), start 00000002
File found: 1 clusters (0 MB), start 00000003
File found: 1 clusters (0 MB), start 00000004
File found: 1 clusters (0 MB), start 00000005
File found: 1 clusters (0 MB), start 00000006
File found: 1 clusters (0 MB), start 00000007
File found: 1 clusters (0 MB), start 00000008
File found: 1 clusters (0 MB), start 00000009
File found: 1 clusters (0 MB), start 0000000A
File found: 16546 clusters (64 MB), start 0000000B
File found: 1 clusters (0 MB), start 00000108
File found: 1 clusters (0 MB), start 000001DD
File found: 1 clusters (0 MB), start 00000200
File found: 1 clusters (0 MB), start 00000201
File found: 1 clusters (0 MB), start 00000211
File found: 1 clusters (0 MB), start 000002CB
File found: 1 clusters (0 MB), start 0000034B
File found: 1 clusters (0 MB), start 000003B5
File found: 1 clusters (0 MB), start 000004B4
File found: 1 clusters (0 MB), start 000004DC
File found: 1 clusters (0 MB), start 0000060F
File found: 1 clusters (0 MB), start 000013BF
File found: 1 clusters (0 MB), start 0000175A
File found: 1 clusters (0 MB), start 0000175B
File found: 1 clusters (0 MB), start 0000175C
File found: 20724 clusters (80 MB), start 000040BC
File found: 1 clusters (0 MB), start 000053F3
File found: 126452 clusters (493 MB), start 000091B1
File found: 28972 clusters (113 MB), start 00027FA5
File found: 97103 clusters (379 MB), start 0002F0D1
File found: 90579 clusters (353 MB), start 00046C20
File found: 4621 clusters (18 MB), start 0005CDF3

The 1 cluster files are highly likely to be directory entries. The rest look promising - their sizes almost exactly match the videos that I can see on the mounted disk image of the card. However, there's a 18MB one, and an 64MB one that are unaccounted for. I think these are our files!

Time for some more code:

firstDataSector = (numReservedSectors * bytesPerSector) + (numFATs * fat1length)
print "firstDataSector %i 0x%.8X" % (firstDataSector,firstDataSector)

def getCluster(cluster):
    address = (((cluster - 2) * sectorsPerCluster) * bytesPerSector) + firstDataSector
    f.seek(address)
    return f.read(bytesPerSector * sectorsPerCluster)

Work out where the data area starts: after the reserved section and FATs. Then work out how to read a cluster. Find it's address and length. Remember that sectors, bytes and clusters are not the same thing. A cluster is (in our case) 8 sectors. A sector is 512 bytes.

Now do something with this:

index = 0
for file in files:
    g = open("recovered/%i" % index,'w')
    for cluster in file:
        g.write(getCluster(cluster))
    g.close()
    index += 1
    print "written %i" % (index)

For each file, open a file and dump all the clusters into it. This will result in the files not having the exact same size they did - they will be padded to the nearest cluster. The actual filesize is in the directory - that we don't look at (yet).



Well, they all play after changing their file extensions to AVI. Except 11 and 33. No go in Quicktime, VLC or mplayer.

gm$ file *
11.avi: RIFF (little-endian) data, AVI, 0 x 0, >30 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
27.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
29.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
30.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
31.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
32.avi: RIFF (little-endian) data, AVI, 720 x 480, 30.00 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)
33.avi: RIFF (little-endian) data, AVI, 0 x 0, >30 fps, video: Motion JPEG, audio: uncompressed PCM (mono, 24000 Hz)

Noticing a pattern here.


Well that looks like the proper RIFF AVI header. Time to learn how AVIs work I guess... in a future blog entry.

2m wingspan, foam construction, ~300w brushless outrunner motor.

But of course I can't leave it stock.

Parkzone Radian Second Flight and First Crash from gm on Vimeo.

Well you quickly learn how to repair these things. 2-part epoxy, tape and hot glue are your friends here.

But I'm standing down here flying the thing, I want to see what it's like up there!

A bit of searching found a suitable camera - the FlyCamOne. It's small and light, and the cam module is separate from the electronics. I cut the camera loose from it's stand, and embedded it in the foam behind the canopy. The camera board is up the front with the speed controller. But the best feature is that it has a 3-pin servo connector on it - it draws power from the plane itself, thereby needing no batteries. As well as this, you can toggle the recording status from the controller - handy if you realise that you forgot to start the camera when the plane is in the air.

So here's my flight yesterday:

Radian Flight 2.0 from gm on Vimeo.

Perfect weather.

Today's weather wasn't so great... but this just makes the video more interesting for you :)

Radian Flight 2.5 from gm on Vimeo.

There’s been a minor change to the portal2 API as of v2.8.3 (to be deployed at LAN 10.09). Functions which attempt to identify you (usage, usage_history and whoami) will attempt to use cookie-based authentication before attempting to identify your computer by MAC address.

This will allow in-browser applications to determine the identity of who you are logged in as before attempting to fall back to who owns the computer.

In other news, portal2 v2.8.3 now has a couple of other changes to it, such as support for clustering and a new graph on the usage page which is generated client side (and much faster than the server-side graph).

I have recently encountered a number of issues relating to populating <select> elements using JavaScript. The largest issue I was having was with <select> elements that were themselves created using JavaScript. In this case the elements would be un-selectable.

If you clicked on them, or tabbed to them; then they would highlight, but nothing else. ( issue example"> issue example">example)

The issue it turns out is caused by the way that Internet Explorer <8 handles <select> lists. More to the point the way that it treats them as special elements, not standard HTML elements. I was simply appending the <option> elements to the <select> as DOM child nodes. While this works correctly in every other web browser I tested (Chrome 4>, FF 3.5>, IE8) it doesn't work in IE7, which was a requirement for the project. It turns out that in Internet Explorer if you want to add more options to a <select> element you need to create new Option objects and append them to the <select>.objects 'array' (I say 'array' as its not really an array... but it behalves enough like one).

(Note, I am using Prototype language extensions in the below example)

var select = document.getElementById('mySelect');
var options = new Hash({'0': 'ACT', '1': 'SA', '2': 'NT', '3': 'NSW', '4': 'Vic', '5': 'Tas', '6': 'WA', '7': 'Qld'});
var i = 0;
options.each(function(opt) {
  select.options[i] = new Option(opt.value, opt.key, null, false);
  i = i++;
});

I hope this helps you if you are also having random issues with <select> elements and Internet Explorer.

If you haven't seen Mumble, you should probably check it out at http://mumble.sourceforge.net/. It's like TeamSpeak, Ventrilo or that kind of thing, but it's fully open source and completely configurable. There are servers for Linux and OS X (and probably Windows if you compile it). There are clients for Mac, Linux, FreeBSD, Windows, Maemo, Android and iPhone. And it's open source so you can code another one.

There's also uMurmur, a minimalist Mumble server designed to run on very low resource devices (OpenWRT running routers etc).

Anyway, the thing is, I want to use push-to-talk on it. But this needs an unused key. Initially I suggested Fn, but Fn-up and Fn-down are PgUp and PgDn (on a Mac laptop keyboard anyway). So every time you use them, you transmit. Then I realised I'd never really pushed the right option key. Perfect. But if I set it to option, it triggers on left option as well.

Enter DoubleCommand. Install it and turn on the "Right Option acts as Enter" key. Now, go into the Mumble settings, and assign it as the PTT key. Then check the "Inhibit" checkbox - this captures the keystroke so it doesn't get passed onto apps (you don't type enter in the frontmost app).

The only downside is if you use a keyboard with an enter key, you can't use it. I might write and submit a patch for DoubleCommand to remap right option to F13 (there's actually up to F24 as you can see in keycodes.h).

But it works, and now I have a PTT trigger that's easy to reach and not used anywhere particularly important.

Well i got bored, so i made a script that will wrap the commands needed to mount your university home drive onto your laptop. At this time it works on OSX, and you may need to change the mount -o opts for linux and install mit krb5 or heimdal.

This will work out of the box on OSX

#!/bin/sh
#Get our username
if [[ ! -n $1 ]];
        then
        echo "Enter a username"
elif [[ ! -n $2 ]]; then
        echo "Enter a mount point"
else
        if [[ ! `kinit $1@AD.ADELAIDE.EDU.AU` ]];
        then
                #Now we need to trim the last digit off to know where we are mounting from
                #Create the mount point
                if [[ ! -d $2 ]]; then
                        mkdir -p $2
                fi
                export SUFFIX=`echo $1 | sed s/.......// `
                #Mount from the correct nfs server.
                mount -t nfs -o vers=4.0alpha -o sec=krb5 nfsusers$SUFFIX.ad.adelaide.edu.au:/root_vdm_33/nfsusers$SUFFIX/nfsusers$SUFFIX/$1 $2
        fi
fi

NOTE: Yes, i know you cant see the whole mount command. It copy pastes fine though.

Some people may also wonder if this means you can hijack someone else’s directory. No you cant, sorry. that is what krb5 is for, to prove your identity.

Put that into a file called mount_nfs.sh . Then chmod +x mount_nfs.sh, and run it as sudo mount_nfs.sh . Enter the details when prompted. It should be

sudo mount_nfs.sh a1111111 /mount/point

On osx a safe mount point is /Volumes/UNINFS

It may help on OSX to install http://web.mit.edu/macdev/www/osx-kerberos-extras.html

Enjoy

PS – yes i cheated with sed, i was annoyed at it.

I’m going to be working on a new series of blog posts with Firstyear in which we’re going to discuss a lot of StreetGeek’s network and server infrastructure. Information about this is around the place (or you can ask us), but we would talk about more publicly about what goes into making a “medium” LAN party work. I say “medium”, which is what I’d personally group all LANs with 100 to 1,000 attendees. As a contrast, I’d call an event like wiLANga with about 50 people “small”, and the major (commercial) US/EU LANs like QuakeCon with > 7,000 people “large”.

StreetGeek itself has seen many major changes over the years, as it grew from a LAN of about 20 people hanging out in The University of Adelaide to running monthly events with over 100 attendees at Colonel Light Gardens Uniting Church, the LAN area at AVCON, and leading SAGAfest in 2009. With this, it’s had to adapt to these new, larger environments, and invest in better hardware.

Here’s what StreetGeek’s network looks like, showing only the switches.

StreetGeek’s backbone is a Cisco-Linksys SGE2010. It’s a 48 port managed full-fabric gigabit switch, with 4 SFP ports. It replaced a previous Alloy 48-port managed gigabit switch, which didn’t have as many features, and had the nasty habit of overheating under the load we put it through. This switch lives on the “mu table” (formerly known as the B/C tables).

Coming off that are three Cisco-Linksys SRW2024 switches. They’re 24-port managed full-fabric gigabit switches, which are connected to the backbone each by two copper links using link aggregation. This means between these switches and the backbone there is 2gbit/s of connectivity, full-duplex. These switches live on D, E and G tables, where they together pushed about 7 TiB of traffic during the 10.06 event, with 1.1GiB/s peaks from clients on those tables (that’s about 10gbit/s). While the amount may seem impressive, in reality it’s really not – the switches could push 8.3GiB/s (72gbit/s) of traffic, which if running flat out for an entire event would add up to about 758.6 TiB.

On the overflow tables, and in the console area, there are Cisco-Linksys SR2024 switches. They’re the unmanaged counterpart of the SRW2024, so they don’t support link aggregation. These are connected in various points around the LAN, generally where there aren’t a large amount of clients to justify extra bandwidth.

At the moment, Firstyear has loaned the use of his Apple Airport Extreme access point. It’s a 802.11n access point, which is also one of the few wireless access points on the market that can do 2.4GHz and 5GHz at the same time (thereby effectively doubling the available spectrum that an access point can occupy). They easily handle having 50 clients on the network, and have coverage around most of the venue. We’re planning to replace this in the future with the LAN purchasing two of it’s own Airport Extremes: one in the main hall, and one in the console area.

We’ve tested these access points with gm trying to perform denial of service attacks against it with thousands of simulated clients, and it held up where other routers would just crash. Previously we had used 3 D-Link 802.11n access points to service the network, however the reliability of these access points under load was absolutely atrocious.

Something that may stand out with this is that the console area has a very poor layout. In the end, the console area doesn’t push enough traffic to justify additional cabling to do things “properly”. The only current-generation console that has gigabit ethernet is the Playstation 3, and nothing on the Playstation 3 actually pushes that kind of bandwidth. Typically, LAN games use a megabit or two per second, which is just tiny. The largest amount of bandwidth is used by the RetroLAN PCs, where 10 machines boot from an iSCSI virtual disk – and even then that’s only used for the operating system.

So this has pretty much covered our layout. We run a lot of services on the network itself, which will be covered in later editions of this series. Most of them are from servers in the office, or from servers in the admin area of the LAN hanging off the backbone.

Alot of people, must wonder how and where we get these ideas of things to do at lans and our server systems.

First, most likely myself or Micolous have an idea of something we want to do. Lets say “how nice would it be if we could have single sign on?”. Or we read something cool that someone has done, or we gain inspiration from our workplaces.

Next, we do alot of research. We learn the topic inside and out. For example, single sign on can be done using kerberos, and there are plenty of good resources to be found googling about it.

We then sandbox it. We will setup VMs, or we will victimise our servers at home to install this system on and use it. Most learning comes from actually doing, and understand the way a service works, in the real world. At home I installed kerberos, and use it for authentications now. I then went and extended that to trial how domain trusts worked with Micolous’ servers. In the majority of cases, we wont install a service on our home systems unless we know it is secure and trustworthy (and that is where our research comes in)

Next, we discuss feasibility of using this at the LAN. Who would it benefit, would it need a large overhaul of systems, would we need to publicise the change, who do we need to inform of the differences?

In the example case of kerberos, it would be a minor change, only affecting the people who administre servers. Most LAN attendees wouldnt need to know that we auth to systems with kerberos. If however it was a change to portal, we would make that change public, even before installing the first package.

Next, we install and configure the systems between lans. This gives us a chance to configure all our systems and relevant services, as well as extensively test it. If it isn’t satisfactory in this testing, we revert our changes, and start again, or we will wait for the next lan to implement the change.

Once the system is tested, we then bring it to the lan. During its first deployment we tend to monitor a systems use, and stability as LAN conditions are vastly different from our homes. In the majority of cases, no issues occur, and it becomes a permanent in place system. If something does go wrong, we fix it and do our best to mitigate and learn from the issue.

Rinse and Repeat.