The problem:

I have a directory full of unclassified media files, some are duplicates, some aren’t, and each one follows a different naming convention.

I even try to classify them from time to time, so you can throw some directories into the pack. Sometimes, I even create two or three directories for the same group-series-category-whatever before I realize there is an existing one with a slightly different name. And frequently a lot of files remain unclassified, many of which could fit into one of the directories I mentioned.

Of course … whenever a new file arrives to my home server, it gets thrown into that very same directory, so Chaos keeps spreading, as it always does.

To clarify things, lets show an example:

drwxrwxrwx 1 user user      4096 2010-07-03 11:50 01_Battlestar_Enterprise
drwxrwxrwx 1 user user      4096 2010-07-03 11:42 02_Startrek_Galactica
drwxrwxrwx 1 user user      4096 2010-07-03 11:50 03_battlestar.enterprise-season.1
-rwxrwxrwx 1 user user 220393472 2010-07-03 02:49 battlestar.enterprise.s1e01.avi
-rwxrwxrwx 1 user user 221227008 2010-07-03 02:50 Battlestar_Enterprise_1_22.mp4
-rwxrwxrwx 1 user user 195393472 2010-07-03 02:49 startrek.galactica.4x15.[ripper_22].mkv

As you can imagine, sorting things up can get really tedious, and there is no automatic way of doing it that I know of.

I had some time this morning and got fed up with it. Every little piece of help is more than welcome, and here is where Python comes to the rescue.

The solution:

There are dozens of ways to do this, but I ended up coding a quick hack to help me sort things out.
It just compares the names of files and directories, and estimates the similarities. Anything above a 50% match is usually correctly estimated.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# (C) 2010, Taher Shihadeh
# Licensed: GPL v2

"""
The script works based only on names of files and directories in a
non-recursive manner.

It takes a path as parameter and tries to determine if the names of
the contents look alike.

It removes separator characters, numbers and file extensions prior to
the comparison.
"""

import os
import sys
import string
from operator import itemgetter

FAST = False # Change this to skip file-to-file comparisons
SEP  = '_-+~.·:;·()[]¡!¿?<>'

def main (path):
    lst1    = os.listdir (path)
    lst2    = lst1
    len_lst = len(lst1)
    count   = 0.0
    results = []

    for x in lst1:
        for y in lst2:
            if x==y:
                continue
            x_dir = os.path.isdir(x)
            y_dir = os.path.isdir(y)

            if FAST and not (x_dir or y_dir):
                continue

            result = {'A': (x, x_dir), 'B': (y, y_dir)}

            str1, str2 = x, y
            if not x_dir:
                str1,_ = os.path.splitext (x)
            if not y_dir:
                str2,_ = os.path.splitext (y)

            result['factor'] = compare (str1,str2)
            results.append(result)

        lst2.remove(x)
        count += 1
        print >> sys.stderr, '%.2f%% done' %((count / len_lst)*200)

    show(results)

def split (str1):
    trans = string.maketrans(SEP, ' '*len(SEP))
    return str1.translate(trans).split()

def clean (lst):
    assert type(lst) == list
    return filter(lambda x: not x.isdigit(), lst)

def compare (str1, str2):
    """Return similarity factor as percentage"""
    aux1 = clean (split (str1.lower()))
    aux2 = clean (split (str2.lower()))

    set_or  = set(aux1) | set(aux2)
    set_and = set(aux1) & set(aux2)

    return (float(len(set_and)) / float(len(set_or)))*100

def show (results):
    """Show most similar last"""
    for x in sorted(results, key=itemgetter('factor')):
        a,b = x['A'],x['B']
        if not b[1] and a[1]:
            a,b = b,a
        print '%.2f \t %s \t --> %s' %(x['factor'], a[0], b[0])

if __name__=='__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        path = os.getcwd()

    main (path)

I don’t think any one is going to use it, but what the hell. It’s a big Internet ;-)

Everything was recorded, so we will upload the slides and videos of all our sessions really soon. For now, only the photo gallery is available. Take a look at the mugshots.

I’m really glad we could make this Summit. It surpased all my expectations. By far. It was an unbelievable experience, and we had lots of fun. Take a look at our family photo. If you want to know which of the guys above is me, here’s a clue: “In brightest day…”.

I’m really looking forward to the next summit. Cherokee Summit 2010 was awesome. I’m sure the next one will be even better.

I’m sure that meeting many of the developers of Cherokee in person will be the highlight for me.

If by any chance you’ll be in Madrid that weekend, don’t forget to register in time and join us.

It’s gonna be legen… wait for it… dary!

You can read Alvaro’s announcement, or you can check out the Summit web-site.

Cherokee will be an important topic, but it won’t be the only one. Those will be a couple of days fully dedicated to High Performance and Scalable Web topics, so there’s room for everyone to join in.  We are commited to reaching the 1.0 milestone of Cherokee by then, so we will also have a party to celebrate it.

It’s going to be fun. I’ll be a speaker at the summit and I’m really looking forward to personally meeting many of the members of the project. Thanks to our sponsors we’ve managed to make the event completely free, so don’t forget to register while we still have free spots!

UPDATE: We’ve written a little brochure (~100KB) that can be used to  let your colleagues know about the summit. Do not hesitate to send it to any coworker or friend who would be interested in attending a High Performance and Scalable Web event.

From here I’d like to thank P.V. Anthony for his invaluable advice on audio production and my old friend Sara Genge for lending her voice to the project (and for her awesome fiction writing, but that is another story).

We wanted to brag about our little baby. After all, not every serious web server out there has a killer interface to configure it. Take a look at our Cherokee Web Server introductory screen-cast.

You might want to see it at full screen for readability.

It’s just one of many to come. We’ve got some more planned, so I’ll let you know when they’re ready.

If you expect a party kit from us you’ve come to the wrong place. We actually believe our software is so good that it is a prize for its own merits. It has been a while since I last announced one of our releases, mainly because I didn’t have much to add besides what was told on the official announcements. As always, a lot of development effort is being invested in our flag product, and this is something that doesn’t go by unnoticed. This weekend we decided to release Cherokee 0.99.25. As you can tell by the .25 part, lots of fixes and enhancements have been added steadily release after release.

I hope you enjoy it. We’ve tried to update all the documentation for this release, and we’ve automated most of the recipes in our cookbook by adding lots and lots of configuration Wizards, so hopefully you’ll be able to set up anything in a matter of seconds. As always feedback and feature requests are more than welcome at the mailing lists. Here are links to download the tarball and the online documentation.

The easiest thing to do would be installing the system normally through PXE (look for the link at the end of this post if you’re interested), but there is a problem. Flash devices have a limited number of write-cycles, and wear levelling is not used on consumer grade cards. There are file systems optimized for flash devices -such as JFFS2- but the benefits are not obvious here. As I understand it, these are designed for industrial devices with direct access to the memory cells. Consumer devices have an abstraction layer that make them transparent and enable us to use them as any other normal storage. It should provide everything we need, such as wear levelling. But as I said before, it normally doesn’t for consumer-type devices.

Alternatives

Having discarded the optimal solution for both technological and budget limitations, we are left with three alternatives.

1. Use the CF as an ordinary device, with total disregard for the premature death of the drive. This is the simplest, and provided you have a lifetime warranty for your media it might not be such a bad idea.
2. Use the CF as if it were a LiveCD, maybe even adding persistence for our changes. This option should prove perfect if we want to use the Thin Client as a desktop system. Temporary changes are written to RAM, permanent changes will be stored on disk and even software updates will remain between reboots. It is a bit more complicated, so I’ll leave this in my TO DO list for now.
3. Install everything by hand on a local copy, configure the system to be able to run with as little disk access as possible (ideally, from a read-only root file-system), and dump it to the CF. Since space is not a problem for me (I have both a 1GB and 8GB CF cards), I’ll prefer this approach. In case everything was set as read-only, we would only  have to remount the file systems as writable during the process. As for SquashFS, though it seems an ideal choice for an embedded system that needs no upgrades, I’ll discuss this in other posts. It is simpler to deal with a “standard” system, not having to recreate binary boot images every time you update the box.

My home server is used to share media and printers on my local network, manage downloads, and above all, provide a permanent access to my home network from wherever I might be. This is, by far, the handiest thing ever. At least it is for me. You never know when you are going to need to bypass Internet filters when you are roaming, for example. The wonders of SSH truly never cease to amaze me :)

I’ve opted for letting some writing be done to the CF, albeit the write-intensive tasks have been dealt with, because it is easier for the system to be updated, and because some of the services I’ll be using refuse to work on read-only mode. However, my next entry will be about using aufs/unionfs, and replicating this setup on totally read-only and compressed system that will run from the 256MB standard CF that comes with the computer.

Things to install

• Basic system: since Jaunty is still in Alpha stage, downloading the whole ISO is not worth it. Just grab the NetBoot mini ISO and download the rest as you go. You’ll have to update frequently until it is released anyhow.
• linux-image-server: A scheduler less suited for multimedia but with some advantages for a server.
• ssh: Access point to/from your server.
• cups: printer support. Note that CUPS alone will add around 100 MB to your installation.
• portmap, nfs-kernel-server: NFS file server.
• samba: sharing files and printers for Windows boxes.
• nmap, mc, wget, curl, unrar, p7zip-full, arj, unzip, unp, unace, qemacs-nox: some handy tools I can’t live without. Mostly Swiss-army knives, decompressors and a tiny Emacs-like editor. It’s bigger than mg, but it has UTF8 support. I love Emacs but it is huge for a system like this.
• usbmount, ntfs-3g: to automatically mount USB devices. Nice if you plan to plug and unplug your external drives to share them on your LAN.
• mldonkey-server: for our every day download-needs.

These were the steps I followed. Keep in mind these are mainly notes to myself. They should work for you, but they might not. At the very least these steps will give you some useful hints. Please note that there might be some rough edges in the tutorial, since I actually wrote this after having everything installed. I tooks some notes during the process, and this setup is working flawlessly for me. I don’t think any important (read non-trivial) steps were left out, but if that’s the case and it is giving you any kind of problem, just ask in the comments!

The CF shouldn’t suffer much IO, so the idea is to install and customize everything on a virtual machine, and then dump the resulting image to the CF after making every needed adjustment.

I originally intended to make the system completely read-only, but then decided that preventing /tmp and /var/log from massacring the CF would be more than enough. You can skip completely some of these steps, since the root-fs will not be totally read-only. For that, check the next post.

Installing the base system

I’ll be using VirtualBox because I find it easier to work with than plain QEMU, particularly since I’m going to be needing the network from within the virtual machine. Just create a new one with a disk size that can fit in you CF. I’ll use a 1GB card that I have laying around. Also configure the CDROM drive to use the NetBoot mini ISO that you have downloaded.

Before booting the machine, add PAE/NX support to the virtual CPU. The generic kernel doesn’t need it, but once you try to boot with the linux-image-server branch you will need this to proceed. That is done from the virtual machine definition section following the path: Details -> General -> Advanced.

Boot the machine and type ‘cli’ if you want to minimize the installations. Then manually partition the hard drive if you want to use the new EXT4 file-system type, and go through the process as you normally would. Of course, disregard the swap partition. You can’t have that on a CF or it will die soon for sure.

I use the French repositories because those are the fastest for me. I always start by modifying the list of repositories on the recently installed system, in case I want to install things not present in universe. Edit it manually or just append the needed info.

echo "deb http://fr.archive.ubuntu.com/ubuntu jaunty \
main restricted universe multiverse" >> /etc/apt/sources.list

And install the packages.

sudo su -
apt-get clean
apt-get update
apt-get install linux-image-server openssh-server samba cups \
 wget p7zip-full portmap nfs-kernel-server nmap mc \
 qemacs-nox unp arj unrar unzip usbmount mldonkey-server
apt-get clean

No real need to mention it, but don’t forget to delete the generic kernel and associated files if you want to save another 100MB or so.

Now to handle issues related to readable-writable paths, as well as config files. Some steps are only useful if you will mount the rootfs as read-only. I’ll mark them as “Optional”:

• Optional: /etc/mtab: We will link it to a file that can describe the mounted file-systems without requiring write access.

rm /etc/mtab
ln -s /proc/mounts /etc/mtab

• /etc/network/interfaces: static IP for the server. Bare in mind you’ll need writable /etc/resolv.conf and /var/lib/dhcp3 if you intend to use dynamic IPs. You’ll probably have to edit /sbin/dhclient-script to account for this if you are going for a full-ro filesystem. This is how my /etc/network/interfaces looks like.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.0.0.100
gateway 10.0.0.1
broadcast 10.0.0.255
netmask 255.255.255.0

• /etc/etc/fstab: define the mountpoints. Ensure that directories that need to be writable ar mounted as tmpfs rather than letting the system write them to flash. Alternatively you can mount them on external media. You need this to run the rootfs mounted as read-only. Even if you are not going to do so, you *MUST* set the ‘noatime’ flag when mounting, or at least ‘relatime’. If you don’t, every single access to the files will require a read operation to modify the access time, and this is pure overhead for the CF that will significantly shorten its life span. Using journaling is probably another bad idea, since many commits will also affect the CF. Either use EXT2, or use this as a temporary solution and go for the SquashFS-Live system of approach #2.

/dev/sda1 / ext4 noatime,errors=remount-ro 0 0
proc /proc proc defaults 0 0
tmpfs /var/log tmpfs defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
tmpfs /var/lib/urandom tmpfs defaults 0 0
# Define the following ones if you use NFS or dynamic IPs.
# Writable paths should be tmpfs or be on external media.
# Both settings are OPTIONAL for our current setup.
tmpfs /var/lib/nfs tmpfs 0 0
tmpfs /var/lib/dhcp3 tmpfs defaults 0 0

• Logging: either log to a tmpfs as shown above, to an external media, remotely or remove logging capabilities. This step is not optional at all. If you choose any option other than the first one, you’ll probably want to take a look at /etc/syslog.conf, /etc/default/syslogd and your /etc/rc?.d for links to /etc/init.d/sysklogd. Don’t forget the syslog.conf man entry. Note that logging to a tmpfs will give you some warnings at boot-time since some files are expected to exist. It’s probably not such a good idea, since you’ll loose the logs between reboots, troubleshooting vulnerabilities becomes more difficult, and so on. If you do proceed with the idea, you should keep this in mind: some logfiles are expected to be present at boot-time, so you’ll get some errors if you boot with an empty /var/log. Mldonkey also doesn’t boot if it can’t access the logs, so populate the directory with a  trivial script such as the following. It  must be run at boot-time before /etc/init.d/bootlogs.sh and /etc/init.d/sysklogd are invoked:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          madoka
# Required-Start:    $local_fs
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      1
# X-Start-Before:    bootlogs sysklogd
# Short-Description: recreate /var/log structure for Madoka
# Description:       recreate /var/log structure for Madoka
### END INIT INFO

do_start () {
 mkdir -p /var/log/mldonkey
 mkdir -p /var/log/news
 cd /var/log

 touch \
 dmesg          \
 mail.warn      \
 user.log       \
 daemon.log     \
 messages       \
 debug          \
 auth.log       \
 mail.err       \
 syslog         \
 mail.log       \
 kern.log       \
 lpr.log        \
 mail.info      \
 news/news.crit \
 news/news.err  \
 news/news.notice

 chgrp adm dmesg
 chown syslog:adm \
 mail.warn       \
 user.log        \
 daemon.log      \
 messages        \
 debug           \
 auth.log        \
 mail.err        \
 syslog          \
 mail.log        \
 kern.log        \
 lpr.log         \
 mail.info

 chown news:news news
 chown root:news news/*
}

case "$1" in
 start)
 do_start
 ;;
 restart|reload|force-reload)
 echo "Error: argument '$1' not supported" >&2
 exit 3
 ;;
 stop)
 # No-op
 ;;
 *)
 echo "Usage: $0 start|stop" >&2
 exit 3
 ;;
esac

I saved it in /etc/init.d/madoka, and added it to my default runlevel to execute before init.d/bootlogs.sh and /etc/init.d/sysklogd

update-rc.d madoka start 05 2 .

Adding some more goodies

• mldonkey: I just made symbolic links to an external drive. /var/lib/mldonkey/incoming, /var/lib/mldonkey/temp and /var/lib/mldonkey/torrents are the most frequently written directories by far. It also rewrites the configuration files whenever changes are made. Those are also stored under /var/lib/mldonkey/.
  

• /etc/usbmount/usbmount.conf: this is great if you are going to have lots of external drives. It automatically mounts the devices wherever you choose to (in my case, under my shared samba directory). It just works after rebooting. I only edited the file to specify the mount point and to add more file-systems that I wish automatically mounted:

FILESYSTEMS="ntfs vfat ext2 ext3"

• /etc/samba/smb.conf: to share files and printers, edit yours to resemble these pointers.

# add to [global] section, and edit the rest to your heart's contempt
workgroup = WARPCORE
printcap name = cups
printing = cups
security = share
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
# make sure [printers] section looks like this
[printers]
browseable = yes
printable = yes
public = yes
create mode = 0700
guest only = yes
use client driver = yes
path = /tmp

#define at least one share point
[shared]
comment = Shared media
path = /media
guest ok = Yes

To have the printers actually work you’ll need to configure them with cups. After having everything tuned, remember to mount the root file-system read-only. Adding the printer is trivial thanks to the administration web interface. Just forward the needed port or make cups accept connections from interfaces other than localhost, and launch your web browser.

ssh -L 2000:localhost:631 10.0.0.100

Then, access the configuration wizard visiting http://localhost:2000. On Windows systems you can add this by simply add ing a new printer with the wizard and browse your samba shares. You should see the printer there, hanging from your samba server. User your disks with the drivers and it should be a done deal.

• /etc/exports: remember to set up this if you are using NFS shares. I export the same things as with Samba.

/media 10.0.0.0/24(rw,sync,no_subtree_check)

Copying the image to the CF

Ok, so now you have a perfectly working system in your virtual machine. That’s great. Time to copy it to the CF. Convert your VDI file to a raw image as shown below, and dump it to the CF.

If you are using the OSE edition of VirtualBox:

vditool DD machine.vdi raw_image.img

If it is the closed-source release, vditool is not available. You’ll have to do it like this:

VBoxManage internalcommands converttoraw machine.vdi raw_image.img

Once you have that, simply dump it to your CF with dd and you are good to go! Mine is detected on /dev/sdc

dd if=raw_image.img of=/dev/sdc bs=1024

Other installation tips

Going for a completely read-only system, even running something as mldonkey, is not that complicated either. Just trace the paths that need to be written frequently with “lsof /” and if you find something that has to modify many files, either replicate the file-structure on tmpfs/ramfs (preferably the former), or better yet, use aufs/unionfs on a temporary file system.  Thats the great thing about the live-system approach that I”ll be using next. I’m still working on some rough edges, but I’ll blog about it as soon as the results are perfect :)

Another, simpler approach if you don’t mind the hassle and some extra wear on your CF, simply make a PXE install and customize afterwards following my steps. Here is a nice and easy tutorial to make an Ubuntu  net installation in case you want to try that.

Anyway I hope you find this useful. It certainly was for me. Let me know if you find a cheap thin client on Ebay and follow this tutorial. A noiseless server does really make a difference!

A while ago I was looking for a similar noiseless solution for my brother in law and a friend, and the itch started all over again. I decided it was a good moment to upgrade my system. The low power consumption and being fanless were a must, but I also wanted it to have integrated gigabit ethernet. So I though using a Thin Client would be a good solution. These are normally fanless and have very little power needs, and some even have decent processors and Gigabit Ethernet. After looking for a while, I settled for a Fujitsu-Siemens Futro S400 that I found dirt cheap in Ebay.

It has GbE, and an AMD Geode NX 1500 processor that is actually an embedded Athlon clocked at 1GHz that runs on 6W: not bad, given it was introduced on 2004. The S400 boots from a 256MB CF and even has a spare IDE connector on-board. It lacks an internal power connector though, so I had to resort to using external media. One could get the 5V/12V from the motherboard, especially since the PCI port is not being used at the moment, but the system already gets hot enough as it is. I don’t want to add another heat source for now. However, unlike last time, I wanted to keep the OS on the compact flash. Sometimes the external media might not be available to boot, and besides… CF are fast for these things. I just swapped it for an 8GB/x133 I had laying around, and that’s much more than I need.

Installing on flash memory, however, requires certain considerations to prevent the installed OS from degrading due to a CF failure. These devices usually have  a limited number of write cycles, so the set up must minimize the write-operations as much as possible. Once it is done, you can use most thin clients exactly as any other computer.

Basically:

1. Create a minimal installation on local machine
2. Customize it to your needs and add any other required packages
3. Make it possible for the root FS to run being mounted read-only
4. Dump the installation to the CF and make it bootable
5. Profit!

It’s not rocket science, but I’ll be writing something about it  for those who could find it useful. Check out the home server tag if you are interested, and good luck!

UPDATE: Here is a photo of the AC adapter, in case you want to Google it.

Since the project has advanced so much over the last months, we have decided to boost the release closer to the 1.0 milestone. Many things have been improved, in stability, features  and performance. The Windows build has received some attention, and though it still has a lot of issues, Stefan de Konik has built a beta Windows package for people to try it out and help us sort out the rest of the problems. Great job as always, Stefan! The admin part is still not running under Windows, but you can always create the necessary config files on another environment and try out the Windows binaries.

In this release the caching mechanisms have been fixed, the web server can now be bound to multiple IPs and ports at the same time, a new balancing strategy has been added (so sticky sessions can now be implemented, for instance) …

To find out more about it, read the official release note for 0.98

Try it out. Cherokee is the fastest web server there is right now.

Her you have some links:

• The mailing lists
• Download link
• Online documentation