The problem:

I have a directory full of unclassified media files, some are duplicates, some aren’t, and each one follows a different naming convention.

I even try to classify them from time to time, so you can throw some directories into the pack. Sometimes, I even create two or three directories for the same group-series-category-whatever before I realize there is an existing one with a slightly different name. And frequently a lot of files remain unclassified, many of which could fit into one of the directories I mentioned.

Of course … whenever a new file arrives to my home server, it gets thrown into that very same directory, so Chaos keeps spreading, as it always does.

To clarify things, lets show an example:

drwxrwxrwx 1 user user      4096 2010-07-03 11:50 01_Battlestar_Enterprise
drwxrwxrwx 1 user user      4096 2010-07-03 11:42 02_Startrek_Galactica
drwxrwxrwx 1 user user      4096 2010-07-03 11:50 03_battlestar.enterprise-season.1
-rwxrwxrwx 1 user user 220393472 2010-07-03 02:49 battlestar.enterprise.s1e01.avi
-rwxrwxrwx 1 user user 221227008 2010-07-03 02:50 Battlestar_Enterprise_1_22.mp4
-rwxrwxrwx 1 user user 195393472 2010-07-03 02:49 startrek.galactica.4x15.[ripper_22].mkv

As you can imagine, sorting things up can get really tedious, and there is no automatic way of doing it that I know of.

I had some time this morning and got fed up with it. Every little piece of help is more than welcome, and here is where Python comes to the rescue.

The solution:

There are dozens of ways to do this, but I ended up coding a quick hack to help me sort things out.
It just compares the names of files and directories, and estimates the similarities. Anything above a 50% match is usually correctly estimated.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# (C) 2010, Taher Shihadeh
# Licensed: GPL v2

"""
The script works based only on names of files and directories in a
non-recursive manner.

It takes a path as parameter and tries to determine if the names of
the contents look alike.

It removes separator characters, numbers and file extensions prior to
the comparison.
"""

import os
import sys
import string
from operator import itemgetter

FAST = False # Change this to skip file-to-file comparisons
SEP  = '_-+~.·:;·()[]¡!¿?<>'

def main (path):
    lst1    = os.listdir (path)
    lst2    = lst1
    len_lst = len(lst1)
    count   = 0.0
    results = []

    for x in lst1:
        for y in lst2:
            if x==y:
                continue
            x_dir = os.path.isdir(x)
            y_dir = os.path.isdir(y)

            if FAST and not (x_dir or y_dir):
                continue

            result = {'A': (x, x_dir), 'B': (y, y_dir)}

            str1, str2 = x, y
            if not x_dir:
                str1,_ = os.path.splitext (x)
            if not y_dir:
                str2,_ = os.path.splitext (y)

            result['factor'] = compare (str1,str2)
            results.append(result)

        lst2.remove(x)
        count += 1
        print >> sys.stderr, '%.2f%% done' %((count / len_lst)*200)

    show(results)

def split (str1):
    trans = string.maketrans(SEP, ' '*len(SEP))
    return str1.translate(trans).split()

def clean (lst):
    assert type(lst) == list
    return filter(lambda x: not x.isdigit(), lst)

def compare (str1, str2):
    """Return similarity factor as percentage"""
    aux1 = clean (split (str1.lower()))
    aux2 = clean (split (str2.lower()))

    set_or  = set(aux1) | set(aux2)
    set_and = set(aux1) & set(aux2)

    return (float(len(set_and)) / float(len(set_or)))*100

def show (results):
    """Show most similar last"""
    for x in sorted(results, key=itemgetter('factor')):
        a,b = x['A'],x['B']
        if not b[1] and a[1]:
            a,b = b,a
        print '%.2f \t %s \t --> %s' %(x['factor'], a[0], b[0])

if __name__=='__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        path = os.getcwd()

    main (path)

I don’t think any one is going to use it, but what the hell. It’s a big Internet ;-)

Everything was recorded, so we will upload the slides and videos of all our sessions really soon. For now, only the photo gallery is available. Take a look at the mugshots.

I’m really glad we could make this Summit. It surpased all my expectations. By far. It was an unbelievable experience, and we had lots of fun. Take a look at our family photo. If you want to know which of the guys above is me, here’s a clue: “In brightest day…”.

I’m really looking forward to the next summit. Cherokee Summit 2010 was awesome. I’m sure the next one will be even better.

Before writing about it,  I wanted to give it a run to see if it could be of any help to a rather absentminded guy. I’m sure many long time Emacs users out there are forgetful at times. I know I am. It seems to fit the profile somehow ;-)

Since I couldn’t rely too much on my memory for these things, I had to find a task management solution. That’s where Org-mode comes in.

If you are like me, maybe Org-mode can save the day. I seem to be able to organize my time a lot better since I started using it.

Org-mode is a mode for keeping notes,  ToDo lists, and project planning in Emacs, with a fast and effective plain-text system. It seems awfully spartan  and simplistic at first, but it is nothing less than magnificent in features. Being a part of Emacs is also a plus for me, since it is the first thing I install on any platform I happen to be working. Besides the OS independence, not being tied at all to a particular application does get extra points. Formats may vary over time, but plain text files are here to stay.

These days I’m using it as an outliner, as a note-taking application, to manage my accounting and, most importantly, as a Getting Things Done (GTD) tool. I don’t quite yet use it for Web and PDF Authoring, but it never hurts to know I could if I wanted.

And for now the deal is working pretty well for me. It is very flexible, has lots of other uses, and also a very rich and knowledgable community, so I totally recommend you take a look at some of the links of this post. It will be worth your while.

You can read Alvaro’s announcement, or you can check out the Summit web-site.

Cherokee will be an important topic, but it won’t be the only one. Those will be a couple of days fully dedicated to High Performance and Scalable Web topics, so there’s room for everyone to join in.  We are commited to reaching the 1.0 milestone of Cherokee by then, so we will also have a party to celebrate it.

It’s going to be fun. I’ll be a speaker at the summit and I’m really looking forward to personally meeting many of the members of the project. Thanks to our sponsors we’ve managed to make the event completely free, so don’t forget to register while we still have free spots!

UPDATE: We’ve written a little brochure (~100KB) that can be used to  let your colleagues know about the summit. Do not hesitate to send it to any coworker or friend who would be interested in attending a High Performance and Scalable Web event.

We wanted to brag about our little baby. After all, not every serious web server out there has a killer interface to configure it. Take a look at our Cherokee Web Server introductory screen-cast.

You might want to see it at full screen for readability.

It’s just one of many to come. We’ve got some more planned, so I’ll let you know when they’re ready.

If you expect a party kit from us you’ve come to the wrong place. We actually believe our software is so good that it is a prize for its own merits. It has been a while since I last announced one of our releases, mainly because I didn’t have much to add besides what was told on the official announcements. As always, a lot of development effort is being invested in our flag product, and this is something that doesn’t go by unnoticed. This weekend we decided to release Cherokee 0.99.25. As you can tell by the .25 part, lots of fixes and enhancements have been added steadily release after release.

I hope you enjoy it. We’ve tried to update all the documentation for this release, and we’ve automated most of the recipes in our cookbook by adding lots and lots of configuration Wizards, so hopefully you’ll be able to set up anything in a matter of seconds. As always feedback and feature requests are more than welcome at the mailing lists. Here are links to download the tarball and the online documentation.

From: torvalds@klaava.Helsinki.FI (Linus Benedict Torvalds)
Newsgroups: comp.os.minix
Subject: What would you like to see most in minix?
Summary: small poll for my new operating system
Message-ID: <1991Aug25.205708.9541@klaava.Helsinki.FI>
Date: 25 Aug 91 20:57:08 GMT
Organization: University of Helsinki

Hello everybody out there using minix -

I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I’d like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

I’ve currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I’ll get something practical within a few months, and I’d like to know what features most people would want. Any suggestions are welcome, but I won’t promise I’ll implement them :-)

Linus (torvalds@kruuna.helsinki.fi)

PS. Yes – it’s free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that’s all I have :-(.

Regarding Linux portability, one could easily loose track. Some hobbies can change the World.

• Wifi: it ships an Atheros AR9285 wireless card. From the Official Linux Wireless wiki we can see that it is supported on kernels >= 2.6.29. Jaunty comes with 2.6.28, but it is not a problem:

sudo apt-get install linux-backports-modules-jaunty

• Sound: Update ALSA. This is for the latest snapshot:

sudo apt-get install build-essential
wget http://ftp.kernel.org/pub/linux/kernel/people/tiwai/snapshot/\
alsa-driver-unstable-snapshot.tar.bz2 -O -| tar xvj
cd alsa-driver-unstable
./configure --enable-dynamic-minors
make
sudo make install-modules
echo "options snd_hda_intel model=hp-dv5" | \
sudo tee -a /etc/modprobe.d/alsa-base

Problem solved. Reboot and enjoy.

I mention this because the time to buy is getting closer and closer, and yet I’m astonished that so little people out there are making a big fuss over the next big thing: ARM netbooks and Linux. Sure, we already have some of these out there from Skytone and Elonex -and probably others-, but those are fairly limited machines performance-wise.

I’m talking about cheap mini-laptops that can be used to surf the web, write reports and even play 720p video, all with an extremely low power consumption, 10+ hour battery life and very little heat generation. These machines are already on their way, will use newer and more powerful ARM processors and will hit the market in the following months.

I believe this is, in fact, a silent revolution that doesn’t even ripple the surface. This will change everything. And why is that? Well… for starters there is a huge market for something like this, like the sells of anything Netbook-related have been steadily showing lately. An it seems hardly possible that Microsoft will release an ARM enabled Windows XP. This means Linux will get yet another boost in market share when these machines become mainstream, although I’m pretty sure Microsoft will still claim a 90%+ share in the netbook segment. Despite this alleged 90%, the rules of the game have changed, and netbooks are not playing by Microsoft’s rules any more.

Things are changing. And with this, cheap, ubiquitous, multimedia network-enabled machines will become a reality. And those will be powered by free software, at last. Debian has been supporting ARM chips for a long time, Ubuntu does since release 9.04 and other mobile and embedded devices have a long history with Linux. And lets not forget about Android! Oh man, I can hardly wait to get my hands on one of these jewels!

It’s a revolution.  It’s quiet, but it’s happening.

The easiest thing to do would be installing the system normally through PXE (look for the link at the end of this post if you’re interested), but there is a problem. Flash devices have a limited number of write-cycles, and wear levelling is not used on consumer grade cards. There are file systems optimized for flash devices -such as JFFS2- but the benefits are not obvious here. As I understand it, these are designed for industrial devices with direct access to the memory cells. Consumer devices have an abstraction layer that make them transparent and enable us to use them as any other normal storage. It should provide everything we need, such as wear levelling. But as I said before, it normally doesn’t for consumer-type devices.

Alternatives

Having discarded the optimal solution for both technological and budget limitations, we are left with three alternatives.

1. Use the CF as an ordinary device, with total disregard for the premature death of the drive. This is the simplest, and provided you have a lifetime warranty for your media it might not be such a bad idea.
2. Use the CF as if it were a LiveCD, maybe even adding persistence for our changes. This option should prove perfect if we want to use the Thin Client as a desktop system. Temporary changes are written to RAM, permanent changes will be stored on disk and even software updates will remain between reboots. It is a bit more complicated, so I’ll leave this in my TO DO list for now.
3. Install everything by hand on a local copy, configure the system to be able to run with as little disk access as possible (ideally, from a read-only root file-system), and dump it to the CF. Since space is not a problem for me (I have both a 1GB and 8GB CF cards), I’ll prefer this approach. In case everything was set as read-only, we would only  have to remount the file systems as writable during the process. As for SquashFS, though it seems an ideal choice for an embedded system that needs no upgrades, I’ll discuss this in other posts. It is simpler to deal with a “standard” system, not having to recreate binary boot images every time you update the box.

My home server is used to share media and printers on my local network, manage downloads, and above all, provide a permanent access to my home network from wherever I might be. This is, by far, the handiest thing ever. At least it is for me. You never know when you are going to need to bypass Internet filters when you are roaming, for example. The wonders of SSH truly never cease to amaze me :)

I’ve opted for letting some writing be done to the CF, albeit the write-intensive tasks have been dealt with, because it is easier for the system to be updated, and because some of the services I’ll be using refuse to work on read-only mode. However, my next entry will be about using aufs/unionfs, and replicating this setup on totally read-only and compressed system that will run from the 256MB standard CF that comes with the computer.

Things to install

• Basic system: since Jaunty is still in Alpha stage, downloading the whole ISO is not worth it. Just grab the NetBoot mini ISO and download the rest as you go. You’ll have to update frequently until it is released anyhow.
• linux-image-server: A scheduler less suited for multimedia but with some advantages for a server.
• ssh: Access point to/from your server.
• cups: printer support. Note that CUPS alone will add around 100 MB to your installation.
• portmap, nfs-kernel-server: NFS file server.
• samba: sharing files and printers for Windows boxes.
• nmap, mc, wget, curl, unrar, p7zip-full, arj, unzip, unp, unace, qemacs-nox: some handy tools I can’t live without. Mostly Swiss-army knives, decompressors and a tiny Emacs-like editor. It’s bigger than mg, but it has UTF8 support. I love Emacs but it is huge for a system like this.
• usbmount, ntfs-3g: to automatically mount USB devices. Nice if you plan to plug and unplug your external drives to share them on your LAN.
• mldonkey-server: for our every day download-needs.

These were the steps I followed. Keep in mind these are mainly notes to myself. They should work for you, but they might not. At the very least these steps will give you some useful hints. Please note that there might be some rough edges in the tutorial, since I actually wrote this after having everything installed. I tooks some notes during the process, and this setup is working flawlessly for me. I don’t think any important (read non-trivial) steps were left out, but if that’s the case and it is giving you any kind of problem, just ask in the comments!

The CF shouldn’t suffer much IO, so the idea is to install and customize everything on a virtual machine, and then dump the resulting image to the CF after making every needed adjustment.

I originally intended to make the system completely read-only, but then decided that preventing /tmp and /var/log from massacring the CF would be more than enough. You can skip completely some of these steps, since the root-fs will not be totally read-only. For that, check the next post.

Installing the base system

I’ll be using VirtualBox because I find it easier to work with than plain QEMU, particularly since I’m going to be needing the network from within the virtual machine. Just create a new one with a disk size that can fit in you CF. I’ll use a 1GB card that I have laying around. Also configure the CDROM drive to use the NetBoot mini ISO that you have downloaded.

Before booting the machine, add PAE/NX support to the virtual CPU. The generic kernel doesn’t need it, but once you try to boot with the linux-image-server branch you will need this to proceed. That is done from the virtual machine definition section following the path: Details -> General -> Advanced.

Boot the machine and type ‘cli’ if you want to minimize the installations. Then manually partition the hard drive if you want to use the new EXT4 file-system type, and go through the process as you normally would. Of course, disregard the swap partition. You can’t have that on a CF or it will die soon for sure.

I use the French repositories because those are the fastest for me. I always start by modifying the list of repositories on the recently installed system, in case I want to install things not present in universe. Edit it manually or just append the needed info.

echo "deb http://fr.archive.ubuntu.com/ubuntu jaunty \
main restricted universe multiverse" >> /etc/apt/sources.list

And install the packages.

sudo su -
apt-get clean
apt-get update
apt-get install linux-image-server openssh-server samba cups \
 wget p7zip-full portmap nfs-kernel-server nmap mc \
 qemacs-nox unp arj unrar unzip usbmount mldonkey-server
apt-get clean

No real need to mention it, but don’t forget to delete the generic kernel and associated files if you want to save another 100MB or so.

Now to handle issues related to readable-writable paths, as well as config files. Some steps are only useful if you will mount the rootfs as read-only. I’ll mark them as “Optional”:

• Optional: /etc/mtab: We will link it to a file that can describe the mounted file-systems without requiring write access.

rm /etc/mtab
ln -s /proc/mounts /etc/mtab

• /etc/network/interfaces: static IP for the server. Bare in mind you’ll need writable /etc/resolv.conf and /var/lib/dhcp3 if you intend to use dynamic IPs. You’ll probably have to edit /sbin/dhclient-script to account for this if you are going for a full-ro filesystem. This is how my /etc/network/interfaces looks like.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.0.0.100
gateway 10.0.0.1
broadcast 10.0.0.255
netmask 255.255.255.0

• /etc/etc/fstab: define the mountpoints. Ensure that directories that need to be writable ar mounted as tmpfs rather than letting the system write them to flash. Alternatively you can mount them on external media. You need this to run the rootfs mounted as read-only. Even if you are not going to do so, you *MUST* set the ‘noatime’ flag when mounting, or at least ‘relatime’. If you don’t, every single access to the files will require a read operation to modify the access time, and this is pure overhead for the CF that will significantly shorten its life span. Using journaling is probably another bad idea, since many commits will also affect the CF. Either use EXT2, or use this as a temporary solution and go for the SquashFS-Live system of approach #2.

/dev/sda1 / ext4 noatime,errors=remount-ro 0 0
proc /proc proc defaults 0 0
tmpfs /var/log tmpfs defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
tmpfs /var/lib/urandom tmpfs defaults 0 0
# Define the following ones if you use NFS or dynamic IPs.
# Writable paths should be tmpfs or be on external media.
# Both settings are OPTIONAL for our current setup.
tmpfs /var/lib/nfs tmpfs 0 0
tmpfs /var/lib/dhcp3 tmpfs defaults 0 0

• Logging: either log to a tmpfs as shown above, to an external media, remotely or remove logging capabilities. This step is not optional at all. If you choose any option other than the first one, you’ll probably want to take a look at /etc/syslog.conf, /etc/default/syslogd and your /etc/rc?.d for links to /etc/init.d/sysklogd. Don’t forget the syslog.conf man entry. Note that logging to a tmpfs will give you some warnings at boot-time since some files are expected to exist. It’s probably not such a good idea, since you’ll loose the logs between reboots, troubleshooting vulnerabilities becomes more difficult, and so on. If you do proceed with the idea, you should keep this in mind: some logfiles are expected to be present at boot-time, so you’ll get some errors if you boot with an empty /var/log. Mldonkey also doesn’t boot if it can’t access the logs, so populate the directory with a  trivial script such as the following. It  must be run at boot-time before /etc/init.d/bootlogs.sh and /etc/init.d/sysklogd are invoked:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          madoka
# Required-Start:    $local_fs
# Required-Stop:     $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      1
# X-Start-Before:    bootlogs sysklogd
# Short-Description: recreate /var/log structure for Madoka
# Description:       recreate /var/log structure for Madoka
### END INIT INFO

do_start () {
 mkdir -p /var/log/mldonkey
 mkdir -p /var/log/news
 cd /var/log

 touch \
 dmesg          \
 mail.warn      \
 user.log       \
 daemon.log     \
 messages       \
 debug          \
 auth.log       \
 mail.err       \
 syslog         \
 mail.log       \
 kern.log       \
 lpr.log        \
 mail.info      \
 news/news.crit \
 news/news.err  \
 news/news.notice

 chgrp adm dmesg
 chown syslog:adm \
 mail.warn       \
 user.log        \
 daemon.log      \
 messages        \
 debug           \
 auth.log        \
 mail.err        \
 syslog          \
 mail.log        \
 kern.log        \
 lpr.log         \
 mail.info

 chown news:news news
 chown root:news news/*
}

case "$1" in
 start)
 do_start
 ;;
 restart|reload|force-reload)
 echo "Error: argument '$1' not supported" >&2
 exit 3
 ;;
 stop)
 # No-op
 ;;
 *)
 echo "Usage: $0 start|stop" >&2
 exit 3
 ;;
esac

I saved it in /etc/init.d/madoka, and added it to my default runlevel to execute before init.d/bootlogs.sh and /etc/init.d/sysklogd

update-rc.d madoka start 05 2 .

Adding some more goodies

• mldonkey: I just made symbolic links to an external drive. /var/lib/mldonkey/incoming, /var/lib/mldonkey/temp and /var/lib/mldonkey/torrents are the most frequently written directories by far. It also rewrites the configuration files whenever changes are made. Those are also stored under /var/lib/mldonkey/.
  

• /etc/usbmount/usbmount.conf: this is great if you are going to have lots of external drives. It automatically mounts the devices wherever you choose to (in my case, under my shared samba directory). It just works after rebooting. I only edited the file to specify the mount point and to add more file-systems that I wish automatically mounted:

FILESYSTEMS="ntfs vfat ext2 ext3"

• /etc/samba/smb.conf: to share files and printers, edit yours to resemble these pointers.

# add to [global] section, and edit the rest to your heart's contempt
workgroup = WARPCORE
printcap name = cups
printing = cups
security = share
socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
# make sure [printers] section looks like this
[printers]
browseable = yes
printable = yes
public = yes
create mode = 0700
guest only = yes
use client driver = yes
path = /tmp

#define at least one share point
[shared]
comment = Shared media
path = /media
guest ok = Yes

To have the printers actually work you’ll need to configure them with cups. After having everything tuned, remember to mount the root file-system read-only. Adding the printer is trivial thanks to the administration web interface. Just forward the needed port or make cups accept connections from interfaces other than localhost, and launch your web browser.

ssh -L 2000:localhost:631 10.0.0.100

Then, access the configuration wizard visiting http://localhost:2000. On Windows systems you can add this by simply add ing a new printer with the wizard and browse your samba shares. You should see the printer there, hanging from your samba server. User your disks with the drivers and it should be a done deal.

• /etc/exports: remember to set up this if you are using NFS shares. I export the same things as with Samba.

/media 10.0.0.0/24(rw,sync,no_subtree_check)

Copying the image to the CF

Ok, so now you have a perfectly working system in your virtual machine. That’s great. Time to copy it to the CF. Convert your VDI file to a raw image as shown below, and dump it to the CF.

If you are using the OSE edition of VirtualBox:

vditool DD machine.vdi raw_image.img

If it is the closed-source release, vditool is not available. You’ll have to do it like this:

VBoxManage internalcommands converttoraw machine.vdi raw_image.img

Once you have that, simply dump it to your CF with dd and you are good to go! Mine is detected on /dev/sdc

dd if=raw_image.img of=/dev/sdc bs=1024

Other installation tips

Going for a completely read-only system, even running something as mldonkey, is not that complicated either. Just trace the paths that need to be written frequently with “lsof /” and if you find something that has to modify many files, either replicate the file-structure on tmpfs/ramfs (preferably the former), or better yet, use aufs/unionfs on a temporary file system.  Thats the great thing about the live-system approach that I”ll be using next. I’m still working on some rough edges, but I’ll blog about it as soon as the results are perfect :)

Another, simpler approach if you don’t mind the hassle and some extra wear on your CF, simply make a PXE install and customize afterwards following my steps. Here is a nice and easy tutorial to make an Ubuntu  net installation in case you want to try that.

Anyway I hope you find this useful. It certainly was for me. Let me know if you find a cheap thin client on Ebay and follow this tutorial. A noiseless server does really make a difference!