natowelch

Cross-posted from n8o.r30.net:

it appears my fortnight of butting my head up against dbmail, ecryptfs, mysql, and sqlite is nearing its end.

My error in attempting to migrate from the Mysql to the SQLite database backend for dbmail became apparent as I began to import three months of mail archives dumped into mbox files through the dbmail IMAP4 server from the old mysql backend. It's somewhere on the order of 15,000-20,000 emails (I get a LOT of email). Once I got up and over ten thousand, the performance slowed down a lot. Even inserting new mail using fetchmail took a second or two per message. Some filtering and de-duping functions just plain stopped working, since they would always time out before completing.

So, it was back to Mysql. I now have most of the mail I've received since I deprecated the Mysql database dumped to mboxes and being transferred back via IMAP4 importation to the Mysql backend... and it's WAY faster, thank goodness.

To get here, though, I had to find a solution to my encryption problem, which is what drove me away from mysql in the first place. Well, no, not really. What drove me away was the fact that it's really not possible to encrypt SOME of a mysql database if it is required to be stored using the innodb storage engine. I had been trying to encrypt just the dbmail database, in its own folder. What was really called for was to just scrap the whole piecemeal and encrypt all of /var/lib/mysql wholesale.

At first, I was trying to do what I had been doing with the sqlite database, and symlink /var/lib/mysql to /home/nato/mysql . That would place the entire database space inside my encrypted home directory. Unfortunately, mysql would not, for reasons I did not investiagate throughly.

Instead, I just used the existing ecryptfs utilities to create a separate excryptfs encrypted file system just for the mysql database. That seemed to work great. All I had to do was tweak one file to make the mountpoint /var/lib/mysql .

So, now, I am celebrating, because my mail is now fast and usable once more. I can hardly believe it. It's been a depressing two weeks.

But, I win. Again. Through perseverance and patience. What a relief!

Now, on to that Google Wave preview I received from Jeff! It's a little slow, interface wise, but I'm digging it so far.

Cross-posted from n8o.r30.net:

I spent all night last night and all day today importing several months worth of email from the mysql backend of dbmail into the sqlite backend of dbmail using Claws-mail to import mbox files over IMAP. I have been unable to use fetchmail to load new mail, and unable to browse existing mail. the dbmail-lmtp daemon that receives it from fatchmail sporadically complains that "the (sqlite) database is locked". shouldn't this be able to do two things at once? I didn't have this problem with MySQL.

To make matters worse, the IMAP performance of jsut browsing mail has slowed intolerably. Apparently, sqlite3 can't handle a GB database of email messages as well as Mysql can.

*wince*

I guess it's time to figure out how to migrate back to Mysql.

I also recall never seeing the corruption/header mangling problem on the Mysql backend either. I've even developed a theory about how it happens. It always only pops up when there are single quotes in the message header, and the symptom is that the header message block, as stored in the database is missing it's trailing characters. sometimes they're just missing the ending carriage returns; sometimes they're actually missing trailing characters on the final header line as well.

Could it be that dbmail is properly escaping single quotes in preparation for insertion into the database, by adding backslash characters as customary, but the forgetting to account for the extra length those escaping characters add to the header string? If the escaped string isn't the same length, but the length isn't updated, it might be chopping off the characters at the string's end by mistake.

When I finally get my email back to where I want (if ever), I may file a bug report with dbmail. but fuck this.

Cross-posted from n8o.r30.net:

There is something terribly wrong with dbmail.

For some reason, it was mangling random email headers, such that the first line of the message body was being tacked onto the final line of the message header, minus one missing character on the end of the final header line.

I managed to isolate it down to the existence of at least two single quotes characters, which are not uncommon to find in the spam scan report tacked onto the header of each delivered message. dbmail does not parse it correctly when inserting it into the sqlite database - although it has never swallowed any messages whole, as far as I can tell.

I have worked around this by altering the exim4 mail server configuration to replace any single quotes int he spam report before inserting it.

What a horrible bug.

Cross-posted from n8o.r30.net:

I'm settling into my new home on Ubuntu Karmic. It's taken a few days to migrate, clean, houskeep, prune, and integrate from it's home on the jaunty partition (and the encfs encrypted filesystem) onto the Karmic partition, which uses ecryptfs. It's all gone very smoothly, without a snag in sight.

It's startling when rox filer can fit every single file and folder in your home directory, including all the hidden files, into ONE screen. Neat.

I have also managed to get my dbmail email backend switched over from a mysql/innodb database to one using sqlite on my encrypted home. It was not difficult to discover the correct file permissions dbmail requires for the sqlite database file, and the dbmail IMAP daemon doesn't seem to mind that it's database is inaccessible until I login for the first time (the dbmail smtp mda runs entirely in userspace, so it doesn't mind either), and doesn't skip a beat when it becomes available on login. Not bad. I'm using fetchmail to transfer mail from my server to dbmail periodically.

Another great thing about ecryptfs is that, while yoru encrypted home is unmounted as soon as all your sessions are logged it, the kernel keeps your key on hand, just as long as you've logged in at least once, so any cron jobs that the user runs prompt the system to automatically re-mount the encrypted home partition before the cron job runs - and, of course, the partition is unmounted again when it completes. Pretty handy.

What this means is that all the authorized logins for my user will get access to the encrypted partition, so long as I have logged in with a password at least once since the last boot. As soon as the system shuts down, the keys are lost until I login again.

I still have to figure out what things to backup and what to delete, and reconstruct a remote backup scheme for everything. I have no fewer than *12* different places where email archives exist over the years; they need to be consolidated, de-duplicated, and stored in some uniform way.

I found a cache of old, old writings and music I was playing with from 2002 all the way back to 1998. That was fun.

Cross-posted from n8o.r30.net:

After downloading the just-released Ubuntu 9.10 "Karmic Koala" live cd via bittorrent at a blistering 800-900Kb/second (seriously, it took 20 minutes to download, and 10 to burn to cd. wow.), I re-partitioned my hard disks and took it for a spin via a clean install.

If I recall correctly, I haven't done a clean install of an operating system since I switched from Debian to Ubuntu many years ago (and I'm not even sure about that - I may well have pulled off some weird hack to upgrade from Debian to Ubuntu without a clean install, but I doubt it). This is a tribute to the stability of Ubuntu, the underlying Debian OS, and the GNU/Linux bedrock at their foundation.

Nonetheless, there's been a LOT of cruft and hackery that have accrued to my filesystem over the years, and I felt the need to make a fresh start and explicitly import the parts of my data and settings I know I wanted to, rather than implicitly submit to whatever deeply ancient and subtle things which may lurk in my filesystem after years of being forgotten.

So, with an eye toward the future, and buoyed by my success with installing the Karmic beta release alongside the existing desktop backup image on my laptop, I decided against just upgrading my existing Jaunty Jackalope release on my desktop PC (an operation that has always gone quite smoothly), and decided instead to exercise the live cd installer's option to install Karmic alongside Jaunty, and dual-boot into another partition. The results have been excellent, and smooth. I'm quite happy with it.

Now, those who know me know I've been a GNU/Linux user for about ten years now. When Ubuntu came out after 2004, it prided itself on targeting average, non-technical desktop users. Since my introduction to GNU/Linux preceded the development of Ubuntu's suite of graphical tools, I knew how to do everything on the command line, instead. This turned out to be a liability with Ubuntu - not because of conflicts with the tools, but because friends of mine using Ubuntu on their own would often find and use the graphical tools to do things I only knew how to do with the obscure command line hacks. With this install, I was pleased to finally learn how to use those, and do things "the Ubuntu way". And it did everything pretty much as I wanted it too. I've put a lot of work into developing and maintaining my own solutions for things, but I am keen to relinquish that and use systems other developers maintain, if they work just as well or better.

First, I used Gparted, rather than fdisk, to edit my partitions, resizing the Jaunty partition down, and adding two smaller 16GB partitions, one for the karmic installation, and one reserved for future use. I plan to mount the Jaunty partition and use it's home directory for both installations, so I can boot back and forth if necessary with the same home directory data. Resizing filesystem partitions takes a while, which is normal, but it went flawlessly.

The second big change I'm going to make is to switch my encrypted filesystems from Encfs to the eCryptfs-based file encryption standard Ubuntu has been developing. When released in Jaunty, last spring, Ubuntu provided users with an encrypted directory, Private/ , which existed alongside Documents/, Desktop/, Music/, etc., where users could move data they wanted to protect, and link to it from its previous location instead. This was pretty much what I'd developed for myself with Encfs, explicitly choosing what I want to protect. The big difference was that the eCryptfs system used the user's login password, rather than a separate one, to wrap the keys that unlocked the encrypted directory, meaning that the Private directory would already be mounted as soon as the user entered their password to login. With my Encfs setup, I had to enter a separate passphrase in an additional step.

But in this release, Ubuntu has continued to improve this system. Now a user's entire home directory can now be encrypted, rather than just the Private/ sub-folder. These two improvements on my Encfs setup go a long way toward making encrypted protection of user data seamlessly transparent, and it's a wonderful advance for the privacy of the Ubuntu OS.

As to the details of installation, I had ZERO problems. Sound just worked (Pulseaudio and all, which frustrated me so much previously that I uninstalled it entirely), networking just worked (Network Manager now grants more user control), the installation of restricted Nvidia graphics drivers just worked (complete with Compiz-powered OpenGL desktop effects eye candy), installation of restricted extras like the Adobe Flash plugin, and mp3 codecs just worked, and a few annoyances I'd been tolerating for years have now disappeared.

Furthermore, it's even snappier in performance. Nautilus file management, Firefox 3.5, and the gedit text editor all loaded and responded noticeably faster than on Jaunty. My boot time was 20 seconds to login, and an additional 15 seconds to desktop - not mind blowing, but definitely better than Jaunty on the same hardware.

I've also noticed something on my laptop - since I installed the Karmic beta on it, I've left it on for longer periods, and it has not tended to overheat, hang, or spontaneously shut off as much as it did on Jaunty. Advertised Improvements to power management seem to be contributing to this.

I'm just tickled pink - this is a very solid release.

Cross-posted from n8o.r30.net:

I have finally completed (mostly) a piece of software I've wanted for a long time.

I don't know how to nutshell it, exactly. It's a frontend for unix shell commands and terminal emulators that gives me flexibility in how it executes commands.

If I want to run a command in a terminal emulator:

# sketsh ls -al

a new terminal window pops up, executes the command (in this case, getting a file listing), and then drops me into a bash shell so I can enter more commands. in the meantime, it has added the command to my bash history so that it is immediately accessible for editing (in case I do something wrong). Simple and handy.

Now let's say that I'm writing a shell command or script that interacts with my graphical desktop environment, but isn't intended to use a terminal window at all. Like many programs, I use the console output to print debugging messages that aren't normally needed. If I execute a command by prepending sketsh with it's -C option,

# sketsh -C ls -al

it will skip executing the command in a terminal window entirely, and just run in the background. If something isn't working right, I can just delete that option from the command line in the script, and the command will execute in a window so I can see the output. When I've fixed any problems, I can then put the -C option back in, and the terminal goes back away. Neat and tidy.

At other times, I want a terminal window to pop up to execute the command, but I don't want it to hang around when it's done (unless, as before, I'm debugging it). For that, there is the -H option, which explicitly tells sketsh to close the window immediately after the command completes. To debug, simply take that option out of the line it's on, and a full terminal pops up and sticks around, with the command already in history for examination.

In this process, and over the years, I have learned a LOT about skillful BASH shell scripting - and how much more there is to it. This pleases me.

It's the perfect century to be a hacker.

I just emerged from a long, drawn out process of evaluating the use of innodb database format, which is the mysql db storage engine required for use by my dbmail local mail server.

I hate it.

I remember having a real headache when trying to migrate mediawiki databases from the old to the new server last winter. innodb likes to plaster every table and every database into ONE giant file, making backups hell to make. I did not know this at first, leading me to have to find a way to explain to a client that I lost six months of work on his wiki, and NONE of the three incremental backups were recoverable. two days later, I figured out where they were stored, and was able to recover everything, but too late to save myself from the embarrassment or the cold chills.

At that time, I figured out how to separate the databases and tables into separate files just fine. But I have been hacking away with the innodb tables I've been using for a few months with dbmail, which I started using this summer to pick up the pace of my email archiving and search, and it turns out those separate files don't actually contain everything necessary. There is still essential data linked back and forth between files in the parent mysql data directory, and I have not been able to recover the data without it. This has forced me to recover from last night's backup image; meaning I have lost a day's mail. Not bad as data disasters go, but still frustrating.

This all started with a more interesting project of trying to use Ubuntu's ecryptfs utilities to create a transparently encrypted filesystem to back the mysql database files. It worked much better than the encfs I've been using in my home directory for many years. I may even switch that over after I upgrade to Karmic Koala.

The trouble with innodb has pissed me off so bad that I am evaluating switching dbmail's database engine over to sqlite, if it turns out that sqlite can handle the performance demands of searching large email archives like mysql.

One thing's for sure; I'm going to need to set up regular mysqldumps of any mediawiki or other innodb-using sites, because the file-based backups that work fine with myisam databases will not cut it. Luckily, there aren't many sites that require it; and their databases are not large.

Bad news: The Electric Sheep package I pulled from that launchpad PPA repository the day before yesterday is incompatible with the version of Mplayer I pulled from another launchpad PPS repo in order to get scaletempo support - the ability to speed up and slow down audio without changing pitch.

Good news: The version of Mplayer included in the upcoming Karmic Koala release of Ubuntu supports both scaletempo and Electric Sheep.

I'd really love to see Multi-pointer X, but last I heard, it wasn't going to be included. Oh well.

I have still been busy coding. I like it.

I have been building a PHP library to interface the new htdbshop backend for our tree site with an Endicia Label server, to generate USPS postage-paid shipping labels automagically for our trees. Nothing out of the ordinary, but I feel good about my work, at the moment. I like the labor it's going to save down the road.

There's been an interesting challenge in learning Postscript, a language I've never bothered to pick up, because, frankly, I hate printers. PDFs have traditionally settled in that milieu, so it rarely got any attention from me, either. Its details and formatting intricacies were opaque, as the web was so much more useful.

But now, I have to manipulate pdfs in order to position shipping labels just so, merge them so that they print two to a page, are concatenated into one pdf document per day, and get a packing pick-list prepended for our fulfillment crew. Eventually, we also want our logo stamped on them.

It would be cool enough to figure out how to do all this manually, with desktop tools, editors, etc. But my challenge is always a step above this: I have to script it, so that it all happens automatically, on the server, on demand.

I'm going to get a kick out of watching this thing run when it's finished; About as much as I am learning how to implement it.

It's my kind of wizardry.

The Drupal open stream API modules I referenced earlier are a broken mess, with way too many dependencies, manual installations, and no upgrade path.

But, my own application is coming along MUCH better. It's in rough shape, but I'm already pulling down my stream and displaying it in a usable way. If I re-index it a bit, it will be easy to throw it into a mysql database permanently.

I still have many qualms about how Facebook handles itself. I have to "register" the application with them, even though I don't intend to let anyone else use it (though if it works, what the hell, right?) But looking at just how MANY people are using this thing, I can feel the pull of the usefulness of it. Facehook, indeed. What I don't want to do is get sucked in only to discover some of the things I want to know about are not coming out of it. What are the odds?

I'm absolutely tickled pink to have written a basic inline spellchecker in Lua for my favorite programmer's text editor, SciTE.

I don't know Lua from a hole in the ground, but I have heard good things about it. It's an extremely simple syntax, which lends itself well to allowing non-programmers to write simple scripts for things like video games (Second Life comes to mind). It was pretty easy to cobble together a script from various Googled sources and examples.

~/bin/scite-aspell.lua:

function decorate_range(pos,len,ind)
   local es = editor.EndStyled
   editor:StartStyling(pos,INDICS_MASK)
   editor:SetStyling(len,INDIC0_MASK + ind)
   editor:SetStyling(2,31)
end

function highlight_word(txt,flags)
  if not flags then flags = 0 end
  local s,e = editor:findtext(txt,flags,0)
  while s do 
    decorate_range(s,e-s,128)
    s,e = editor:findtext(txt,flags,e+1)
  end

end

function inline_aspell(filename)
    spellh = io.popen('aspell list < '..filename)
    decorate_range(0,editor.Length,-1)
    for line in spellh:lines() do
        --print (line)
        highlight_word(line,SCFIND_WHOLEWORD)
    end
end

The script simply sends your file to aspell, grabs a list of misspelled from it, and underlines them all in red.

If you want to use this, be sure to add this onto the end of your
~/.SciTEUser.properties:

command.name.2.*=Inline Spellcheck
command.2.*=inline_aspell $(FilePath)
command.subsystem.2.*=3
command.mode.2.*=savebefore:yes
ext.lua.startup.script=$(SciteUserHome)/bin/scite-aspell.lua

... and restart SciTE. You should now find 'Inline Spellcheck' on your Tools menu.

I'm greatly relieved by this. I had been considering migrating to gedit as my default text editor. SciTE is lean and mean and fast, and has everything I've really wanted in a programmer's editor (syntax highlighting, quick syntax checking and build tools, regular expression search/replace, auto-indent, etc.), but the lack of native inline spellchecking (you can run it through external spellcheck filters easily enough, but that's just painful) has bugged me for a long time, and it's something that gedit has as part of its plugin architecture. gedit's external tools plugin - then one that's supposed to allow you to trigger compile/build/execute commands or other filters is HORRID, and the shortcut keys you're supposed to be able to assign them don't work half the time. That, and it still loads too slowly for my taste, even if you leave off most of the plugins I don't need.

Thankfully, then, I get to keep my favorite editor.

Pretend to care, will ya? ;p

**UPDATE:** Yes, there's a bug. It highlights all occurrences of misspelled words, even if they are short, and occur as part of words they are supposed to appear in when said words are spelled correctly. I will fix this eventually. **FIXED** in the above version. WOW that was easy.

I have been pressing on in recent days, to replace almost every desktop application I use with something better.

It started with dbMail, which I'm deploying on a mysql server on my local workstation. I haven't finished the deployment I want (which will be on an encrypted fs, hopefully), but the basics are working well enough.

Then, last week, I have been slowly migrating to Firefox 3.5, which has noticeably improved performance. Half of that is because the new version is faster. The other half is the fact that I've been using the same Firefox profile for years - probably since it was called Firebird. cleaning out the cruft and surgically adding back settings and configs deliberately has doubtless made things cleaner.

Last night, of course, there was the discovery of Aqualung for playing audio. I think I'm going to have it feed a local icecast server, so I can stream to my laptop in the other room (although it overheats way too easily, meaning I can't really leave it on when I'm not using it).

Tonight, was the trickiest search: replacing KMail. Since I was using dbmail as my backend, My choice of frontends was a lot more portable. I didn't have to think about migrating my data between this new choice. dbmail's mysql db should make anything pretty fast.

Ever since KDE4 came out, holy hell, has KMail been annoying. over the past year, the pet peeves have been piling up. Things like only including selected text in a message when replying, sporadic timeouts, and the fact that it leaves unencrypted traces of bits of messages and URLs I click on from email in strange places are starting to pile up.

So I went and checked out some of the trendier webmail packages. I figured, as long as I could use them to access an IMAP server, they couldn't be all bad. I'm not sure if I WANT to pile yet another application into the hands of the browser, so that it all crashes down at the same time, but I looked at it at least. I've never seen Gmail, frankly, but I hear it's fast "ajaxy" and innovative. It just doesn't make sense to me to let Google have my email, when I run a mailserver for dozens of people on my own. While nice and light, webmail packages like roundcube seemed a bit... skimpy for what I wanted. Plus, the AJAX designs in some of these packages essentially end up breaking the web. I would do things I expect to be able to do with links in a browser, like opening a message in a new tab, only to find out I could only open the message by double clicking it, which would replace the message list window with the message display window, which would make me take FOREVER to navigate through my inbox. Webmail that attempts to fuse email and the web, and manages to break BOTH is not a step up.

So I went back to look at the leaders in desktop email clients, and tried them out using IMAP accounts, instead of the old-fashioned POP3. I found some old, old, nasty bugs in Mozilla Thunderbird, which looked so promising, with the advanced tagging and saved searches features they came up with for 2.0. I couldn't believe it, but saved searches and message filtering conditions WILL NOT WORK, period, on an IMAP account if you're planning to use any but the most popular email headers. Mailing Lists headers? perhaps. but User-Agent? Forget it. This bug has been haunting Mozilla for years, but they don't seem to want to fix it.

I toyed with Evolution for awhile, but in comparison with Thunderbird, it was too slow. The interface design was awkwardly cryptic. I even toyed with the idea of going back to mutt, my favorite text console client from days of yore, which I still fire up when doing maintenance on my inboxes and archives from time to time.

But then, after search through package caches for long forgotten email clients, much like for aqualung last night, I found the second tier clients - the ones that aren't popular because they're MADE for geeks. This one had what I wanted Claws Mail big, nasty, robust command-line filtering facilities I could use to pass messages through to shell scripts, python, or php. Slim and fast. Old and stable. Very confgurable, like people used to make fun of KDE for being. Unlike Thunderbird, it allows IMAP-based searching of obscure email headers to be done quickly. It doesn't technically have saved searches, but I quickly noticed that the "quicksearch history" began to fill up as I used it, and that that history is stored in a plain-as-can-be text file in the preferences directory, so it would be pie to feed pre-set searches to that file prior to starting it up if I want them.

And then, I remembered waaay back, when I used to use this client, one of the things I loved about it, that I never saw anywhere else: numeric scoring. You can assign a score to a message based on filter conditions. It's like tagging, but with finger grained control. Tagging is nice and all, but what I wanted to do with them is set a way to subtly and precisely SORT messages, based on how their filter scores allow them to bubble to the top, or sink like a stone, in my inbox.

There's a lot to do to get this working, but I'm guessing I'm going to be happy with this application.

Next stop: moving from XFCE4 to GNOME!

I have spent most of the evening attempting to find a suitable replacement audio player. After trying and rejecting dozens of valiant (and not-so-valiant) attempts, Aqualung was the last one I tried. It's brilliant, simple, doesn't demand I use its own library format, and does exactly what I want it to.

I set it up to open audio files, it opens and plays them. I drag and drop audio files and play lists, it opens and plays them. the playlist is embedded in the main window. It is easy to read using the GTK widget toolkit, not some horrid "skin" (although horrid skins are available for those who want them). I can edit and save playlist files painlessly in the standard, portable m3u format. It has a ton of settings to tweak.

It's fast to startup. It does not scan my library while forcing me to wait. I can play files from the file system, where I have already organized them to my heart's content. It has robust command line options, making it easy to control remotely. //It has gapless playback//. It supports mp3, FLAC, ogg vorbis, AAC wav, MOD files, and wma. It has great documentation. It does software volume control by default. It can output to OSS, ALSA, JACK. It manages format conversions, volume normalization, id3 mass-retagging, and podcasts. It handles LADSPA plugins, allowing you to add effects to your playback. It has not crashed on me once.

It's available on GNU/Linux, Windows, Mac OS X, and even BSD. I highly, highly recommend it.

There's only one thing I haven't figured out: how to invoke a new instance of aqualung that will load a playlist and immediately play it. It works for individual audio files, and it works if the instance is already loaded. As it stands, I end up having to invoke it to load the playlist in the background, wait two seconds, THEN tell it to play by remote control. Annoying. A better solution is to include a short, silent wav file just before the playlist on the command line. It then plays the silent wav file, and proceeds to the playlist normally. Maybe I'll file a bug report.

I wonder how censorship-resistant Google Wave will be?

Based on my cursory studies of it's architecture, it could be very much so.

Wave is open-source. That means anyone can alter the code to behave as they wish. The protocol is federated, meaning that anyone can run their own Wave server. This is because it runs on XMPP, AKA Jabber protocol, the open instant messaging protocol that I have used every day for many years. Google Talk also uses it, participating in the global Jabber instant messaging network.

The federated architecture means that, there are two tiers of service provision - clients, that talk to their servers, and servers that talk to each other peer-to-peer. If you don't have the chops to run anything more complex than a Wave client (in your web browser, in this case), you at least have a choice of whose servers you register your account with. The Wave server, like the jabber server, stores all your contacts, logs, messages, documents - all your data. You can choose Google, or you can trust your company's IT people, or any third party who cares to install a Wave server, keep it running, and allow you to register an account. Or, if you're up to the technical challenge (like I will likely be), you can run or even modify your very own private server.

The difference between the client and the server is illustrated nicely by the address one uses uses in wave, which is identical to the format one uses in jabber, and, in a more familiar way, with email: user@domain.com. the client is identified by the user name, and the server is identified by the domain name.

The peer-to-peer server-to-server protocol is the one that allows this level of freedom and flexibility. When a "Wave" thread is created, it's only stored on the server of the user that created it. The federation protocol allows the creator to then invite other wave users to join the wave/thread to view, comment, or edit it regardless of what wave server they are subscribed to. to accomplish this, the creator's server contacts the invitee's server, and sends it the contents of the wave. The servers then collaborate to keep the contents of the wave synchronized and up to date in real time as the users edit them.

This is important to censorship resistance, it seems to me. Let's say I'm running my own private wave server at my domain, r30.net. My Wave address is nato@r30.net. Suppose bob@subpoenamagnet.com invites me to a private wave where he shares with me things man-was-not-authorized-to-know. If subpoenamagnet.com staff is over his shoulder, slaps his wrist, and deletes the wave, I already have it, and my server can't be forced to delete it if I don't want it to.

There's even more to this. These waves are stored in a wiki-like fashion, with complete revision editing histories. There's a "playback" feature that the Wave developers have demonstrated that allows you to "rewind" the editing of a wave and play it back from the beginning. This is a very handy feature, because it allows someone invited late to a hefty, sprawling wave with lots of confusing activity to go back to the beginning of the wave and see how it developed even though they were only invited into it recently. What this means is that, once I'm invited into a wave thread, not only do I receive it's current state, but it's complete history as well. Someone could well have added a naughty piece of data, and deleted it, but it would still exist in the revision history (this is why I don't understand why deleting something from a Wikipedia page qualifies as censorship). Extraordinary measures will need to be taken to actually delete information from a wave, instead of just removing it from its most recent state.

Now, add the provision for "robots". Robots are an API offered to wave clients that allow applications to view, edit and interact with a wave in the same way that users are. This allows you to set up things like filters that can watch for incoming messages, and share them with your friends based upon their content. I can imagine a decentralized syndicated news feed network, where you get pieces of data offered to you by friends also interested in the same thing. Distributed, automatic, authenticated, and private distribution networks are the ideal solution to censorship.

This is looking promising.

A few months ago, I tried out some beta packages from an Ubuntu PPA repository called Blueman. It's a more robust set of tools to deal with bluetooth devices. I was focusing on getting my PC tethered to it as a backup Internet connection the next time my broadband goes down (or I'm on the road). It works with the Ubuntu Network Manager application to setup mobile broadband fairly easily.

Of course, I despise network manager, because it's constantly doing things I don't want it to do, and offers no way to stop it. That's why I switched to Wicd. And, naturally, the two packages conflict with one another.

Nonetheless, I pulled down and archived the required packages and dependencies so that I would have them on hand in case I were to suddenly find myself offline.

Blueman, and the accompanying bluetooth stack upgrade, are pretty unstable. I had my share of crashes and sudden disappearances. But each time I did a fresh reboot, at least, I was able to dial-up my phone over bluetooth and get online. It's more like a stack of rocks than a house of cards, but once the fiddling and setup were done, I was able to connect using both my desktop and my laptop.

I just recently switched from a 500Mb add-on data plan to a far-cheaper "unlimited on-device browsing" plan from Fido. The standard, ridiculous $5/Mb transfer fees now supposedly apply, but at least I'm not paying $25/mo for a data plan I don't use. I'm curious to see what and how various Internet activities will show up on next month's bill. I better stop using it like this until I know what I'm really in for in terms of costs.

This lead me to try to find the difference between how they meter these things. There are two data profiles on the phone, "Fido WAP", and "Internet". The former was the one the on-device browser was configured to use, and includes channeling everything to a static-IP proxy server complete with username/password of 'fido'/'fido' (likely a standard practice). The 'Internet' profile had no proxy settings, however. I'm guessing they bill for any data that goes over the router, but not anything that goes through the proxy server. I don't know how easy it would be to cheat by configuring, say, my browser to use the proxy server (and even spoof the phone's browser). I'll tinker with it when I know more.

Google Wave looks promising.

A "wave" is probably best thought of as a discussion thread, with real-time elements, strong authentication (say goodbye to a lot of spam), and collaborative document-editing features.

I'm not excited so much for the new interface enhancements: real-time typing transmission, blurring the line between web comments, email, and instant messaging, and blending them all into one medium.

Not even so much for the ability to add bots as participants, which is certainly an extremely cool feature. This allows things like extensible, plug-in, real-time spellchecking, translation, polling, interfaces to flickr, twitter, facebook, syndication feeds, SMS and IM messaging, et al.

Nor even for the ability to embed waves, complete with all their included real-time posting and updating functionality, on ANY web page, blog, wiki, what-have-you, although that's positively mind-blowing. This alone will do wonders for adoption, since it overcomes the need to draw users into a walled garden by letting any website plant their own. Once you have an account on any wave server to participate in any wave on any site, you can use that same account to particpate in any other wave, anywhere on the web (or even off the web).

No, the thing I was most impressed by was the distributed/federated server architecture. That means that it's not only distributed to clients, but it's distributed on the server side, so anyone can run their own server and keep waves private so long as they only include local users. You don't have to trust Google to store, backup, or keep your wave data private, like you must to use Gmail. The initial servers are going to be open sourced (so they say), and the wave client/server and inter-server protocols will be open and standardized, so developers can implement their own wave servers and clients (be they web browsers, or server-based "robot" extensions).

The first thing I thought about this was that, "gee, it sounds a lot like Jabber's server federation architecture." Wouldn't you know: it's being built on XMPP. To be fair, this kind of client/server federation architecture bears resemblances to SMTP/POP3/IMAP used for email, and even Usenet. I'm happy to see Google letting go a bit of its dominance on the server side for the sake of confronting privacy and autonomy concerns.

I'm impressed.

(Via Le Boing)

Put me on the record now as not trusting of Microsoft, Apple, or Google.

Doug Rushkoff has a rather convoluted way of saying people are worked up over nothing when worrying about the implications of cloud computing, and of Google's new Chrome OS, just announced yesterday.

Being a long-time resistor of the idea of letting a for-profit enterprise take care of my data and software, I was a little surprised to hear this argument coming from a well-known proponent of openness. I was even a little crushed, perhaps. But then, after some analysis, it turns out he wasn't really talking about me.

While there have been "cloud computing" efforts before, they always ran up against people's (false) notions of computer privacy, virus contagion, and fear of dependence.

Rushkoff is one of those people that can actually make me reconsider and scrutinize my ideas. Were my notions of privacy, independence, and security, really "false"? I scoffed at first, but I decided to at least give Doug the benefit of the doubt, and listen to his argument.

( Unto the knot )

So here's what I think my DJing toolchain is going to look like:

jackd -R -dalsa -dhw:0,3 -r44100 -p1024 -n2 -P &
ecasound -i jack &
cle mplayer -ao jack:port=ecasound -af scaletempo -quiet -slave sometrack.mp3
cle mplayer -ao jack:port=ecasound -af scaletempo -quiet -slave someothertrack.mp3
[...]

JACKd provides the low-latency sound server and signal routing between mplayer and ECAsound. ECAsound will be standing between mplayer and the soundcard; I intend to use it to control the signal routing and mixing between the stereo channels of each mplayer "deck" (of which there can be unlimited numbers, apparently!) and the four output channels - stereo mains, and, of course, stereo headphone monitors. On my cheapo Sound Blaster Live! value, I've got a front and rear stereo surround, so I just split the "front" jack to the speakers, and use the "rear" jack for the headphones. Pie! ECAsound can apply LADSPA effects, if desired (EQs, compressor/limiters, phasers/flangers... etc.) and volume levels, of course, all of which can be controlled in real-time through any MIDI controller.

The reason I'm stuck on Mplayer is because of the scaletempo audio filter, which handles the pitch-moderation necessary to keep a track at the same pitch while changing the playback speed, avoiding the "chipmunk effect". Having it handled at the source, by a native mplayer plugin, instead of having to hack something together later in the output chain should make things much, much easier. Then I can use mplayer's slave mode with a script to control the seeking for cue points and "nudging", and the playback speed for tempo synchronization and beatmatching.

I am pretty happy about this, because I have just put this together and verified that the latency through this chain is TINY, as far as I can tell. I was tapping keys to launch cue points in time with the currently playing track with somewhat better success than I was with Mixxx last night. And I haven't even tried to install OSSv4 or even a real-time kernel. After spending a few hours frustrated by trying to connect bits with UNIX fifos, this is looking good.

I love ecasound. It helped me out a lot when I was trying to peel annoying ads off of Magnatune tracks (which they no longer tack on, thankfully), and it is one of the best-documented audio tools I've seen. Nice for an application with no GUI whatsoever - which actually makes it eminently usable to a hacker, because you can script everything. A GUI is a Graphical USER Interface, but a Command Line Interface is, in essence, an Application Programming Interface at the same time as a user interface. That API accessibility is what GUIs have lost for a long time, and are starting to reclaim with things like Applescript (which is certainly old, but confined to the Mac OS ghetto) and DBus.

I hit a brick wall today when I discovered I couldn't mix mplayer's output across two outputs (one for fronts, one for monitoring).

So instead, I went back to review some old DJ software. Mixxx is starting to look much better. It was not hanging and crashing as much as the last time I tried it, and it's under active development. Apparently they are grabbing the attention of hardware controller vendors, who are releasing open source drivers. The beta version even has a scripting environment that looks fun.

BPMDJ also released a new version recently. I guess I'm not real excited about how the interface is put together, but the tempo analysis is top-notch, and using the "beatgraph" visualization to set cue points is a nice tool I've not seen elsewhere.

So after an email exchange with the author, I was able to decipher the binary format used by its index files, and wrote a php script (meh. it was fast, and I knew it) to parse the beat period, calculate the tempo, and use eyeD3 to insert it into the mp3 file's id3 tags. Mixxx, once it loads the file, will then pick the bpm out of that tag, and synchronize track playback based on that.

The beta version of Mixxx also saves cue points in it's library. I wonder if I could translate that to the id3 Event Timing Codes tag? That way all that data is stored in the file.

I suppose I'm about to find out whether the existing tools are useful enough that I won't be reinventing any wheels after all.

Also interesting in the Linux audio front is a recent review of Linux sound drivers, which speaks very highly of OSSv4. There's a widespread impression that OSSv3 was left behind for ALSA back in the day when the the original developer decided to close the source (a more than adequate reason). But since that time, 4front has re-released OSS version 4 under the GPL, giving it an opportunity for a second look. And it's very, very good-looking indeed, with latencies well below ALSA in most cases, backward-compatibility with all ancient applications that never switched to ALSA (indeed, often the problem is that apps don't support OSS), and non-blocking behavior now the default.

I will need to check this out soon.