1. First impressions
2. Asset management
3. Under the hood: file format internals

Cost and hardware requirements

The first thing you notice about Aperture, even before you buy it, is its hefty hardware requirements. I had to upgrade the video card on my PowerMac G5 (dual 2GHz, 5.5GB RAM) to an ATI X800XT, as the stock nVidia 5200FX simply doesn’t make the cut.

Aperture costs $500, not far from the price of the full Photoshop CS2. Clearly, this product is meant for professionals, just like Final Cut Pro. The pricing is not out of line with similar programs like Capture One PRO, but it is rather steep for the advanced amateurs who have flocked to DSLRs and the RAW format. Hopefully, Apple will release a more reasonably priced “Express” version much as they did with Final Cut Express.

File management

Like its sibling iPhoto, Aperture makes the annoying assumption that it will manage your photo collection under its own file hierarchy. It does not play nice and share with other applications, apart from the built-in Photoshop integration, which merely turns Photoshop into an editor, with Aperture calling all the shots and keeping both image files and metadata databases firmly to itself. Photoshop integration does not seem to extend to XMP interoperability, for instance.

This is a major design flaw that will need to be addressed in future versions — most pros use a battery of tools in their workflow, and expect their tools to cooperate using the photographer’s directory structure, not one imposed by the tool. Assuming one Aperture to rule them all is likely going to be too confining, and the roach-motel like nature of Aperture libraries is going to cause major problems down the road. Copying huge picture files around is very inefficient — HFS supports symbolic and hard links, there is no reason to physically copy files. This scheme also renders Aperture close to useless in a networked environment like a magazine or advertising agency, where media files are typically stored on a shared SAN, e.g. on XServe RAID boxes using Fibre Channel.

Fortunately, the file layout is relatively easy to reverse-engineer and it is probably just a question of time until third-party scripts become available to synchronize an Aperture library with regular Pictures folders and XMP sidecar metadata files, or other asset management and metadata databases like Extensis Portfolio or even (shudder) Canto Cumulus. Apple should not make us jump through these hoops – the purpose of workflow is to boost productivity, not hinder it. In any case, Apple is apparently hinting Aperture can output sidecar files, at least according to PDN’s first look article

Performance

Aperture is not the lightning-fast RAW converter we have been dreaming of. Importing RAW files is quite a sluggish affair, taking 2 minutes 15 seconds to import a 665MB folder with 86 Canon Rebel XT CR2 RAW files. In comparison, Bridge takes about a minute to generate thumbnails and previews for the same images. The comparison is not entirely fair, as Aperture’s import process yields high-resolution previews that allow you to magnify the image to see actual pixels with the loupe tool, whereas Bridge’s previews are medium resolution at best. The CPU utilization on my dual G5 is far from pegged however, which suggests the import process was not particularly tuned for SMP or multi-core systems, nor that it even leverages OS X’s multithreading. Aperture will work with other formats like scanned TIFFs as well (import times are even slower, though).

Once import is complete, viewing the files is very smooth and fast. The built-in loupe tool is particularly addictive, and very natural for anyone who has worked with a real loupe on a real light table. A cute visual effect (Quicktime, 6MB) has the loupe flip over as you reach the edges of the screen. The loupe will also magnify the thumbnails, although that will pause execution for the time it takes to read the thumbnail’s preview data into memory.

Workflow innovations

Aperture has two very interesting concept: stacks and versions. Stacks group together multiple images as one. It is very common for a photographer to take several very similar photos. Think of bracketed exposures, or a sports photographer shooting a fast action sequence at 8 frames per second, or a VR photographer making a series of 360-degree shots for use in an immersive panorama. Aperture’s stacks allow you to manage these related images as a single unit, the stack. It is even capable of identifying candidates for a stack automatically using timestamps.

Stacks

This article is work-in-progress, and this section has to be fleshed out

Versions

Versions is a concept clearly drawn from the world of software configuration control systems like CVS or Visual SourceSafe. Aperture does not touch the original image, adjustments like changing the color balance simply record the series of operations to achieve the new version of the image in the metadata database, just like CVS only stores diffs between versions of a file, to save space. This suggests Apple plans future versions of Aperture with shared image repositories, as most modern systems work that way, with a shared central repository, and individual copies for each user, with a check-in/check-out mechanism with conflict resolution.

The parameters for a transform take a trifling amount of memory, and the photographer can experiment to his heart’s content with multiple variants. Photoshop now has equivalent functionality with the introduction of layers comps in CS, but they still feel like bolted-on features rather than integral to the product.

In the early nineties, French firm FITS introduced a groundbreaking program named Live Picture to compete with Photoshop. Its chief claim to fame was that it could deal with huge images very efficiently, because it recorded the operations as a sequence of mathematical transforms rather than the result, in a way eerily reminiscent of PostScript. The transforms were only applied as needed at the display resolution, thus avoiding the need to apply them to the full-resolution image until the final output was required, while still managing to deal with zooming adequately. The software was promising, but the transformation logging technique limited the types of operations that could be performed on images, and due in part to its high price and specialized scope, the product died a slow and painful death. In its current incarnation, it lives on, after a fashion, in the moribund Flashpix image format and a web graphics server program imaginatively named ImageServer.

Chained transforms is a very elegant approach when compared to the brute-force of Photoshop — in Aperture a finished image is represented as a master image and a series of transformations to be applied to it to achieve each version, much like Photoshop keeps a history of changes made to the image in memory (but not in the final disk file). Since Aperture’s transforms are fairly simple, they can be executed in real time by modern graphics cards that support Core Image.

Target market

Keep in mind there are two types of pro photographers: those who divide photographers in two groups, and the others… More seriously:

1. Those who try to build up a portfolio of images over their career, where royalties and residual rights will provide them with financial support when they retire. Most fine art, landscape or nature photographers are in this category, and photojournalists could be assimilated (except their employer owns the rights to the archive in the latter case).
2. Those who do work-for-hire. Wedding photographers, event photographers, product, catalog, advertising and industrial photographers fit in this category.

The first type need a digital asset management database to retrieve and market their images more effectively, and a distribution channel. Most farm out that representation work to the likes of Corbis or Getty Images.

The second type will work intensely on a project, take a lot of frames and show many variants to a client for approval. Once the project is done, it is archived, probably never to be used again, and they move on to the next one. In most cases, the rights to the images remain with those who commissioned them, not the photographer, and thus there is no incentive to spend much effort in organizing them with extensive metadata beyond what is required for the project. These users need a production workflow tool that will streamline the editing and client approval process, the latter mostly performed via email or password-protected websites nowadays.

Aperture’s projects (flat, not nested hierarchically, and thus not all that scalable) and vaults (archives where projects go when they are no longer needed or finished) are a clear indication it is intended mostly for the second type of photographer. Apple did not convey the specialized focus of the product forcefully enough, but the blogosphere’s buzz machine bears much of the blame for raising unwarranted expectations about what the product is about.

Aperture is good for one thing: let wedding photographers and the like go through the editing (as in sorting through slides on a light table, not retouching an individual image) as efficiently as possible. Most wedding pros simply cannot afford the time to individually edit a single picture beyond white balance, tonal adjustments, cropping and sharpening, and Aperture’s built-in tools are perfectly adequate for them.

That is also why there is such a slow import process — this prep work is required to make the actual viewing, side by side comparison, sorting and manipulation as smooth, interactive and responsive as possible to avoid disrupting the photographer’s “flow”. The goal is to have a very smooth virtual light table, not a filing cabinet, where you can move slides around, group them in stacks, toy around with versions, and compare them side by side on dual 30 inch Cinema Displays. The user experience was clearly designed around what the average art director (with his or her loupe almost surgically implanted) is familiar and comfortable with.

The positioning as “iPhoto Pro” obscures the fact that Aperture is sub-optimal for managing or indexing large archives with rich metadata. For the first type of pro photographer (which is also what the average advanced amateur will relate to), a digital asset management database like the excellent Kavasoft Shoebox or the more pedestrian but cross-platform and workgroup-oriented Extensis Portfolio and Canto Cumulus will better fit their needs, with Adobe Bridge being probably sufficient for browsing and reviewing/editing freshly imported photos (when it is not keeling over and crashing, that is).

Conclusion

Aperture is clearly a 1.0 product. It shows promise, but is likely to disappoint as the reality cannot match the hype that developed in the month or so between the announcement and the release. The problem is that there is a Cult of the Mac that raises unrealistic expectations of anything coming out of Cupertino.

The hype around Aperture was certainly immense, I am sure Apple was as surprised as any by how positive the response was (after all, they released Aperture at a relatively obscure pro photo show). They are probably furiously revising plans for the next release right now. I consider Aperture 1.0 more a statement of direction than a finished product.
]]> http://majid.info/blog/aperture-first-impressions/feed/ 0 http://majid.info/blog/aperture-internals/ http://majid.info/blog/aperture-internals/#comments Thu, 01 Dec 2005 08:15:46 +0000 majid http://www.majid.info/mylos/stories/2005/12/01/apertureInternals.html Continue reading →]]> Last in a series:

1. First impressions
2. Asset management
3. Under the hood: file format internals

This article was never completed because I switched to Lightroom and lost interest. What was done may be of interest to Aperture users, although the data model probably changed since 1.0

Aperture stores its library as a bundle with the extension .aplibrary. This is a concept inherited from NeXTstep, where an entire directory that has the bundle bit set is handled as if it were a single file. A much more elegant system than Mac OS Classic’s data and resource forks.

Inside the bundle, there is a directory Aperture.aplib which contains the metadata for the library in a file Library.apdb. This file is actually a SQLite3 database. SQLite is an excellent, lightweight open-source embedded relational database engine. Sun uses SQLite 2 as the central repository for SMF, the next-generation service management facility that controls booting the Solaris operating system and its automatic fault recovery, a strong vote of confidence by Sun in SQLite’s suitability for mission-critical use. SQLite is also one of the underlying data storage mechanisms used by Apple’s over-engineered Core Data framework.

You don’t have to use Core Data to go through the database, the /usr/bin/sqlite3 command-line utility is perfectly fine for this purpose. Warning: using sqlite3 to access Aperture’s data directly is obviously unsupported by Apple, and should not be done on a mission-critical library. At the very least, make sure Aperture is not running.

ormag ~/Pictures/Aperture Library.aplibrary/Aperture.aplib>sqlite3 Library.apdb
SQLite version 3.1.3
Enter ".help" for instructions
sqlite> .tables
ZRKARCHIVE             ZRKIMAGEADJUSTMENT     ZRKVERSION
ZRKARCHIVERECORD       ZRKMASTER              Z_10VERSIONS
ZRKARCHIVEVOLUME       ZRKPERSISTENTALBUM     Z_METADATA
ZRKFILE                ZRKPROPERTYIDENTIFIER  Z_PRIMARYKEY
ZRKFOLDER              ZRKSEARCHABLEPROPERTY
sqlite> .schema z_metadata
ormag ~/Pictures/Aperture Library.aplibrary/Aperture.aplib>sqlite3 Library.apdb
SQLite version 3.3.7
Enter ".help" for instructions
sqlite> .tables
ZRKARCHIVE             ZRKKEYWORD             ZRKVOLUME
ZRKARCHIVERECORD       ZRKMASTER              Z_11VERSIONS
ZRKARCHIVEVOLUME       ZRKPERSISTENTALBUM     Z_9VERSIONS
ZRKFILE                ZRKPROPERTYIDENTIFIER  Z_METADATA
ZRKFOLDER              ZRKSEARCHABLEPROPERTY  Z_PRIMARYKEY
ZRKIMAGEADJUSTMENT     ZRKVERSION
sqlite> .schema zrkfile
CREATE TABLE ZRKFILE ( Z_ENT INTEGER, Z_PK INTEGER PRIMARY KEY, Z_OPT INTEGER, ZASSHOTNEUTRALY FLOAT, ZFILECREATIONDATE TIMESTAMP, ZIMAGEPATH VARCHAR, ZFILESIZE INTEGER, ZUUID VARCHAR, ZPERMISSIONS INTEGER, ZNAME VARCHAR, ZFILEISREFERENCE INTEGER, ZTYPE VARCHAR, ZFILEMODIFICATIONDATE TIMESTAMP, ZASSHOTNEUTRALX FLOAT, ZFILEALIASDATA BLOB, ZSUBTYPE VARCHAR, ZCHECKSUM VARCHAR, ZPROJECTUUIDCHANGEDATE TIMESTAMP, ZCREATEDATE TIMESTAMP, ZISFILEPROXY INTEGER, ZDATELASTSAVEDINDATABASE TIMESTAMP, ZISMISSING INTEGER, ZVERSIONNAME VARCHAR, ZISTRULYRAW INTEGER, ZPROJECTUUID VARCHAR, ZEXTENSION VARCHAR, ZISORIGINALFILE INTEGER, ZISEXTERNALLYEDITABLE INTEGER, ZFILEVOLUME INTEGER, ZMASTER INTEGER );
CREATE INDEX ZRKFILE_ZCHECKSUM_INDEX ON ZRKFILE (ZCHECKSUM);
CREATE INDEX ZRKFILE_ZCREATEDATE_INDEX ON ZRKFILE (ZCREATEDATE);
CREATE INDEX ZRKFILE_ZFILECREATIONDATE_INDEX ON ZRKFILE (ZFILECREATIONDATE);
CREATE INDEX ZRKFILE_ZFILEMODIFICATIONDATE_INDEX ON ZRKFILE (ZFILEMODIFICATIONDATE);
CREATE INDEX ZRKFILE_ZFILESIZE_INDEX ON ZRKFILE (ZFILESIZE);
CREATE INDEX ZRKFILE_ZFILEVOLUME_INDEX ON ZRKFILE (ZFILEVOLUME);
CREATE INDEX ZRKFILE_ZISEXTERNALLYEDITABLE_INDEX ON ZRKFILE (ZISEXTERNALLYEDITABLE);
CREATE INDEX ZRKFILE_ZMASTER_INDEX ON ZRKFILE (ZMASTER);
CREATE INDEX ZRKFILE_ZNAME_INDEX ON ZRKFILE (ZNAME);
CREATE INDEX ZRKFILE_ZPROJECTUUIDCHANGEDATE_INDEX ON ZRKFILE (ZPROJECTUUIDCHANGEDATE);
CREATE INDEX ZRKFILE_ZUUID_INDEX ON ZRKFILE (ZUUID);
sqlite> .schema z_metadata
CREATE TABLE Z_METADATA (Z_VERSION INTEGER PRIMARY KEY, Z_UUID VARCHAR(255), Z_PLIST BLOB);
sqlite> .schema z_primarykey
CREATE TABLE Z_PRIMARYKEY (Z_ENT INTEGER PRIMARY KEY, Z_NAME VARCHAR, Z_SUPER INTEGER, Z_MAX INTEGER);
sqlite> select * from z_primarykey;
1|RKArchive|0|1
2|RKArchiveRecord|0|0
3|RKArchiveVolume|0|1
4|RKFile|0|2604
5|RKFolder|0|23
6|RKProject|5|0
7|RKProjectSubfolder|5|0
8|RKImageAdjustment|0|1086
9|RKKeyword|0|758
10|RKMaster|0|2604
11|RKPersistentAlbum|0|99
12|RKPropertyIdentifier|0|119
13|RKSearchableProperty|0|84191
14|RKVersion|0|2606
15|RKVolume|0|0
sqlite>

One useful command is .dump, which will dump the entire database in the form of the SQL commands required to recreate it. Even with a single-photo library, this generates many pages of output.

Here is my attempt to reverse engineer the Aperture 1.5.2 data model. CoreData, like all object-relational mappers (ORMs) leaves much to be desired from the relational perspective. The fact SQLite foreign keys constraints are not enforced (and not even set by CoreData) doesn’t help. Click on the diagram below to expand it.

All tables are linked to Z_PRIMARYKEY which implements a form of inheritance using the column Z_ENT to identify classes. The only table that seems to use this today is ZRKFOLDER, where the rows can have a Z_ENT of 5 (Folder), 6 (Project) or 7 (ProjectSubfolder). For clarity, I have omitted the links between all tables and Z_PRIMARYKEY.

ZRKIMAGEADJUSTMENT looks like the table that records the transformations that turn a master image into a version, Live Image style.

My father took most of my childhood photos (like these) on Kodachrome slide film. Kodachrome was the only color game in town for a long time, but was eventually superseded in the marketplace by C41 color print films and finer grained E6 slide film.

Kodachrome has a distinctive sharpness (acutance, not resolution), and excellent durability when stored in the dark. Many photographers still shoot Kodachrome for its special “look”, even though processing options are diminishing and Kodak jacked up the price. Kodak recently announced it is closing its Fair Lawn, NJ processing lab, the last Kodak-owned plant in the US for Kodachrome, and there are now only three labs left worldwide that can run the complicated process (Dwayne’s in Kansas, Kodak in Lausanne, Switzerland, and a lab in Japan). Kodachrome was actually discontinued for a while, and brought back after strident protests, but the writing is on the wall.

Projectors and light tables

Every now and then, we would dust off the slide projector and have a slide show. I even remember building a surprisingly effective slide projector when I was 9 using Legos, a flashlight and a jar of peanut butter filled with water as the lens. Slide projectors are hard to find, a pain to setup and most people groan instinctively when one comes up, associated as they are with dreary slide show of other people’s vacation pictures. The LCD computer monitor is the successor to the projector, and many people no longer have prints made at all, perhaps because they subconsciously realize that the 500:1 contrast ratio of a LCD monitor yields significantly livelier images than prints can achieve.

For those of us who still shoot slide film, there is a better alternative, light tables and loupes. A light table is just what the name implies – a piece of frosted plastic illuminated by daylight-balanced fluorescent tubes. Basic models like my Porta-trace shown above are inexpensive. Loupes, on the other hand, are a different story.

Loupe basics

Loupes (French for “magnifying glass”) are high-quality magnifiers, originally used to help focus images on a ground glass, and later to view slides or negatives on a light table. You can find them in all shapes and sizes, at prices from $15 for a cheap plastic model, all the way to over $300 for a Zeiss loupe for viewing 6×6 medium format slides. Slides viewed on a light table with a high-quality loupe are a treat for the eyes, because of the high contrast (1000:1) that you cannot get with prints (more like 100:1).

There are two ways you can use a loupe: use a high-power (10x or higher) to check slides or negatives for critical focus), or a medium-power loupe to evaluate an entire frame (usually 5x-6x for 35mm, 3x-3.5x for medium format). Viewing an entire frame is more challenging than just checking for focus in the center, because the loupe must provide excellent optical quality across the entire field of view. There are variable magnification (zoom) loupes available, but their optical quality is far below that of fixed magnification loupes, and they should be avoided for critical examination of slides or negatives.

I have accumulated quite a few loupes over time. The most famous brand in loupes is Schneider-Kreuznach, a German company noted for its enlarger, medium format and large format lenses. Many other brands make high-quality loupes, including Rodenstock, Pentax, Nikon, Canon, NPC, Leica and Zeiss. I do not live in New York, and have thus not had the opportunity to compare them side by side at a place like B&H Photo, so I pretty much had to take a leap of faith based on recommendations on the Internet at sites like Photo.net.

• Peak 10x Plalupe
• Contax Carl Zeiss Triotar T* 5x (larger photo)
• Edmund Optics 50x direct view measuring microscope
• Calumet-branded Rodenstock 3x Aspherical loupe for 6×6
• Edmund Optics 10x Hastings Triplet
• Schneider 10x magnifier

Peak

The Peak was my first loupe. Dirt cheap, and reasonably good for the price, but that’s pretty much all it has going for it (more on that below).

Zeiss

I was put off by reports on the plastic construction of the new line of Schneider loupes, and opted for a Zeiss loupe instead, based on the reputation of Zeiss lenses (my first camera was a Zeiss, and I also have a Zeiss lens on my Hasselblad).

The Zeiss Triotar 5x loupe (the box does not mention Contax, but as it is made in Japan, it is presumably made in the same factory) comes in a cardboard box that can be turned into a protective case by cutting off the tabs on both ends. It does not include a carrying pouch or protective box, which is regrettable, specially for a product as expensive ($160), but apparently most high-end loupe manufacturers do not bother to include one. It does not include a neck strap either, which could be more of an issue for some. How can you look like a glamorous New York art director without a loupe around your neck? More seriously, the strap is particularly useful if you are going to use the loupe for focusing medium or large format cameras against a ground screen.

The loupe is shipped with two acrylic bases that screw into the loupe’s base. One is frosted, and is used as a magnifier to view prints or other objects, with ambient light filtering through the base to illuminate the object. The black base is used to shield out extraneous light when concentrating on a slide or negative on a light table or a ground glass. Some loupes have a design with a clear base and a removable metal light shield. Which design you prefer is mostly a matter of personal taste. The loupe has a pleasant heft to it, and impeccable build quality. The main body of the loupe itself is solidly built of black anodized metal, with a knurled rubber focusing ring.

The optical quality is what you would expect form Zeiss. Crystal clear, sharp across the field of view, and no trace of chromatic aberration in the corners. You can easily view an entire 35mm frame and then some, although I suspect eyeglass wearers might find the eye relief a little bit short.

Edmunds pocket microscope

The Edmunds direct view microscope is a versatile instrument, available in many magnifications, with or without an acrylic base (highly recommended) and with or without a measurement reticle (metric or imperial). Due to the high magnification, the image has a very narrow field of view (only 3mm), and is quite dim. Unlike the others, the image is reversed, which requires some adaptation time. The level of detail you can observe on slides taken with a good film like Fuji Provia 100F, using a good lens and a tripod, is absolutely stunning. This is a rather specialized instrument, but well worth having in your toolkit.

Rodenstock

The Rodenstock 3x 6×6 aspheric loupe has a list price of $350 and usually retails for $250. Calumet Photo sells the exact same loupe under their own brand for a mere $149 (I actually got mine for $109 during a promotion), which is not that much more than a cheap (in more ways than price) Russian-made Horizon.

There are naturally fewer loupes available to view medium format slides or negatives than for 35mm. Schneider, Mamiya/Cabin, Contax/Zeiss and Rodenstock make high-grade loupes for this demanding market. If you have a “chimney” viewfinder on your MF camera, you can actually use that as a loupe.

Rodenstock is famous for its large-format and enlarging lenses, and this loupe is very highly rated. The construction is plastic, but still well-balanced and not too top-heavy. It does not carry the feel of opulence that the Zeiss has, or even the very nicely designed Mamiya/Cabin loupes (more on that below), but is still clearly a professional instrument. It has a two-element aspherical design for sharpness across the entire field of view, and coated optics. It comes with a red neck cord, and the base has a removable plastic skirt that slides in place and can be reversed between its clear and dark positions. The eyepiece has a rubber eyecup and a knurled rubber grip for the focusing ring.

I compared it side by side at Calumet San Francisco with the Cabin 3.5x loupe for 6×4.5 or 6×6. The Cabin had a solid metal constuction (somewhat top-heavy), but its screw-in skirts are less convenient than the slide-in one used in the Rodenstock, and the image circle is too tight for my Hasselblad 6×6 slides. I think that loupe was really designed for 645 format and opportunistically marketed for 6×6, when the 6×7 loupe would actually be more appropriate for that usage. The optical quality is very similar and both are excellent loupes. I did not try the Mamiya/Cabin 6×7, unfortunately, as it was not available in the store, but in any case the Rodenstock was a steal.

The optics are excellent, as could be expected, with crisp resolution all the way into the corners and no trace of chromatic aberrations. There is a smidgen of pincushion distortion, however, but not enough to be objectionable (I took the slightly convex curved square skirt out to make sure this was not just an optical illusion).

One thing to watch out for: even though the optics are coated, they are very wide and you have to be careful to keep your eye flush with the eyecup to obscure any overhead light sources like lightbulbs or fluorescent panels and avoid seeing their reflections in the loupe’s glass.

The most comprehensive resource for medium format loupes on the Web is Robert Monaghan’s page on the subject.

Edmunds Hastings triplet

This isn’t really a competitor to the other loupes, as it has a very narrow field of view of only 10mm in diameter. It is also tiny, and I carry mine in my gadget bag. It has a folding jeweler’s loupe design with a folding metallic shield to protect it. Optical quality is of the highest order.

Schneider

Despite its plastic construction, this loupe exudes quality. Unfortunately, the strap is really flimsy – the rubber cord is merely glued into the metal clip, and will easily pull out. I glued mine back, and crimped it with needle-nose pliers for good measure, but I don’t know how robust this arrangement will be.

The optics are excellent, without any trace of chromatic aberration. The usable field of view is surprisingly wide for a loupe with this magnifying power, although your eye has to be perfectly positioned to see it. I estimate the FOV diameter at 20mm, as you can almost see the full height of a 35mm mounted slide. I have an Edmund Optics magnifier resolution chart (it came with the Hastings triplet), and the Schneider outresolves it across the field of view . This means the Schneider exceeds 114 line pairs per millimeter across the frame, quite remarkable performance.

The importance of a good loupe

For a real-world test, I took my 6×17 format Velvia 100F slides of the Golden Gate Bridge, and looked at the suspension cables. The picture to the left shows the details I was looking at (but the fuzzy 1200dpi scan on an Epson 3170 does not remotely do justice to the original). Each bundle of 4 cables (4 line pairs) takes 0.04mm on the slide (I used the 50x Edmunds inspection microscope to measure this), hence you need 100lp/mm to resolve it. The Schneider 10x, Edmunds 10x and Zeiss 5x loupes all resolve the four cables clearly. My old el cheapo Peak 10x loupe did not, nor the Epson scanner, which led me until recently to believe my slides were slightly blurry because I had forgotten my cable release that day. So much for the theory you do not need an expensive 10x loupe to assess critical focus because only the center counts…

Rationale

I was encountering deadlocks in a multi-threaded CORBA server (implemented using omniORB). Debugging using GDB gave me too low-level information, and what I needed was an equivalent of the GDB command “info threads”. There was no such facility available from within Python’s standard library, so I rolled my own.

David Beazley added advanced debugging functions to the Python interpreter, and they have been folded into the 2.2 release.

I used these hooks to build a debugging module that is useful when you are looking for deadlocks in a multithreaded application. It basically has a single function that will return a list of the stack frames for all Python interpreter threads in the process.

Guido van Rossum added in Python 2.3 the thread ID to the interpreter state structure, and this allows us to produce a dictionary mapping thread IDs to frames.

This functionality is now integrated in Python 2.5′s batteries-included sys._current_frames() function.

Of course, I disclaim any liability if this code should crash your system, erase your homework, eat your dog (who also ate your homework) or otherwise have any undesirable effect.

Building and installing

Python 2.2 or later is required. Thread ID to frame dictionary extraction is only available in Python 2.3 and later, and will generate a NotImplementedError if used from 2.2.

Download the source tarball threadframe-0.2.tar.gz. You can use the Makefile or directly with the setup.py script. I have built and tested this only on Solaris 8/x86 and Windows 2000, but the code should be pretty portable. There is a small test program test.py that illustrates how to use this module to dump stack frames of all the Python interpreter threads. A sample run is available for your perusal.

For Windows users, I have available pre-compiled binaries, built using Mingw32 and GCC 2.95.2. Just copy the file threadframe.pyd in any location in your Python path and you should be able to run the test script test.py.

License

This code is licensed under the same terms as Python itself.

Change history

Release 0.2 (2004-06-10)

Distutils based setup.py contributed by Bob Ippolito. Bob also noticed that thread_id was added to the Python interpreter state, and contributed a patch to get a dictionary mapping thread_ids to frames instead of a list.

Release 0.1 (2002-10-11)

Initial release for Python 2.2: threadframe-0.1.tar.gz

• Introduction
• Features
• History
• Screen shots
• Known bugs
• Credits
• Download
• Updates
• Post scriptum

Introduction

Temboz is a RSS aggregator. It is inspired by FeedOnFeeds (web-based personal aggregator), Google News (two column layout) and TiVo (thumbs up and down). I have been using FeedOnFeeds for some time now, but that software seems to have stopped evolving, and I had a number of optimizations to the user experience I wanted to make.

Features

Already implemented:

• Multithreaded, download feeds in parallel.
• Built-in web server.
• Two-column user interface for better readability and information density. Automatic reflow using CSS.
• Ratings system for articles
• Real-time hunter-gatherer user interface: items flagged with a “Thumbs down” disappear immediately off the screen (using Dynamic HTML), making room for new articles. No laborious flagging of items as in FeedOnFeeds.
• Filtering entries (using Python syntax, e.g. 'Salon' in feed_title and title == "King Kaufman's Sports Daily", or simply by selecting keywords/phrases and hitting “Thumbs down”).
• Ability to generate a RSS feeds from “Thumbs Up” articles, which is why Temboz would be a true aggregator, not just a reader.
• Ad filtering
• Automatic garbage collection: every day between 3AM and 4AM, uninteresting articles (by default those older than 7 days) are purged of their contents (but not metadata such as titles, permalinks or timestamps) to keep the database size manageable. After 6 months (by default), they are deleted altogether
• Automatic database backups daily (immediately after garbage collection)

On the to do list:

• Write better documentation
• Handle permanent HTTP redirects for feed XML URLs
• Automatic pacing of feed polling intervals using the average and standard deviation of observed feed item inter-arrival times, to reduce bandwidth usage and load for both client and server. Most feeds should be polled on a daily rather than hourly interval (e.g. my own, since I update once a week on average), but the mechanisms for a feed to indicate its polling rate preferences are quite inconsistent from one flavor of RSS/Atom to another.
• “Survivor mode” – vote feeds that no longer perform off the aggregator based on relevance statistics.
• Ability to cluster together articles (I tried a heuristic of looking for common URLs they are all pointing to, but this didn’t work well in practice).
• Portability to Windows, distribution as a standalone package.

History

I have been using it successfully for well over a year. It still has rough edges, with some administration functions only doable using the SQLite command-line utility. Here is a screen shot showing the reader user interface. The article highlighted in yellow was given a “Thumbs Up”. You can also see the user interface at work in a view of the last 50 articles I flagged as “thumbs up” among the feeds I read.

Screen shots

Click on a screen shot thumbnail for a full-sized version

The first screen shot shows the article reading interface, using a two-column layout. Clicking on the “Thumbs down” icon makes the article disappear, bringing a new one in its place (if available). Clicking on the “Thumbs up” icon highlights it in yello and flags it as interesting in the database.

The feed summary page shows statistics on feeds, starting with feeds with unread articles, then by alphabetical order. Feeds can be sorted based on other metrics. You have the option of “catching up” with a feed (marking all the articles as read). Feeds with errors are highlighted in red (not shown).

Clicking on the “details” link for a feed brings this page, which allows you to change title or feed URL, and shows the RSS or Atom fields accessible for filtering.

Feeds can be filtered using Python expressions.

Known bugs

You can check outstanding bug reports, change requests and more at the public CVStrac site.

Credits

Temboz is written in Python, and leverages Mark Pilgrim’s Ultra-liberal feed parser, SQLite 2.x, Cheetah.

Download

You can download the current version: temboz-0.8.tar.gz I welcome any feedback you may have, specially as concerns improving installation.

The CVS version is far ahead of 0.8 in features. I have not yet had the time to test and document the migration procedure from 0.8 to 1.0, but if you are a new Temboz user I strongly advise you to get a nightly CVS snapshot instead (they are what I run on my own server): temboz-CVS.tar.gz or temboz-CVS.zip.

Updates

For news on Temboz, please subscribe to the RSS feed.

Temboz has a CVStrac where you can submit bug reports or change requests, and a Wiki, where all future documentation will ultimately reside.

Post scriptum

The name “Temboz” is a reference to Malima Temboz, “The mountain that walks”, an elephant whose tormented spirit is the object of Mike Resnick’s excellent SF novel, Ivory.

This article remains popular thanks to Google and the like, but it was written 7 years ago and the models described are ancient history. The general principles remain, you are often better off with a camera that has fewer but better quality pixels, though the sweet spot shifts with each successive generation. The more reputable camera makers have started to step back from the counterproductive megapixel race, and the buying public is starting to wise up, but this article remains largely valid.

My current recommendations are:

• Dispense with entry-level point and shoot cameras. They are barely better than your cameraphone
• If you must have a pocketable camera with a zoom lens, get the Canon S95, Panasonic LX5, Samsung TL500 or Olympus XZ-1. Be prepared to pay about $400 for the privilege.
• Upping the budget to about $650 and accepting non-zoom lenses gives you significantly better optical and image quality, in cameras that are still pocketable like the Panasonic GF2, Olympus E-PL2, Samsung NX100, Ricoh GXR and Sony NEX-5.
• The Sigma DP1x and DP2x offer stunning optics and image quality in a compact package, but are excruciatingly slow to autofocus. If you can deal with that, they are very rewarding.
• The fixed-lens Fuji X100 (pretty much impossible to get for love or money these days, no thanks to the Sendai earthquake) and Leica X1 offer superlative optics, image and build quality in a still pocketable format. The X1 is my everyday-carry camera, and I have a X100 on order.
• If size and weight is not an issue, DSLRs are the way to go in terms of flexibility and image quality, and are available for every budget. Models I recommend by increasing price range are the Pentax K-x, Canon Rebel T3i, Nikon D7000, Canon 5DmkII, Nikon D700 and Nikon D3S.
• A special mention for the Leica M9. It is priced out of most people’s reach, and has poor low-light performance, but delivers outstanding results thanks to Leica lenses and its sensor devoid of anti-alias filters.

Introduction

As my family’s resident photo geek, I often get asked what camera to buy, specially now that most people are upgrading to digital. Almost invariably, the first question is “how many megapixels should I get?”. Unfortunately, it is not as simple as that, megapixels have become the photo industry’s equivalent of the personal computer industry’s megahertz myth, and in some cases this leads to counterproductive design decisions.

A digital photo is the output of a complex chain involving the lens, various filters and microlenses in front of the sensor, and the electronics and software that post-process the signals from the sensor to produce the image. The image quality is only as good as the weakest link in the chain. High quality lenses are expensive to manufacture, for instance, and often manufacturers skimp on them.

The problem with megapixels as a measure of camera performance is that not all pixels are born equal. No amount of pixels will compensate for a fuzzy lens, but even with a perfect lens, there are two factors that make the difference: noise and interpolation.

Noise

All electronic sensors introduce some measure of electronic noise, among others due to the random thermal motion of electrons. This shows itself as little colored flecks that give a grainy appearance to images (although the effect is quite different from film grain). The less noise, the better, obviously, and there are only so many ways to improve the signal to noise ratio:

• Reduce noise by improving the process technology. Improvements in this area occur slowly, typically each process generation takes 12 to 18 months to appear.
• Increase the signal by increasing the amount of light that strikes each sensor photosite. This can be done by using faster lenses or larger sensors with larger photosites. Or by only shooting photos in broad daylight where there are plenty of photons to go around.

Fast lenses are expensive to manufacture, specially fast zoom lenses (a Canon or Nikon 28-70mm f/2.8 zoom lens costs over $1000). Large sensors are more expensive to manufacture than small ones because you can fit fewer on a wafer of silicon, and as the likelihood of one being ruined by an errant grain of dust is higher, large sensors have lower yields. A sensor twice the die area might cost four times as much. A “full-frame” 36mm x 24mm sensor (the same size as 35mm film) stresses the limits of current technology (it has nearly 8 times the die size of the latest-generation “Prescott” Intel Pentium IV), which is why the full-frame Canon EOS 1Ds costs $8,000, and professional medium-format digital backs can easily reach $25,000 and higher.

This page illustrates the difference in size of the sensors on various consumer digital cameras compared to those on some high-end digital SLRs. Most compact digital cameras have tiny 1/1.8″ or 2/3″ sensors at best (these numbers are a legacy of TV camera tube ratings and do not have a relationship with sensor dimensions, see DPReview’s glossary entry on sensor sizes for an explanation).

For any given generation of cameras, the conclusion is clear – bigger pixels are better, they yield sharper, smoother images with more latitude for creative manipulation of depth of field. This is not true across generations, however, Canon’s EOS-10D has twice as many pixels as the two generations older EOS-D30 for a sensor of the same size, but it still manages to have lower noise thanks to improvements in Canon’s CMOS process.

The problem is, as most consumers fixate on megapixels, many camera manufacturers are deliberately cramming too many pixels in too little silicon real estate just to have megapixel ratings that look good on paper. Sony has introduced a 8 megapixel camera, the DSC-F828, that has a tiny 2/3″ sensor. The resulting photosites are 1/8 the size of those on the similarly priced 6 megapixel Canon Digital Rebel (EOS-D300), and 1/10 the size of those on the more expensive 8 megapixel DSLR Canon EOS-1D Mark II.

Predictably, the noise levels of the 828 are abysmal in anything but bright sunlight, just as a “150 Watts” ghetto blaster is incapable of reproducing the fine nuances of classical music. The lens also has its issues, for more details see the review. The Digital Rebel will yield far superior images in most circumstances, but naive purchasers could easily be swayed by the 2 extra megapixels into buying the inferior yet overpriced Sony product. Unfortunately, there is a Gresham’s law at work and manufacturers are racing to the bottom: Nikon and Canon have also introduced 8 megapixel cameras with tiny sensors pushed too far. You will notice that for some reason camera makers seldom show sample images taken in low available light…

Interpolation

Interpolation (along with its cousin, “digital zoom”) is the other way unscrupulous marketers lie about their cameras’ real performance. Fuji is the most egregious example with its “SuperCCD” sensor, that is arranged in diagonal lines of octagons rather than horizontal rows of rectangles. Fuji apparently feel this somehow gives them the right to double the pixel rating (i.e. a sensor with 6 million individual photosites is marketed as yielding 12 megapixel images). You can’t get something for nothing, this is done by guessing the values for the missing pixels using a mathematical technique named interpolation. This makes the the image look larger, but does not add any real detail. You are just wasting disk space storing redundant information. My first digital camera was from Fuji, but I refuse to have anything to do with their current line due to shenanigans like these.

Most cameras use so-called Bayer interpolation, where each sensor pixel has a red, green or blue filter in front of it (the exact proportions are actually 25%, 50% and 25% as the human eye is more sensitive to green). An interpolation algorithm reconstructs the three color values from adjoining pixels, thus invariably leading to a loss of sharpness and sometimes to color artifacts like moiré patterns. Thus, a “6 megapixel sensor” has in reality only 1.5-2 million true color pixels.

A company called Foveon makes a distinctive sensor that has three photosites stacked vertically in the same location, yielding more accurate colors and sharper images. Foveon originally took the high road and called their sensor with 3×3 million photosites a 3MP sensor, but unfortunately they were forced to align themselves with the misleading megapixel ratings used by Bayer sensors.

Zooms

A final factor to consider is the zoom range on the camera. Many midrange cameras come with a 10x zoom, which seems mighty attractive in terms of versatility, until you pause to consider the compromises inherent in a superzoom design. The wider the zoom range, the more aberrations and distortion there will be that degrade image quality, such as chromatic aberration (a.k.a. purple fringing), barrel or pincushion distortion, and generally lower resolution and sharpness, specially in the corners of the frame.

In addition, most superzooms have smaller apertures (two exceptions being the remarkable constant f/2.8 aperture 12x Leica zoom on the Panasonic DMC-FZ10 and the 28-200mm equivalent f/2.0-f/2.8 Carl Zeiss zoom on the Sony DSC-F828), which means less light hitting the sensor, and a lower signal to noise ratio.

A reader was asking me about the Canon G2 and the Minolta A1. The G2 is 2 years older than the A1, and has 4 million 9 square micron pixels, as opposed to 5 million 11 square micron sensors, and should thus yield lower image quality, but the G2′s 3x zoom lens is fully one stop faster than the A1′s 7x zoom (i.e. it lets twice as much light in), and that more than compensates for the smaller pixels and older sensor generation.

Recommendations

If there is a lesson in all this, it’s that unscrupulous marketers will always find a way to twist any simple metric of performance in misleading and sometimes even counterproductive ways.

My recommendation? As of this writing, get either:

• An inexpensive (under $400, everything is relative) small sensor camera rated at 2 or 3 megapixels (any more will just increase noise levels to yield extra resolution that cannot in any case be exploited by the cheap lenses usually found on such cameras). Preferably, get one with a 2/3″ sensor (although it is becoming harder to find 3 megapixel cameras nowadays, most will be leftover stock using an older, noisier sensor manufacturing process).
• Or save up for the $1000 or so that entry-level large-sensor DSLRs like the Canon EOS-300D or Nikon D70 will cost. The DSLRs will yield much better pictures including low-light situations at ISO 800.
• Film is your only option today for decent low-light performance in a compact camera. Fuji Neopan 1600 in an Olympus Stylus Epic or a Contax T3 will allow you to take shots in available light without a flash, and spare you the “red-eyed deer caught in headlights” look most on-camera flashes yield.

Conclusion

Hopefully, as the technology matures, large sensors will migrate into the midrange and make it worthwhile. I for one would love to see a digital Contax T3 with a fast prime lens and a low-noise APS-size sensor. Until then, there is no point in getting anything in between – midrange digicams do not offer better image quality than the cheaper models, while at the same time being significantly costlier, bulkier and more complex to use. In fact, the megapixel rat race and the wide-ranging but slow zoom lenses that find their way on these cameras actually degrade their image quality over their cheaper brethren. Sometimes, more is less.

Updates

Update (2005-09-08):

It seems Sony has finally seen the light and is including a large sensor in the DSC-R1, the successor to the DSC-F828. Hopefully, this is the beginning of a trend.

Update (2006-07-25):

Large-sensor pocket digicams haven’t arrived yet, but if you want a compact camera that can take acceptable photos in relatively low-light situations, there is currently only one game in town, the Fuji F30, which actually has decent performance up to ISO 800. That is in large part because Fuji uses a 1/1.7″ sensor, instead of the nasty 1/2.5″ sensors that are now the rule.

Update (2007-03-22):

The Fuji F30 has been superseded since by the mostly identical F31fd and now the F40fd. I doubt the F40fd will match the F30/F31fd in high-ISO performance because it has two million unnecessary pixels crammed in the sensor, and indeed the maximum ISO rating was lowered, so the F31fd is probably the way to go, even though the F40 uses standard SD cards instead of the incredibly annoying proprietary Olympus-Fuji xD format.

Sigma has announced the DP-1, a compact camera with an APS-C size sensor and a fixed 28mm (equivalent) f/4 lens (wider and slower than I would like, but since it is a fixed focal lens, it should be sharper and have less distortion than a zoom). This is the first (relatively) compact digital camera with a decent sensor, which is also a true three-color Foveon sensor as cherry on the icing. I lost my Fuji F30 in a taxi, and this will be its replacement.

Update (2010-01-12):

We are now facing an embarrassment of riches.

• Sigma built on the DP1 with the excellent DP2, a camera with superlative optics and sensor (albeit limited in high-ISO situations, but not worse than film) but hamstrung by excruciatingly slow autofocus and generally not very responsive. In other words, best used for static subjects.
• Panasonic and Olympus were unable to make a significant dent in the Canon-Nikon duopoly in digital SLRs with their Four-Thirds system (with one third less surface than an APS-C sensor, they really should be called “Two-Thirds”). After that false start, they redesigned the system to eliminate the clearance required for a SLR mirror, leading to the Micro Four Thirds system. Olympus launched the retro-styled E-P1, followed by the E-P2, and Panasonic struck gold with its GF1, accompanied by a stellar 20mm f/1.7 lens (equivalent to 40mm f/1.7 in 35mm terms).
• A resurgent Leica introduced the X1, the first pocket digicam with an APS-C sized sensor, essentially the same Sony sensor used in the Nikon D300. Extremely pricey, as usual with Leica. The relatively slow f/2.8 aperture means the advantage from its superior sensor compared to the Panasonic GF1 is negated by the GF1′s faster lens. The GF1 also has faster AF.
• Ricoh introduced its curious interchangeable-camera camera, the GXR, one option being the A12 APS-C module with a 50mm f/2.5 equivalent lens. Unfortunately, it is not pocketable

According to Thom Hogan, Micro Four Thirds grabbed in a few months 11.5% of the market for interchangeable-lens cameras in Japan, something Pentax, Samsung and Sony have not managed despite years of trying. It’s probably just a matter of time before Canon and Nikon join the fray, after too long turning a deaf ear to the chorus of photographers like myself demanding a high-quality compact camera. As for myself, I have already voted with my feet, successively getting a Sigma DP1, Sigma DP2 and now a Panasonic GF1 with the 20mm f/1.7 pancake lens.

Update (2010-08-21):

I managed to score a Leica X1 last week from Camera West in Walnut Creek. Supplies are scarce and they usually cannot be found for love or money—many unscrupulous merchants are selling their limited stock on Amazon or eBay, at ridiculous (25%) markups over MSRP.

So far, I like it. It may not appear much smaller than the GF1 on paper, but in practice those few millimeters make a world of difference. The GF1 is a briefcase camera, not really a pocketable one, and I was subconsciously leaving at home most of the time. The X1 fits easily in any jacket pocket. It is also significantly lighter.

High ISO performance is significantly better than the GF1 – 1 to 1.5 stops. The lens is better than reported in technical reviews like DPReview’s—it exhibits curvature of field, which penalizes it in MTF tests.

The weak point in the X1 is its relatively mediocre AF performance. The GF1 uses a special sensor that reads out at 60fps, vs. 30fps for most conventional sensors (and probably even less for the Sony APS-C sensor used in the X1, possibly the same as in the Nikon D300). This doubles the AF speed of its contrast-detection algorithm over its competitors. Fuji recently introduced a special sensor that features on-chip phase-detection AF (the same kind used in DSLRs), let’s hope the technology spreads to other manufacturers.

Introduction

I am in the process of migrating from Windows to Mac OS X as my primary home computing environment. IMatch is one of the key applications I need to migrate, but it is not available on the Mac. Ghola is an attempt to reproduce the key functionality of IMatch, and possibly go beyond. It is also a good way to learn Cocoa programming (my last programming experience on the Mac goes back to Think C 4.0 accessing raw QuickDraw calls).

Use cases

1. Assign categories to images or a folder of images.
2. Search for images matchin an expression of categories.
3. Restructure a category (e.g. splitting a category grown too large into multiple, more manageable subcategories).

Requirements (incomplete)

The following are the key features from IMatch that need to be carried over:

• The flexible set-oriented category system.

• Support for RAW images (CRW and NEF).

• Fast retrieval performance.

• Extensible metadata.

• Offline media support.

The following are the requirements for Ghola beyond IMatch:

• Ability to group very similar variants of an image together. This would allow to group an image original, retouched versions, cropped and resized versions, or multiple very similar images taken in succession, yet manage them as a single logical unit. The role of each image in the group would be identified as well as part of its group membership, and each group would have a leader used by default.

• Manage assets beyond images, such as PDF files.

• Highly efficient categorization user interface.

• Scriptable in Python.

• HTML gallery generation, integrated with Mylos.

Implementation directions

The system core will be implemented in Python, and C if necessary. It should be as portable as possible. Possibly even multi-user and remotable.

SQLite will be used as the core database. Fast, simple, easy to manage.

GUI front-ends will use the native toolkits (PyObjC) whenever possible for optimal user experience and Aqua compliance.

A command-line UI could be more efficient for category assignment, if it is augmented with features like completion.

We may have to use bitmap indexes for efficient category indexing. Boolean operations on Python long integers are surprisingly fast, and C might not be needed at this point.

Colophon

The System name is a reference to Frank Herbert’s Dune (the original series, of course, not the opportunistic add-ons by Brian Herbert. It has the nice side benefit of not being already used by another open-source program.

Mylos is my home-grown weblog management software. I wrote my first web pages by hand in Emacs and RCS in 1993, but stopped maintaining them in 1996 or so. I only restarted one with Radio last year. After a year of weblogging, however, I find I am frustrated by the limitations of Radio as well as its web-based user interface (I am one of those rare people who prefer command-line user interfaces and non-WYSIWYG HTML editors). I guess I could have extended Radio using UserLand’s Frontier language it is implemented in, but I have no interest in learning yet another oddball scripting language.

I decided in April 2003 to roll my own system, implemented in Python. In my career at various ISPs, I had to kill home-grown content-management system (CMS) projects gone awry, and I was certainly aware that these projects have a tendency to go overboard. Still, it has taken me three months of (very) part-time work to get the system to a point where it generates usable pages and imports my legacy pages from Radio without a hitch.

The implemented requirements for Mylos are:

• Migration of my existing Radio weblog entries and stories (done, but not in an entirely generic fashion, is theme-dependent)
• All pages are static HTML, no requirements for CGI scripts, PHP, databases or the like
• Implemented and extensible in Python
• Separation of content and presentation using themes (based on Webware Python Server Pages and CSS)
• Support for navigational hierarchy
• Articles are stored as regular files on the filesystem where they can be edited using conventional tools if necessary, no need for proprietary databases
• Extensible article metadata
• Atom 1.0 syndication, with separate feeds for subcategories
• Use only relative URLs in hyperlinks to allow easy relocation
• Automatic entry HTML cleanup for XHTML compliance
• A CSS-based layout where the blogroll doesn’t wrap around short bodies (e.g. on permalink pages for short articles).
• reasonable defaults, e.g. don’t try to create a weblog entry for an image that is colocated with an article, just copy it
• Built-in multithreaded external link validation.
• Automatic URL remapping (/mylos/ becomes relative to the Mylos root, relative URLs in an entry are automatically prefixed in containers like home pages).
• Ability to review an article before publishing
• Lynx compliance
• Automatically cache external images in weblog entries in case they disappear (but do not use them as such due to potential copyright issues)
• Set robots meta tag so only permalinks are indexed and cached by search engines, for better relevance to search engine users (albeit at the cost of lower rankings for the home page).
• Sophisticated image galleries fully integrated with the navigation
• Automatic code fragment colorization using Pygments

These features are planned but not yet implemented:

• Keyword index.
• Enhanced support for books via Allconsuming and Amazon.
• Automated dependency tracking to re-render only the pages affected by a change (via SCons)
• Multi-threaded rendering (via SCons)
• Automatically add height, width and alt tags to img tags
• Auto abbreviation glossary as tooltip help using <acronym> tags
• Typographically clean results, as done by SmartyPants
• Feedback loop via on-page comments
• Notification of new comments by email
• Ability to promote a weblog entry to a story if it reaches critical mass

These features are “blue-sky”, don’t hold your breath for them:

• Updates by email
• User-submitted ratings for articles
• Support for multilingual weblogs

Features thet are not planned at all (anti-requirements) include:

• Synchronization or upload to server – rsync does this far better
• Text editor – use $VISUAL or $EDITOR, whether Emacs, vi, or whatever
• Web user interface – Radio’s web interface has very poor usability in my personal opinion, and this is due to the fact it is web-based, not any fault of Userland’s
• RSS 1.0 – RDF seems like an exercise in intellectual masturbation
• Blogger API or similar – although someone else could certainly write a bridge in Python if needed

The software is currently not in a state where it can be used by anyone else. I am not sure if there is any demand for such a tool in any case, if so, I would certainly consider documenting it better and making a SourceForge project out of it.

By the way, the system is named “Mylos” after a city in the magnificent illustrated series “Les Cités Obscures” by Belgian architects and writers Schuiten and Peeters, more specifically L’Enfant Penchéee

Windows gets a lot of heat from Unix zealots (I am one myself) for being unreliable, but any operating system that attempts to comprehensively support all the wide variety of oddball peripherals and software out there is going to experience the same integration problems; certainly, Linux is converging towards Windows in terms of the number of security advisories released. Of course, using an obsolete version like 98 or ME without modern protected memory is a prescription for disaster, but the NT-based versions, i.e. 2000 and XP can have reasonable reliability, at least for desktop usage.

The rest of this article describes my strategy minimizing entropy in my Windows systems.

Separation

The way I approach my Windows configuration is to establish a clear separation between Operating System/Applications and Data.

The OS and Applications do not mean anything special to me other than the amount of work required to reinstall. Data represent actual productive work on my part and must be protected. I separate OS/Applications from data clearly, and make regular checkpoints of OS/Apps after installation and every now and then before I make major changes like installing an application or OS service packs. If my system becomes unstable at some point in time, I can easily revert to a known stable configuration.

The specific tools used to provide this backup of system configuration are a question of personal choice. A number of commercial software like Roxio GoBack or Powerquest SecondChance (since discontinued) purport to do this, as does Windows XP.

I personally don’t trust these programs all that much, and prefer to make a total backup of my system using Norton Ghost. To ensure my data is not erased when I restore from a Ghost image, I have at least two partitions on each of my systems:

1. C: for Windows and applications (NTFS)
2. D: for my personal data (NTFS on desktops, FAT32 on laptops)
3. I: for my Ghost images on desktops (FAT32), on a different drive than C: so I can survive a drive failure

That way I can destroy C: at my leisure, in the worst case I will have to reinstall a couple of applications and reapply some settings that were lost since the last release. My data sits safely on the D: partition (and backups).

Backup strategy

I don’t trust CD-R media or removable drive cartridges for backup purposes, and tape is either too slow or too expensive in the case of DLT. I keep full duplicates of my data partition and some Ghost images on a pair of 100GB external FireWire drives, one I keep at home and one at work. I rotate them weekly so even if my house burns down I will have lost in the worst case only a week of work or photos.

Limitations of this method

This technique doesn’t work very well if the underlying hardware configuration changes too often, and assumes a linear install history. If I install software A, then B, then C, I can go back from A+B+C to A+B or A but not to A+C.

How-to

This section shows how to extricate the data from OS/Apps which Windows and most apps usually try and commingle. The TweakUI utility from Microsoft is an absolute must-have. It is a control panel that allows you to change the behavior of the OS in vital ways that are not accessible otherwise short of editing the Registry directly.

Outlook

Outlook files are the single largest data files on my system (Ghost images do not count). By default, Outlook will create its PST file in the Documents and Settings directory. You can either relocate this directory to the D: partition, or create a new PST file in a location of your choice and use Advanced properties in the properties dialog for the PST in Outlook to make it the default location for POP delivery, after which you can close the old one and delete it.

My Documents

And derivatives like “My Pictures”, can be relocated to D: using TweakUI.

Favorites

For IE users, TweakUI allows you to relocate the Favorites directory to another place than the default. This way, you will not lose your favorites when you have to restore your system.

For Netscape/Mozilla users, the Profile manager utility allows you to set up a new profile with files that are stored where you want, e.g. on the D: partition.