My Digital Workflow
This page describes the system I came up with, over time, for working with large numbers of digital image files. My goal is to enter some images into this website, while maintaining a usable and permanent archive. The software I use is perl and MySQL on the linux operating system; the hardware is Compact Flash memory cards, a card reader, several hard drives and a DVD burner.
After inserting a flash card and mounting the Compact Flash card reader, I run a perl script I wrote to download the images. It gets a listing of all the image files, then scans their exif data to determine the date the photo was taken.
I store my images in date-named folders in YYYY-MM-DD format, so the perl script copies all the images from the flash card to appropriate named folders. Any folders that don't already exist are created on the fly.
The perl script then updates a database table, adding an entry for each image file. The fields include the folder and filename for each image, date and time taken, it's filesize, and md5sum. This table currently has 119,347 entries (as of October, 2008), starting in February of 2000. This table gives each image file a database ID number which can identify it's exact location.
As part of my backup system, I make data DVDs of the image files whenever there are enough to fill up a new 4+GB DVD. I also recently implemented a backup hard drive, which I syncronize with the main drive using rsync.
Sorting into the website
I created a web interface to my image archive which allows me to browse exif thumbnails of the files in any folder in a web browser. This is a perl CGI script which extracts exif thumbnails on the fly and stores them in a folder in another location on the hard drive.
The CGI script allows me to select individual thumbnails (with a checkbox) which I want to process further for including in my website. Reduced versions are generated from the high-resolution originals for the selected files. These are made on the fly using ImageMagick and stored in the same folder as the exif thumbnails.
The remaining steps are the most labor-intensive. Images are grouped into sets, edited down, and the levels are adjusted. Next, they are carefully keyworded and descriptive information is added. They are then entered into the website, and given a website photo ID number. The original file id number is preserved, and still identifies the original digital file in the archive.
This system evolved over several years and different variations, and will probably change further.