The fallacy of the general purpose tool

I think the same goes for programming languages, and therefore I have learned quite a few, just like I have stocked my pinboard with tools that do one thing very well. Many people find learning new languages onerous and will try to find the one language that does everything they want to do well. Their arguments against learning a new language often involves something like “but I can do that just fine in my language, and then I don’t have to learn a new language”. Here’s an example from real life that illustrates the advantages of special purpose languages.

My father approaches me with this problem. He uses an accounting package which can import OFX files (they’re an SGML/XML format for financial records), but there is a slight problem. The ZAR needs to be changed to USD and the dates (which are tags of the form 20111201 – yes, no closing tag as it’s SGML) have to be changed from YYYYMMDD to YYYYDDMM. Now, turns out that Python has a nice OFX module. So does Java. But the fastest way to make the changes to his files is probably sed:

sed -r -i -e 's/ZAR/USD/' -e 's/<DT(.*)>
([0-9]{4})([0-9]{2})([0-9]{2})/<DT1>243/' filename.ocx

This does the replacement in-place, is really fast and is quick enough to throw together. This is the kind of thing that sed shines at. In fact, it is exactly what it was designed to do, so it is unsurprising that it does it so well.

To do the same thing in Python (even without using the OFX module) requires a bit more effort:

import os
import re
from tempfile import mkstemp

filename = "test.txt"
patternStrings = ["ZAR", r"<TD([^>]+)>([0-9]{4})([0-9]{2})([0-9]{2})"]
replacements = ["USD", r"<TD1>243"]

# Compile patterns
patterns = [re.compile(p) for p in patternStrings]

# Create temporary file to hold outputs
_, tempfile = mkstemp()

# Process file
outfile = open(tempfile, 'w')
for line in open(filename):
    for pattern, replacement in zip(patterns, replacements):
        line = pattern.sub(replacement, line)
    outfile.write(line)

outfile.close()

os.rename(tempfile, filename)

Needless to say, the difference is even more pronounced in Java due to the large amount of boilerplate needed.

import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.util.Scanner;
import java.util.regex.Pattern;

public class Fixer {
    static File infile = new File("test.txt");
    static String[] patternStrings = {"ZAR", "<TD([^>]*)>([0-9]{4})([0-9]{2})([0-9]{2})"};
    static String[] replacementStrings = {"USD", "<TD$1>$2$4$3"};

    public static void main(String[] args) throws IOException {
        Pattern[] patterns = new Pattern[patternStrings.length];
        Scanner in = new Scanner(new FileReader(infile));
        
        // Compile patterns
        for (int i=0; i<patternStrings.length; i++) {
            patterns[i] = Pattern.compile(patternStrings[i]);
        }
        
        // Create temporary file to hold outputs
        File tempfile = File.createTempFile("tmp", "tmp");

        // Process file
        FileWriter outfile = new FileWriter(tempfile);
        while (in.hasNextLine()) {
            String line = in.nextLine();
            for (int i=0; i < patterns.length; i++)
                line = patterns[i].matcher(line).replaceAll(replacementStrings[i]);
            
            outfile.write(line + System.getProperty("line.separator"));
        }        
        outfile.close();
        
        // move temp file back to filename
        tempfile.renameTo(infile);
    }
}

Now, at this point someone is bound to say “but wait, people don’t write desktop apps in sed!”. But that’s quite the point – Java was designed with programming in the large in mind, but it is really tedious for short programs. In many cases, learning a small domain specific language and using it to solve your small problem is faster than learning how to do the same thing in your “one language”.

I wish I had more time to put examples together, but this situation just presented itself and got me thinking.