Year: 2017

This post overlaps a little bit with the the previous one, recapping my programming history.

We ended the previous part in 2002 when I finished my Masters and started teaching, but I want to go into a little more detail on my history with scripting languages.

I encountered Matlab for the first time in my second year (in 1998) in a course called CRV220. I still have the study guide so I can say with confidence that we were taught a combination of Turbo Pascal and Matlab. At the time, the Turbo Pascal environment was much nicer than the Matlab one. Even though the Borland IDE was text-based, it had syntax highlighting and step-by-step debugging. It looked like this:

By contrast, in 1998 Matlab was in version 5.2 and looked like this:

We typed Matlab code into Notepad and executed it from this command window. The most impressive part of Matlab was that it could produce graphical output very easily. But the development environment was non-existent. It was, however, very clear that you could solve engineering problems using much less Matlab code than Turbo Pascal. Matlab was also the first language I encountered which had a REPL – you could type commands into the command window one at a time and see the result. I really liked this concept.

In 1998 also bought an HP 48 GL calculator and spent a lot of time programming it do solve problems like balancing chemical equations and so on. We were often allowed to use our calculators in tests, but they were often cleared before we wrote. The HP introduced me to the idea of stack-based languages.

In my final year (in 2000), we did our final year project calculations using a spreadsheet called Quattro Pro, since Excel had not yet achieved the dominant position it currently enjoys. We did not do much programming in our final year.

By the time I started my Masters in 2001, Matlab had just undergone a major rework in the release of version 6, moving to a Java-based workspace and for the first time Matlab was an IDE with an integrated code editor, debugging, and all the features from the Turbo Pascal IDE that I had missed when I first encountered Matlab. I found this new environment quite comfortable and used it happily throughout my Masters work. I used it to do the actual simulation and to produce all the graphs in my dissertation. But I had also bumped up against one of the things that Matlab was not very good at when I was writing the code to make the call graph I posted in the previous part: Matlab was really terrible at text processing.

By 2002 I was completely immersing myself in the Unix way. I had started to see the power of the “small tools doing one thing well” philosophy and spend a great deal of time reading the GNU Info pages for all the various unix tools while also learning about Richard Stalman and the Free Software movement. I was using Matlab on my Linux machine and learning bash. This is when I started learning Emacs, after reading In the beginning was the commandline where Neal Stephenson says the following about it:

“I use emacs, which might be thought of as a thermonuclear word processor. It was created by Richard Stallman; enough said. It is written in Lisp, which is the only computer language that is beautiful. It is colossal, and yet it only edits straight ASCII text files, which is to say, no fonts, no boldface, no underlining. In other words, the engineer-hours that, in the case of Microsoft Word, were devoted to features like mail merge, and the ability to embed feature-length motion pictures in corporate memoranda, were, in the case of emacs, focused with maniacal intensity on the deceptively simple-seeming problem of editing text.”

Teaching

In the first semester of 2002 I was the teaching assistant for CRV210 (the same subject where I had learned Matlab, now moved to the first semester). In the second semester I took over CMO320 (Mass transfer) from a lecturer who had moved to New Zealand. At this point I was done with my Masters and thinking of but had not yet enrolled for my PhD. I barely kept my head above water in that first semester, having to revise the material even as I taught it. In the second go around I started using more computer methods in my teaching, using Excel and Matlab to solve mass transfer problems.

As I did this, I started realising that our students were not well prepared to deal with computer solutions of these problems. I eventually moved over to teaching our computer course.

Other languages

I started evaluating different scripting languages which would be better at the kinds of string processing I had done for the call graph program and came across Perl and Python around this time. The earliest proper Python program I could find in my records was from 2004 and was for creating the same kind of file I used to do the call graph (the GraphViz package). I also learned the classic grep, sed and awk combination for text processing in Linux, motivated largely by reading “Unix Text Processing“

I picked up Fortran around 2005 as I started looking for ways to speed up my simulations. In the process I learned that modern Fortran was not at all like the FORTRAN 77 that people love to mock. It is a pretty decent language for numerical work. I learned R in 2007 while I was looking for better ways to handle statistical graphics. At that time R was the best language to learn for statistical work. Some might say it still is, even though Python has the big momentum for data science as of 2017.

Teaching programming

Original challenge: Standard library only

I set myself a little challenge to write a simple spreadsheet-like program in Python in less than 100 lines, using only the standard library.

The code in this repository manages it in 100 non-blank lines, including comments. 88 if you ignore the comments. The full code is also at the end of this post. Note: I have also redone the code in PyQT5. This is discussed after the tkinter code.

This is what it the program looks like when you run it:

You can click on cells to edit their formulae and move around with the arrow keys. Formulae are evaluated once you press Return or navigate way from a cell. Every cell is evaluated as a Python expression. This means you don’t do the Excel thing of having formulae start with =. Everything is a fomula. So the formula in A2 in the screenshot above is valid and will evaluate to 1 once focus moves away from the cell. The math module is imported by default, so you can use sin and cos and so on. I use a very cheap and nasty call of eval to make it all work, which you should never do for production code.

Other than using eval, I also play a little fast and loose with the way objects are allowed to touch each other in the code. I allow each cell to access all their siblings directly instead of going through an intermediary. I probably also break a couple of other OO rules along the way with the way the cells propagate their calculations when they are changed. But the whole code only took a couple of hours to write and it actually works.

And now, here is the full code:

#!/usr/bin/env python

import tkinter as tk
import math
import re
from collections import ChainMap

Nrows = 5
Ncols = 5

cellre = re.compile(r'b[A-Z][0-9]b')


def cellname(i, j):
    return f'{chr(ord("A")+j)}{i+1}'


class Cell():
    def __init__(self, i, j, siblings, parent):
        self.row = i
        self.col = j
        self.siblings = siblings
        self.name = cellname(i, j)
        self.formula = '0'
        self.value = 0
        # Dependencies - must be updated if this cell changes
        self.deps = set()
        # Requirements - values required for this cell to calculate
        self.reqs = set()

        self.var = tk.StringVar()
        entry = self.widget = tk.Entry(parent,
                                       textvariable=self.var,
                                       justify='right')
        entry.bind('<FocusIn>', self.edit)
        entry.bind('<FocusOut>', self.update)
        entry.bind('<Return>', self.update)
        entry.bind('<Up>', self.move(-1, 0))
        entry.bind('<Down>', self.move(+1, 0))
        entry.bind('<Left>', self.move(0, -1))
        entry.bind('<Right>', self.move(0, 1))

        self.var.set(self.value)

    def move(self, rowadvance, coladvance):
        targetrow = (self.row + rowadvance) % Nrows
        targetcol = (self.col + coladvance) % Ncols

        def focus(event):
            targetwidget = self.siblings[cellname(targetrow, targetcol)].widget
            targetwidget.focus()

        return focus

    def calculate(self):
        currentreqs = set(cellre.findall(self.formula))

        # Add this cell to the new requirement's dependents
        for r in currentreqs - self.reqs:
            self.siblings[r].deps.add(self.name)
        # Add remove this cell from dependents no longer referenced
        for r in self.reqs - currentreqs:
            self.siblings[r].deps.remove(self.name)

        # Look up the values of our required cells
        reqvalues = {r: self.siblings[r].value for r in currentreqs}
        # Build an environment with these values and basic math functions
        environment = ChainMap(math.__dict__, reqvalues)
        # Note that eval is DANGEROUS and should not be used in production
        self.value = eval(self.formula, {}, environment)

        self.reqs = currentreqs
        self.var.set(self.value)

    def propagate(self):
        for d in self.deps:
            self.siblings[d].calculate()
            self.siblings[d].propagate()

    def edit(self, event):
        self.var.set(self.formula)
        self.widget.select_range(0, tk.END)

    def update(self, event):
        self.formula = self.var.get()
        self.calculate()
        self.propagate()
        # If this was after pressing Return, keep showing the formula
        if hasattr(event, 'keysym') and event.keysym == "Return":
            self.var.set(self.formula)


class SpreadSheet(tk.Frame):
    def __init__(self, rows, cols, master=None):
        super().__init__(master)
        self.rows = rows
        self.cols = cols
        self.cells = {}

        self.pack()
        self.create_widgets()

    def create_widgets(self):
        # Frame for all the cells
        self.cellframe = tk.Frame(self)
        self.cellframe.pack(side='top')

        # Column labels
        blank = tk.Label(self.cellframe)
        blank.grid(row=0, column=0)
        for j in range(self.cols):
            label = tk.Label(self.cellframe, text=chr(ord('A')+j))
            label.grid(row=0, column=j+1)

        # Fill in the rows
        for i in range(self.rows):
            rowlabel = tk.Label(self.cellframe, text=str(i + 1))
            rowlabel.grid(row=1+i, column=0)
            for j in range(self.cols):
                cell = Cell(i, j, self.cells, self.cellframe)
                self.cells[cell.name] = cell
                cell.widget.grid(row=1+i, column=1+j)


root = tk.Tk()
app = SpreadSheet(Nrows, Ncols, master=root)
app.mainloop()

Part 2: PyQT5

I posted this blog to Reddit and one of the commenters mentioned that PyQT5 has a nice table widget which is more spreadsheet-like. So I re-did the spreadsheet in PyQT5. The code is functionally similar at the moment and also has approximately the same number of lines (87 non-comment code lines). The main differences are that I could use the QTableWidget to handle all the navigation. I had to do some interesting things with delegates to enable to editing mode I wanted, though, which cost me all the lines I saved from the navigation. I think my next step is going to be to extract the calculation logic into a common class which both of these can use – there is way too much duplication going on here right now!

So, this is what the PyQT5 one looks like. It’s a little more compact, but otherwise very similar. The interaction is slightly more spreadsheet-like though, and you get automatic highlighting and selection of multiple cells for free. Very nice.

And here is the code. You will notice a great deal of similarity. The key to getting the edit mode thing to work properly is the piece of magic with SpreadSheetDelegate. I didn’t think of that myself – I stole it from this PyQT5 example, which has a much nicer spreadsheet (you can change colours, fonts, print etc) with much worse formula logic.

#!/usr/bin/env python

import re
import sys
from collections import ChainMap
import math

from PyQt5.QtCore import Qt
from PyQt5.QtWidgets import (QApplication, QMainWindow, QTableWidget,
        QTableWidgetItem, QItemDelegate, QLineEdit)


cellre = re.compile(r'b[A-Z][0-9]b')


def cellname(i, j):
    return f'{chr(ord("A")+j)}{i+1}'


class SpreadSheetDelegate(QItemDelegate):
    def __init__(self, parent=None):
        super(SpreadSheetDelegate, self).__init__(parent)

    def createEditor(self, parent, styleOption, index):
        editor = QLineEdit(parent)
        editor.editingFinished.connect(self.commitAndCloseEditor)
        return editor

    def commitAndCloseEditor(self):
        editor = self.sender()
        self.commitData.emit(editor)
        self.closeEditor.emit(editor, QItemDelegate.NoHint)

    def setEditorData(self, editor, index):
        editor.setText(index.model().data(index, Qt.EditRole))

    def setModelData(self, editor, model, index):
        model.setData(index, editor.text())


class SpreadSheetItem(QTableWidgetItem):
    def __init__(self, siblings):
        super(SpreadSheetItem, self).__init__()
        self.siblings = siblings
        self.value = 0
        self.deps = set()
        self.reqs = set()

    def formula(self):
        return super().data(Qt.DisplayRole)

    def data(self, role):
        if role == Qt.EditRole:
            return self.formula()
        if role == Qt.DisplayRole:
            return self.display()

        return super(SpreadSheetItem, self).data(role)

    def calculate(self):
        formula = self.formula()

        if formula is None or formula == '':
            self.value = 0
            return

        currentreqs = set(cellre.findall(formula))

        name = cellname(self.row(), self.column())

        # Add this cell to the new requirement's dependents
        for r in currentreqs - self.reqs:
            self.siblings[r].deps.add(name)
        # Add remove this cell from dependents no longer referenced
        for r in self.reqs - currentreqs:
            self.siblings[r].deps.remove(name)

        # Look up the values of our required cells
        reqvalues = {r: self.siblings[r].value for r in currentreqs}
        # Build an environment with these values and basic math functions
        environment = ChainMap(math.__dict__, reqvalues)
        # Note that eval is DANGEROUS and should not be used in production
        self.value = eval(formula, {}, environment)

        self.reqs = currentreqs

    def propagate(self):
        for d in self.deps:
            self.siblings[d].calculate()
            self.siblings[d].propagate()

    def display(self):
        self.calculate()
        self.propagate()
        return str(self.value)


class SpreadSheet(QMainWindow):
    def __init__(self, rows, cols, parent=None):
        super(SpreadSheet, self).__init__(parent)

        self.rows = rows
        self.cols = cols

        self.cells = {}

        self.create_widgets()

    def create_widgets(self):
        table = self.table = QTableWidget(self.rows, self.cols, self)

        headers = [chr(ord('A') + j) for j in range(self.cols)]
        table.setHorizontalHeaderLabels(headers)

        table.setItemDelegate(SpreadSheetDelegate(self))

        for i in range(self.rows):
            for j in range(self.cols):
                cell = SpreadSheetItem(self.cells)
                self.cells[cellname(i, j)] = cell
                self.table.setItem(i, j, cell)

        self.setCentralWidget(table)


if __name__ == '__main__':
    app = QApplication(sys.argv)
    sheet = SpreadSheet(5, 5)
    sheet.resize(520, 200)
    sheet.show()
    sys.exit(app.exec_())

The problem

One of my postgraduate students is writing a simulation code in Python. Some of the requirements have involved us generating files containing Python programs to be executed later. But while I was going through the code today I spotted this:

    dist_code = 'def {0}(random_number):n    values = {1}n    cum_integrate = {2}n    return numpy.interp(' 
                'random_number, cum_integrate, values)'.format(funk_name, list(values), list(cum_integrate))

    return ast.parse(dist_code)

What’s happening here? We’re building a function from arguments. I was intrigued, as I thought this would be basically equivalent to:

    def f(random_number):
        values = list(values)
        cum_integrate = list(cum_integrate)
    f.__name__ = funk_name
    return f

I told him so and went along my way. Later, we had this interesting exchange on Slack which showed that he didn’t understand functions as they are. Perhaps you don’t either?

Imponderables

Do you understand the output of this code?

funcs = []
for i in range(10):
    def f1():
        print(i)
    funcs.append(f1)

funcs[0]()

Output: 9

What about this:

i = 1
def f2():
    print(i)

f2()
i = 2
f2()

Output:
1
2

The two effects are explained the same. Because a function’s internal code isn’t executed when it is defined, the name i will only be looked up when it is run. It will then follow the LEGB rule (Local, Enclosing, Global, Built-in) at the place where it is called. Since i is local, it will resolve to the same variable and show the same value in the first exaple and different values in the second example.

Solutions

Here are a couple of ways of getting the desired behaviour of remembering the value the function was defined with. In all cases it involves creating a new name which will take precedence in the scoping rules.

Method 1: Default argument

funcs = []
for i in range(10):
    def f(i=i):  # New name created inside f scope
        print(i)
    funcs.append(f)

Method 2: Function in a function

def function_factory(i):  # This i is a new name
    def f():
        print(i)
    return f

funcs = []
for i in range(10):
    funcs.append(function_factory(i))

Method 3: Objects

class NumberPrinter:
    def __init__(self, i):
        self.i = i  # This is the new name
    def call(self):
        print(self.i)

funcs = []
for i in range(10):
    funcs.append(NumberPrinter(i).call)

You can do this in two ways – the other involves redefining a dunder method:

class NumberPrinter:
    def __init__(self, i):
        self.i = i  # This is the new name
    def __call__(self):
        print(self.i)

funcs = []
for i in range(10):
    funcs.append(NumberPrinter(i))

Now we don’t actually have functions in the list, but objects. We can do a little more introspection this way.

One last thing

I often see people defining functions over and over when they want to do something with a local variable. It typically comes up when people are using higher-order functions to solve equations or minimise a function:

for i in range(10):
    def f():
        print(i)
    f()

Convince yourself that the code above and the code to follow do the same things. This one just does it faster, since we’re not incurring the cost of building a new function object.

def f():
    print(i)

for i in range(10):
   f()

Disclaimer: This post is unresearched and contains no links as I’m just working from an idea I had this morning. I’d love to hear if I got something wrong, but I’m going fast to stop myself from getting stuck on this.

In the beginning was the word

Language developed long before we wrote it down. But I think as soon as we started speaking to one another we had the problem that it is possible to say things that are different from the way things actually are. This is unlike in-person pointing and grunting, where we are restricted to the way things are in the world. We can’t point at an object that doesn’t exist, but we can speak of it.

This is a wonderful advancement because it allows use to communicate things like plans for the future or relate stories about events which happened in the past. It’s also the entry point to abstract thought and math. On the downside it allows for a whole set of possibilities in subterfuge. It allows us to promise things we have no intention of doing. It allows us to lie.

I think to balance this in a positive direction, we have developed great social pressure not to be caught in a lie. But when all communication is oral, there is an out – you can deny what you said. It is understood that we have imperfect recall, and therefore a defence like “that’s not what I said” is somewhat defensible.

Then we started writing things down

Now we record everything

I recently posted this tweet:

The world would be better if we had one language in common, drove on the same side of the road and used the same units. #standards

— Carl Sandrock (@chthonicdaemon) January 9, 2017

I was surprised by the number of people who protested on Facebook. Most of the problems were with the language rather than the other elements I mentioned. There were a couple of different arguments raised:

1. People are attached to their language and wouldn’t agree to stop talking their language. 
2. Things would be boring if they were the same everywhere
3. Less diversity means less creativity
4. It is impossible to decouple language, culture and identity and people don’t want to change them

Let me first say that when I was writing the original tweet I was careful to word it as “we had one language in common” rather than “everyone spoke the same language”. The sense I was attempting to convey is not that I would like for everyone to give up their current language to speak only one language but that I would be happy if everyone spoke one common language (perhaps in addition to other languages). What is important to me is mutual intelligibility. In other words, I would like to be able to understand what another person is trying to tell me regardless of where in the world that person comes from. I realise that many people strongly identify with their language and see language as an integral part of their culture and identity. When I was thinking about the concerns raised by people I was thinking about differences which would be equivalent to me in terms of universal intelligibility but which would involve different sacrifices or advances in technology to implement.