Negative Feedback

Category: Uncategorized

  • More history, some musing about syncing files

    It’s interesting how your tools affect your choices and how you can end up in a place which works for your current needs but doesn’t make the next step you’re trying to get to easy.

    To understand my current situation regarding managing my files, it’s probably best to go into a little bit of history.

    Early computer use: no syncing

    I enrolled as an undergraduate in 1997. When I arrived on campus I was given my very first e-mail address and I remember spending huge amounts of time in the computer labs writing to all of my friends (using a program called Pine, which looked like this):

    Pine screenshot

    Files were small and you carried them around with you on things that we called stiffies. The Americans called them 3½-inch floppy disks. The capacity of a stiffy was 1.44 MB. This was enough for most of your work in a semester. Stiffies were notoriously unreliable, so you never really worked on the disk only, because for all you knew, they could die tomorrow. Stiffies were for moving files from one computer to another, not for long-term storage. Some people had CD writers, but campus computers didn’t.

    I didn’t have a computer in my res room until my second year, and when I did it wasn’t connected to any kind of network until third year. I could only check my mail and do IRC on campus computers.

    Once I got a computer in my room, an issue emerged: sometimes I would start working on something in my room and I would need to continue working on it on campus. So I would copy it to a stiffy and ride my bike to campus, where I would copy it over and keep working. This process led to copies of the files on both machines with no clear way to keep track of which one I had worked on last. So if I forgot my disk on my desk and worked on the campus copy instead of the version I had in my room, things would break. Luckily files were small at first, but technology rapidly improved and by final year (2000) we had CD writers. At this point, we had Internet connections, but they were so slow that it was much faster to burn a CD and bike to campus than to send the data over the Internet. The fastest dial-up modem speed was 56 kbps, which would require more than a day to transfer the 700 MB that fits on a CD.

    For my entire undergraduate career, therefore, I could never sync my account on campus with my computer at res or the one at home. I just had some random files in various places which I had to transfer piecewise.

    Postgraduate: two computers, external HDD

    When I enrolled for my Masters in 2001, I moved into a lab and got assigned a computer. Now I had a computer in my room at res and one at the lab and I often found myself having to transfer files between them. I installed a removable hard-drive bay enclosure and had a hard drive that I would shuttle between the two locations. Moving things this way was a big deal. IDE hard drives don’t support hot-swapping, so I would sync to the hard drive, turn off my computer, remove the hard drive and then boot up with the hard drive in place at the new location. I believe that this hard drive was either a 500 MB or 1 GB. It is around this time that I started using Unison to sync the external hard drive to the machine where I was working. This meant I could survive forgetting to bring the hard drive to the lab, since Unison stores a hash of all the files and can recognise if two files have changed. I synced basically all of my files except for music files and movies, which were too large to fit on this external hard drive.

    Two desktops, laptop, local network

    When I started working for the University of Pretoria in 2003, I continued to use the HHD sync strategy for a while, and when I got a laptop, I started treating it similarly to the HHD – I would use Unison, but now using the (wired) network to connect the machines instead of plugging in a hard drive. I continue to use this strategy today. Every morning before starting to work on my desktop, I sync. Every evening before leaving, I sync. This way I have a complete copy of my working files on both machines (my laptop and my desktop). This provides a level of redundancy (I don’t think of this as a backup strategy, that’s handled by Bacula on my desktop and these days by Time Capsule on my laptop).

    A change: cheap Internet

    During my first couple of years at the university, we were charged R1/MB for internet access. Let me repeat that in units that make more sense to a modern reader: that’s R1000/GB. This means that any kind of large file transfer using the Internet was completely out of the question. It also led to me turning into a packrat. Since downloading software or even articles was so expensive, I developed a huge folder of downloads that I would maintain to avoid having to re-download files.
    Rummaging around my old e-mail, I find that in 2005 we had a limit of R60/month in place. That’s right, ten years ago the University limited my monthly bandwidth to 60 MB. In 2011 we were being charged 16 c/MB (R160/GB) on student access, but staff caps had been dropped. 
    Dropbox launched at the end of 2008, registering dropbox.com in 2009. This is when the idea of easily syncing files over the Internet really started taking hold. I think I installed Dropbox in 2010, when our IT department dropped caps.
    At the moment I use Dropbox for sharing files with other people and I still use Unison to handle syncing the files on my laptop and my desktop.

    A remaining issue: campus proxy/firewall

    Even though we are no longer charged for Internet access on campus, we still have to log in to our proxy server to get access to the Internet. This does not work transparently for all applications. In addition, for obvious reasons, we can’t set up a server on a machine inside campus which will be visible off-campus. I have only recently (early 2015) been able to set up a virtual machine inside our DMZ which would allow me set up a central repository for files.
    I have to switch Dropbox between proxy and no proxy every time I move from home to campus.

    Side-effects

    Because I built my syncing strategy when Internet access was either slow or expensive or both, and because I’ve been working on two different computers regularly, I have avoided dependence on client-server technologies. Specifically, I have not depended on a client-server database like MySQL or PostgresSQL. Instead, when I required SQL database functionality, I’ve use SQLite, which allows me to continue with the file-based syncing strategy.
    But now I’ve started investigating NoSQL technologies like Mongodb or Redis (and later, perhaps something like Hadoop). It doesn’t appear as though these things have a simple file-based alternative similar to SQLite.

    Why change what works?

    It is clear from my interaction with industry that their model is very solidly built around central databases with access from clients. This is a similar model to all the Internet services we use now. I’m familiar with setting up these systems, but my students and I often have only intermittent access to the the Internet. The students because they can’t afford it, myself because I travel using the Gautrain, and often have to work in a disconnected state. Not to mention load shedding.
    Now I am interested in building a better system which has some of the nice properties I have come to rely on with Unison, or at least allows for limited offline use with periodic syncing. I know this will not be as simple as setting up two directories to sync, but I’m getting the horrible feeling that each system will have its own problematic story and that the central server strategy is just a way of avoiding sync rather than a solution.
    If anybody has a suggestion, I’d be pleased to hear it.

  • Numpy matrices work for general objects

    A student was having difficulty using a class I had created to represent transfer function objects. I discovered that they were actually placing these objects in a numpy.matrix and trying to multiply things by one another. To my surprise, this actually worked! I had to investigate. The code here is all in Python.

    I created a quick tracing object which just kind of logs the operations that are done on it as an expression:

    class operation:
    def __init__(self, name):
        self.string = name
    def __repr__(self):
        return self.string
    def __str__(self):
        return self.string
    def __add__(self, other):
        return operation("({} + {})".format(str(self), str(other)))
    def __mul__(self, other):
        return operation("{}*{}".format(str(self), str(other)))
    def __radd__(self, other):
        return operation("({} + {})".format(str(other), str(self)))
    def __rmul__(self, other):
        return operation("{}*{}".format(str(other), str(self)))
       
    Now I can do some interesting stuff:
        In []: a*b
    Out[]: a*b
    In []: a + b
    Out[]: (a + b)
    Ok, but what about matrix multiplication?
    import numpy as np
In []: tracer = np.matrix([[a, b, c],
[d, e, f],
[g, h, i]])

In []: tracer*tracer
Out[]:
matrix([[((a*a + b*d) + c*g), ((a*b + b*e) + c*h), ((a*c + b*f) + c*i)],
        [((d*a + e*d) + f*g), ((d*b + e*e) + f*h), ((d*c + e*f) + f*i)],
        [((g*a + h*d) + i*g), ((g*b + h*e) + i*h), ((g*c + h*f) + i*i)]], dtype=object)
    That kind of blew me away! It means I can probably get away with a whole lot less code than I thought I would need to implement my own matrix-like objects.
    In addition, it appears that some numpy functions will attempt to call corresponding functions in the object they act on:
        In []: np.exp(a)
---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)
 in ()
----> 1 np.exp(a)
AttributeError: exp
    This error shows that np.exp is trying to access the exp attribute on this object. Let’s extend our object with an exp method like this:
        def exp(self):
        return operation("e^({})".format(self))
    Now (after re-instantiating a-i from above), we have
    In []: np.exp(a)
Out[]: e^(a)
    This is going to be super useful.

  • Understanding electricity terminology

    With the recent bout of load shedding, everyone’s been writing about electricity. The problem is that many people get things wrong, or apply conventions which are less useful than they think.

    The basics

    Electricity is fundamentally about the flow of electrons. You may say that it is also about magnetic fields, but most of the terminology you’ll be seeing in the articles about load shedding is about making electrons move. Now, electrons are very small, so it makes sense to count them in big groups rather than individually. In the SI system, the coulomb (C) is defined as exactly 6.241×1018 electrons. You can think of this as equivalent to something like a truckload of coal. Since every material we work with has many electrons already, just having them is not particularly useful. Electrons can do useful things when they are in motion. When there is a flow of electrons in a conductor, this is referred to as current and measured in ampere (A). 1 A = 1 C/s, although technically the ampere is the base unit and the coulomb is the derived unit in SI, so 1 C = 1 A⋅s . To continue our analogy, if coulomb is like “truckloads”, ampere would be “truckloads per day”.

    Flowing electrons can be harnessed to do work in the same way that a flowing river can be harnessed. The rate at which they can do work is related not only to how fast they are flowing (the current) but also to the potential difference (measured in volts, V) between the end points. In a river, this would be the pressure. Work is measured in joules (J), which is also the unit of energy. The rate at which work is done (also called the power) is measured in watt (W). 1 W = 1 J/s. For a constant current (DC), the power is the product of the current and the potential difference. This means that 1 W = 1 V⋅A. This is approximately true for power supplied by batteries. For a thoroughly mixed metaphorical space, let’s say that energy (J) is like a distance and power (W) is like a speed.

    It gets a little more complicated if the current is not constant. The kind of electricity Eskom supplies is sinusoidally varying (AC), which means that we need to distinguish the “apparent power” and the “real power”. This Wikipedia page is a pretty good resource for this idea. The calculation doesn’t change the units in SI, although there are conventions which I will discuss a bit later.

    In terms of the load shedding, W is the unit that will be used to talk about the amount of load to be shed. Load in this context is the same as power.

    Orders of magnitude

    Some of the SI units discussed above are not sized reasonably for everyday use. For instance, a 100 W lightbulb burning for one day will consume 8 640 000 J of energy. For this reason the SI has prefixes for different orders of magnitude. I can choose to express the energy consumed by that lightbulb as 8.64 MJ to save space. It’s anyone’s guess why the load shedding limits are reported in MW rather than GW. Why say 1000 MW when you could have said 1 GW?

    The problem of time

    In the discussion above, I have restricted myself to the SI system. The SI unit of time is the second (s), and all the units which reference time are built using the second. Time calculations are tricky, because of the fact that there are 60 seconds in a minute and 60 minutes in an hour. Those factors aren’t powers of 10, so they don’t fit smoothly into the decimal system that SI uses. This means that in various industries, the quantities discussed above have been measured using different time units. For instance, your electricity bill probably specifies your electricity usage using the “kWh”, which is the energy used by a 1 kW device operating for 1 hour. Notice that this is the same dimensional combination as the joule. 1 kWh = 1 kW⋅h and 1 J = 1 W⋅s. In fact, 1 kWh = 3.6 GJ, so there’s no real nead for the kWh, even in terms of easy magnitudes. The real difference boils down to the difficulties of manipulating factors of 60.

    When measuring the storage capacity of batteries, one will mostly see A⋅h being used rather than the dimensionally-similar C. Smallish batteries like AAs typically have capacity of around 10 kC (meaning that this is the number of electrons they can push around a circuit), but you are more likely to see that reported as 3000 mAh. I believe this is again due to the problems of time calculation, as there shouldn’t be much other difference between using A⋅h instead of A⋅s.

    The kWh is such a popular unit of energy that it is even used for derived rates. People will report the average energy production of  the Jasper solar facility as 180 000 MW-hours annually rather than saying that it will produce 648 TJ per year or produce at a rate of 20.5 MW on average over a year

    Conventions

    There are people reading this who will object viscerally to the calculation above, especially if they have been in the electricity industry. There are certain conventions regarding units which are widespread but don’t really make much sense from a dimensional point of view. One of them is that electrical energy is measured primarily in the kWh family of units, while the joule is restricted to other forms of energy. They would say that it is not proper to report the energy production of that solar plant in TJ as that sounds more like the energy supplied by fuel.

    Similarly, if you refer to the discussion about power calculations for time-varying current, there are people who insist on saying V⋅A is different from W rather than saying apparent power is different from real power and using W for both.

    Peaks and averages

    The last confusing thing about talking of energy is in being clear about peaks and averages. To continue the car analogy, it is pretty clear when someone says that they will travel 100 km in 1 hour that they will average 100 km/h but that they probably spent some of the time above that speed and may even have hit 160 km/h at some point. Remembering that the distance is like energy and that speed is like power, we can say that the 774 PJ of energy SA used in 2010 according to Wolfram Alpha means that we averaged 24.5 GW for that year. Why do we have a problem if Eskom has around 41 GW of generating capacity? For one thing, not all that capacity is on line at once. Due to various factors Eskom only has about 24 GW of generating capacity on line right now. Of course, the other problem is that peak demand is more than average demand. The load shedding of 4 November 2014 happened on a day where there was 28 GW of demand, leaving Eskom 4 GW short.

    While I’m on this topic, let’s also explain the somewhat confusing fact that the Jasper solar energy plant I linked to earlier reports “Size: 96 MW-DC installed capacity; 75 MW-AC net generation, Electricity Production: approximately 180,000 MW-hours annually”. We translated that last number to 20.5 MW. So what’s happening between the 96 MW and the 20 MW? The 96 MW is what the panels will produce when they have 1 MW/m² of solar irradiation. In SA, we have more like 1.2 MW/m², so they’ll see more than that at peak production. This is a bit like the maximum speed in the book that came with your car. Often you can get better than that if you run at the coast or use better fuel than they tested with. The 75 MW is pretty clear, that’s the amount of AC power they will deliver under the same conditions as the 96 MW was calculated. This includes the use of the station of its own power and conversion losses. Of course, the sun is only available part of the day, and the light is not as strong for the whole day either. There is also maintenance and other stoppages, which cuts the effective rate of production down to the 20 MW number. This final number divided by the “nameplate capacity” of 96 MW is known as the capacity factor, and 20 % is about par for the course for solar.

    Wrap-up

    So there you have it – this should enable you to decipher the different terms used in articles talking about electrical energy (and perhaps to do some research about that battery backup for your power at home). My last word on the matter is that you should be on the lookout for common misperceptions about units, like that kWh for some reason means kW/h instead of kW⋅h. Hopefully the discussion above will show you why kW/h doesn’t make any kind of sense.

  • One day software bootcamp for engineers

    Last week I spent a day going over the basics of the Unix shell, version control using git and some Python with engineering students. Here are some of the things I said, links to interesting tools and my insights from crowd feedback.

    Motivation

    Every year, new project students arrive on campus and are asked to do larger projects than they are used to. The methods they have used for data processing up to now are usually not up to the task. The ubiquity of the spreadsheet means that most of them have opened their files in a spreadsheet and then manually applied whatever operations they needed. In my fourth year projects and also at postgraduate level, the problems of scale mean that this method probably won’t work.

    I’ve been inspired by the people from Software Carpentry to set up a longish session where I help students to install the software they need and start to understand the power of the tools. The goal is to get them over the initial stage where you don’t know anything to a place where they can search for the answer themselves. I don’t have permission from the Software Carpentry guys to call this one of their events, and I only loosely based my session on theirs. They appear to have more of a life-sciences focus, whereas I tried to use engineering-type data in my session.

    I also chose to do this in Windows, but to use a Unix shell in order to ease the transition for those who would end up using Linux for their simulation. It doesn’t make sense to throw someone into the deep end here. It’s better to feel like you can do something at the end of a session than being made to feel powerless.

    Installers

    I chose the following installers for the session:
    • The Windows version of git. This is a good start as it includes git-bash, which also has a great subset of the unix tools.
    • The Python(x,y) distribution of Python. There are newer distributions around, but this one is still up to date and very user-friendly. If you are going to be doing science in Python, you need a distribution like this, as the batteries-included philosophy of Python doesn’t extend to science (yet).

    Unix shell

    I start with a review of where Unix comes from. As an aside, I highly recommend In the beginning… was the commandline for a pep-talk about how awesome the command line is as well as the most complimentary description of Emacs I have ever read. It helps to keep a picture of a teletype in mind when working on the terminal as it informs many of the design decisions.
    Now that everyone is in line-by-line mode, we can start the installer. The first two screens provide talking points about the differences between the terminal and the shell, as well as a profitable digression on the problem of different end-of-line conventions.
    After running the git installer, you have access to git bash, which you can start from the start menu. 
    Of course, we start with ls, and explain the idea of flags as well as the concept of a working directory (pwd and cd are introduced here, as well as . and ..). Then I move on to other commands, including echo, which allows me to explain how blobbing is handled by the shell rather than by each command we will be using. Most of this part is just me showing how useful the shell can be, with examples motivating cat, cut, grep, head, wc and some others. As the examples spin out, we start to use redirection and piping. I think it is more important to leave with a sense of awe at the possibilities, as well as some basic understanding of what to search for than any real expertise in the command line. It may also be useful to go through the Software Carpentry shell lessons.

    Version control

    Now that everyone is roughly comfortable with the command line, it’s time to start using git bash for what it’s named for: git.
    I’m not going to recapitulate the whole lesson here, especially because it’s better written up at Software Carpentry but the core is instilling a good set of nomenclature built around pictures which represent the structures git uses. First, here is how the repository commands work:
    Figure showing the parts of a git repository, with commands to move between them
    So this takes us up to basic commits. Now, we need to think about how commits relate to one another and how branching works. I use a visualisation similar to what GitHub uses, with dots for commits and arrow-boxes for branches.
    If you’re reading this to learn git, I recommend that you do the git code school for the basics and then work through the challenges on Learn Git Branching, which will give you a way to think about how branching and commit chains work:
    This interactive branching game is a great way to learn about git branching.

    Python for Octave/Matlab users

    All of the students in the session had learned programming in their undergraduate course, but most of them had learned Octave. This means that basic ideas from programming are already understood, but that the differences between Octave and Python are more important, because they may be working with a different mental model.
    I started with the way names work in Python, explaining the idea that Python variables shouldn’t be thought of as containers, as they are in Octave, but rather like labels. I did some of the stuff interactively, but if you’re reading this now, you only really need this article, which explains it with some great visuals. I can also highly recommend the Online Python Tutor site which allows you to generate these kinds of diagrams for running code and see each step. Here’s what that looks like for one of the examples in the article
    I also covered some tips on looping, mentioning the excellent Loop like a Native talk.

    Once these gotchas were covered, I also switched from the IPython terminal to the IPython Notebook. I gave a brief overview of how everything works and also mentioned that they need to study up on all the interesting magics.

    Then it was time to pack up!

    I can see why the SC guys do this as a two-day workshop, and I think if I do this again I will encourage students to bring files from their particular projects so that they can do some actual work rather than just examples. It’s always more meaningful to learn on a problem you actually want to solve than on some contrived example.

  • Managing scientific data and interchange with industry

    I spoke at a recent symposium on the difficulties in managing data interchange with industry.

    Just for some context, “industry” here stands for personnel involved in process control and modelling activities. My direct experience is in the petrochemical, mining and paper and pulp industries. I also show an example of data from a piece of analytical equipment.

    I’ve uploaded the source on GitHub here, and you can see the notebook without downloading it by using NBViewer.

    I’ve spoken about the pattern of intermediates on this blog before (in a slightly different context).

    Notably absent from this discussion are any kind of database, since my experience is that if you can’t e-mail it, it might as well not exist when talking to the kinds of people I work with. I’ve used sqlite quite extensively myself, but I have found most people want something they can double-click and just have it work.

    If you have different experiences, I’m always eager to learn. I am specifically interested in how people who use client-server databases for their data handle the problem of interchange.