Lecturer rant: there should be a system for marks

I use “we” there because I am part of faculty. I am amazed because of these stupid things (and these are just the ones I know of):

The university yearbooks are written and maintained by hand. Yep, if you clicked on some of those links and saw lists of repetitive seemingly machine-generated content, you may infer that this is all on a database somewhere and simply sucked in to this document by some reporting system. You would be wrong. There are people in each department tasked with maintaining the system. Changes to the document are posed in the form of a manually generated diff and applied by humans. This means there is no automated consistency checking and no way of generating the dependency tree of a particular subject to answer a question like “I have failed GFU 320 – does this cost me a year?” because the dependencies are simply listed in the document. I have written scripts that scrape the PDFs for useful information like the number of periods allocated etc to check that they are consistent with the timetable (see the next point) and generate graphs like the one below that show the dependencies of our course. Why is there not an automatically generated graph like this with clickable links into the subject descriptions on the UP website? The graph is generated using Graphviz

Unfortunately, this is not the end, because the venue bookings are handled on a totally different system which does not write back to Syllabus. This means that at any given moment it is literally impossible to reconstruct the exact timetable as it stands without talking to every lecturer involved, as the ad-hoc changes are only relayed to them and not recorded centrally beyond the subject name (not the group or other information you may need to generate the timetable). The whole situation is silly and costs everyone involved hours of their time as they sift through the book trying to verify that the information is correct. What makes it worse is that several other universities are allowing staff access to the syllabus plus system while we are using what boils down to one step above manual scheduling.

Student marks

Student marks are calculated by each lecturer using home-rolled spreadsheets that are almost guaranteed to have an error in somewhere. The rules to determine for instance if a student qualifies for a supp are subtle and often incorrectly implimented. To give you an idea, here is the logic for if a student qualifies for a supp in the school of engineering from the 2010 yearbook:

In the School of Engineering a supplementary examination is only granted in instances where:

1. A final mark of between 45% and 49% was achieved;
2. A final mark of between 40% and 44% was achieved and where the candidate also achieved either a semester mark or an examination mark of 50% or higher;
3. A pass mark has been obtained, but the required subminimum in the examination section of the module or divisions thereof has not been obtained.
4. A final mark of between 40% and 49% has been obtained in first-year modules on 100 level.

I have made a graph showing the regions for a 50% semester mark exam split:

Is it not glaringly obvious that this should at least be coded up in a reusable spreadsheet function that can be distributed to everyone? Of course, that’s just a sticking-plaster, because the system is broken in more subtle ways.

At the beginning of each course, an automatically-generated class list is sent to you by admin. It used to be a CSV file, now it is an Excel file (already my scripting has taken a knock right there). You then calculate marks in any way you want and send the marks back. Unfortunately they now send you a revised class list which does not have the same students in the same order as the previous ones. So what could have been a simple copy-and-paste now either becomes tedious lookup calculations or even more tedious manual lining up of the data. I think it is entirely obvious that we need things like the following ideas:

• Accelerated marking system you scan in the memo and the papers, define active regions on the memo and mark directly into the system. The papers have a barcode sticker from a pile you issue the students at the beginning of each year, so your (lecturer) workflow involves going into the system and clicking through the memo for each paper. Totally transparent to the student and externals, a lot faster than carting around tons of papers and shuffling through them.
• Grade calculation based on a tree structure with calculation nodes most all grade calculation can be seen as a tree-like aggregation of primary grades. Final mark is weighted average of semester and exam, each paper is sum of questions, which is sum of subquestions etc. You need a builder and navigator, with colour highlighting and live stats to answer questions like ‘why is this guy’s semester mark so low’ or ‘what will my class average look like if I adjust my semester mark’.
• Audit trails all changes should be historied (BIG fail with spreadsheet-based systems). If an adjustment is made, this should be noted for this student so that one can track the total ‘free marks’ a particular student has received. This can also be used in
• Automatic edge case detection: Students that are ‘close’ to particular boundaries like distinctions or failing or supps should be identified by the system and automatic adjustments should be suggested. This would of course be historied so that you can review how much action has been taken. With a proper system, this would include all the student’s subjects, so that the system would warn you if a student (for instance) only needs your subject to pass his whole course and has not received a lot of free mark. I have even thought a good system could be that you start your course with 0.5 percent per test (so 0.5 for semester mark, 0.5 for exam and 0.5 for final mark), so 1.5 per course. This is the adjustment we allow today anyway, as we round each of those marks. Lecturers can use some of these marks for adjustments. The system could even find optimal distribution of the free marks to maximize the chance that the student succeeds in the course. I mean, we’re only probably repeatable within about 2% to 5% on marks anyway.
• Aggregated statistics lecturers should be able to see the distributions of the other courses this group has, and be warned about strange anomalies like an otherwise good student scoring really low or an average student doing particularly well.

The thing that amazes me most is that we have all this student manpower doing projects to design websites and database systems and intelligent algorithms for fault-finding and inference, but we’re not using any of it on our own systems! I am sure that the development of a system with some of the features above would not be beyond our student body. A part of the problem is that no-one sees development of such systems as part of their job. If one person were to try to develop such a system they would end up wasting a lot of their own time, as the total loss of time is distributed among so many people. This means there should be a higher-up decision about introducing more efficiency in the system rather than a lone developer working on it. Unfortunately, the small losses aren’t accounted for in an actionable way and things keep going like they’re going. Perhaps I will should send a link to this blog post to my head of department…