{"id":304,"date":"2026-07-08T08:42:57","date_gmt":"2026-07-08T06:42:57","guid":{"rendered":"https:\/\/sandrock.co.za\/carl\/?p=304"},"modified":"2026-07-08T08:42:57","modified_gmt":"2026-07-08T06:42:57","slug":"the-crime-in-south-africa-wikipedia-page","status":"publish","type":"post","link":"https:\/\/sandrock.co.za\/carl\/2026\/07\/the-crime-in-south-africa-wikipedia-page\/","title":{"rendered":"The Crime in South Africa Wikipedia page"},"content":{"rendered":"\n

I guess it says something about me that I have a folder on my mac called “pointproving”. One of the things that exists here is my consolidation of the murder rate in South Africa.<\/p>\n\n\n\n

This work stems from another saying of mine. You can never say “Wikipedia is wrong”, you should always say “Wikipedia was wrong”, because anyone can correct errors on Wikipedia. And when I stepped in to the Crime in South Africa<\/a> Wikipedia page, I was sure there were conflations of the murder rate (murders per 100k people, per year) and the simple number of murders per day. These numbers are in similar orders of magnitude for South Africa (83 murders per day and 43 murders per 100k per year respectively for the last reporting period), and often people will choose the larger or smaller side to make a point.<\/p>\n\n\n\n

I’m particularly interested in the murder rate specifically as it is the least easily gamed crime statistic. <\/p>\n\n\n\n

Data sources<\/h1>\n\n\n\n

Solving this issue has involved a lot of reading and researching through the various sources of statistics. To get murders per population I clearly need the number of murders and the population. I therefore have two main sources of data.<\/p>\n\n\n\n

The SAPS crime statistics page<\/a>, which has improved a lot since I started doing this in 2019, but has also moved from a single consolidated annual report to quarterly reports and started incorporating their own population estimates in their crime stats spreadsheets. These spreadsheets are not delivered in a format that is easily machine-readable and are slightly different for every year.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The Stats SA publication code “P0302”: Mid-year population estimates, on years that aren’t covered by the census and the census data on those that are. In typical Stats SA fashion, there is no stable url that just lists all the individual year’s versions of this publication, you just have to search for P0302 on the search page. At least the lookup code is stable. Again, the data is not in easily machine readable format, but rather a long pdf with narrative and tables.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

What I have done<\/h1>\n\n\n\n

When I arrived on the Crime in South Africa page, the murder stats were in one big table, and there was a poorly copied long-term stats chart that didn’t connect to the “current” stats that started in 1994. I digitised the chart, ingested the table and made one single chart for the long term. <\/p>\n\n\n\n

\"\"
Long-term murder rate in South Africa<\/figcaption><\/figure>\n\n\n\n

I also produced a nice per-province chart that I’m quite happy with.<\/p>\n\n\n\n

\"\"
Murder rate in South Africa by province<\/figcaption><\/figure>\n\n\n\n

Since 2019, I’ve been updating the page once a year, when the new stats drop from SAPS.<\/p>\n\n\n\n

This was all done “manually”, typing code with my fingers like an animal. I also had to find and incorporate each new version of the crime stats spreadsheets and population estimates laboriously writing new parsers.<\/p>\n\n\n\n

LLM helpers<\/h1>\n\n\n\n

For this round of updates I’ve had LLM help and I’ve now got a fully automated pipeline from the spreadsheets to the wiki page. <\/p>\n\n\n\n

This involved creating a bot user on Wikipedia. The whole data set is also now in a sqlite database instead of in several manually-updated spreadsheets and Jupyter notebooks.<\/p>\n\n\n\n

The next round is going to be much less painful and with fewer opportunities for human error.<\/p>\n","protected":false},"excerpt":{"rendered":"

I guess it says something about me that I have a folder on my mac called “pointproving”. One of the things that exists here is my consolidation of the murder rate in South Africa. This work stems from another saying of mine. You can never say “Wikipedia is wrong”, you should always say “Wikipedia was […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-304","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/posts\/304","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/comments?post=304"}],"version-history":[{"count":1,"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/posts\/304\/revisions"}],"predecessor-version":[{"id":311,"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/posts\/304\/revisions\/311"}],"wp:attachment":[{"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/media?parent=304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/categories?post=304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sandrock.co.za\/carl\/wp-json\/wp\/v2\/tags?post=304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}