Negative Feedback

Year: 2018

  • Gautrain API reverse engineering

    Gautrain API reverse engineering

    The iOS 12 update contained a very cool feature: Siri Shortcuts. This replaces the Workflows app. I was using Workflows to send my wife an SMS with my estimated time of arrival at home when I left work. With the new Shortcuts app I could have this all happen automatically, and it also supports reading from Web APIs.
    The first part, however, is to figure out how to get data from the Gautrain website.

    mitmproxy

    mitmproxy or Man In The Middle Proxy is a piece of software you can run on your computer which will stand between a device and the internet and allow you to inspect what is going on. I might cover installation in another post, but basically I just followed instructions to get it going and then fired up the Gautrain app to see what it was doing.

    Sequence of events with Gautrain App

    First endpoint to check service is live

    https://api.gautrain.co.za/user-service/api/0/mobile/isLive/1.0.0

    Returns true or false.
    The app then hits

    https://api.gautrain.co.za/transport-api/api/0/agencies

    And gets a whole list of agencies, including this one:

    [
....
{
        "culture": "en",
        "description": null,
        "href": "https://platform.whereismytransport.com/api/agencies/edObkk6o-0WN3tNZBLqKPg",
        "id": "edObkk6o-0WN3tNZBLqKPg",
        "name": "Gautrain"
},
...
]

    The important part here is "name": "Gautrain" and that id which is used later.
    Also interesting is that they are clearly using whereismytransport.com, which is the first time I’ve heard of this service.
    The app then asks ask for a list of stops:

    https://api.gautrain.co.za/transport-api/api/0/stops/gautrain

    The response is a list of stops like this one:

    [
...
{
    "agency": {
        "culture": "en",
        "description": null,
        "href": "https://platform.whereismytransport.com/api/agencies/edObkk6o-0WN3tNZBLqKPg",
        "id": "edObkk6o-0WN3tNZBLqKPg",
        "name": "Gautrain"
    },
    "code": null,
    "geometry": {
        "coordinates": [
            28.23794,
            -25.74762
        ],
        "type": "Point"
    },
    "href": "https://platform.whereismytransport.com/api/stops/_rkqSHvRE0Scvbcsuy0EVw",
    "id": "_rkqSHvRE0Scvbcsuy0EVw",
    "modes": [
        "Rail",
        "Bus"
    ],
    "name": "Hatfield"
},
...
    {
    "agency": {
        "culture": "en",
        "description": null,
        "href": "https://platform.whereismytransport.com/api/agencies/edObkk6o-0WN3tNZBLqKPg",
        "id": "edObkk6o-0WN3tNZBLqKPg",
        "name": "Gautrain"
    },
    "code": null,
    "geometry": {
        "coordinates": [
            28.05693,
            -26.10858
        ],
        "type": "Point"
    },
    "href": "https://platform.whereismytransport.com/api/stops/jXU-OlvxukW8wfc7JeVeXw",
    "id": "jXU-OlvxukW8wfc7JeVeXw",
    "modes": [
        "Rail",
        "Bus"
    ],
    "name": "Sandton"
}
...
]

    I’ve kept the important ones for me – I’m going from Hatfield to Sandton and back, so I need those coordinates.
    Right, so the next thing they hit is

    https://api.gautrain.co.za/transport-api/api/0/journey/create

    This is the first request which has parameters. You can see that they are using the coordinates for Hatfield station and Sandton station as the start and end points.

    {
    "geometry": {
        "coordinates": [
            [
                28.23794,
                -25.74762
            ],
            [
                28.05693,
                -26.10858
            ]
        ],
        "type": "MultiPoint"
    },
    "maxItineraries": 5,
    "omit": {
        "agencies": [
            "A1JHSPIg_kWV5XRHIepCLw",
            "CA3o3RuuGUmoDKfyAPNTsg",
            "NfBxKfzMA0exbwToc-7o2g",
            "XxDwxnin_Eu_T7_wBJIJRA",
            "Hwe1673sC0yuT6fyANnDZQ",
            "DcXaTl-5hkGRHKenAIi_5w",
            "WaasqFZwEEa5VqbIAJKsJQ",
            "wZfSY3o0LUGHiqeoAQDeIQ",
            "W7A63uTwIECgQvU9MxcCIw",
            "Ka2d4V1g3E65VqheANhvVw",
            "FTGTH38Tm0C5O6gAAQifSQ"
        ],
        "modes": []
    },
    "only": {
        "agencies": [
            "edObkk6o-0WN3tNZBLqKPg"
        ],
        "modes": []
    },
    "profile": "ClosestToTime",
    "time": null,
    "timeType": "DepartAfter"
}

    So that list of agencies to omit is obviously from the original request. I’m hoping I don’t need that. For my purposes I just need to know when the next train will arrive and how long it will take to get to the destination.
    The response from this request is quite huge, but the first bit shows the important parts:

    {
    "agencies": null,
    "earliestDepartureTime": null,
    "geometry": {
        "coordinates": [
            [
                28.23794,
                -25.74762
            ],
            [
                28.05693,
                -26.10858
            ]
        ],
        "type": "MultiPoint"
    },
    "href": "https://platform.whereismytransport.com/api/journeys/ZPtEXRtgeEq1mqljAE4new",
    "id": "ZPtEXRtgeEq1mqljAE4new",
    "itineraries": [
        {
            "arrivalTime": "2018-09-22T05:29:52Z",
            "departureTime": "2018-09-22T04:56:20Z",
            "distance": {
                "unit": "m",
                "value": 51662
            },
            "duration": 2012,
    ...
    Lots more output

    So when is the next train?

    I’m only really interested in when the next train will arrive at Sandton and I don’t want to type all that stuff into a mobile app, I want a minimal version.

     Further investigation yields the simplified request:

    {"geometry":
 {"coordinates":
  [[28.23794,-25.74762],
  [28.05693,-26.10858]],
  "type":"MultiPoint"},
  "profile":"ClosestToTime",
  "maxItineraries"5,
  "timeType":"DepartAfter",
  "time":null,
  "only":{"agencies":["edObkk6o-0WN3tNZBLqKPg"],
  "modes":[]}}

    This is something I can put into the automation app eally easily. More info on that to follow.
    For posterity, this is that call being checked via curl:

    curl -d '{"geometry":{"coordinates":[[28.23794,-25.74762],[28.05693,-26.10858]],"type":"MultiPoint"},"profile":"ClosestToTime","maxItineraries":5,"timeType":"DepartAfter","only":{"agencies":["edObkk6o-0WN3tNZBLqKPg"]}}' -H "Content-Type: application/json" -X POST https://api.gautrain.co.za/transport-api/api/0/journey/create

    Guess it’s also time to create an account at whereismytransport.com!

  • Case study: R (tidyverse) vs Python (pandas)

    Case study: R (tidyverse) vs Python (pandas)

    The problem

    An old friend e-mailed me a challenge today. He was working with a data file which contains measurements of compounds from groundwater taps. The file looks like this (when opened in a spreadsheet program):

    In other words, it contains measurements for a particular date for a particular tap point and compound.

    He was using the following R code to plot how the Iron levels varied over time at different sampling points:

    library(tidyverse)

ggplot(data) +
  geom_line(mapping = aes(x=Date,y=Tap4.Fe,color="red")) +
  geom_line(mapping = aes(x=Date,y=Tap5.Fe,color="blue")) +
  geom_line(mapping = aes(x=Date,y=Tap6.Fe,color="purple")) +
  geom_line(mapping = aes(x=Date,y=Tap7.Fe,color="green")) +
  geom_line(mapping = aes(x=Date,y=Tap8.Fe,color="orange")) +
  geom_line(mapping = aes(x=Date,y=Tap9.Fe,color="brown"))
    

    

    But he could sense that this was not an optimal solution to the problem and wanted some advice on getting this to work better.
    

    
Tidy data
    

    
My first comment to him was that this data is not in the format which the tidyverse wants. Hadley Wickham appears to have revitalised the R language with his clever packages and wrote a manifesto of sorts to the benefits of tidy data, which I feel is really just a no-nonsense approach to data normalisation. There are many things wrong with the data in its current form, but I see spreadsheets like this quite often.
    

    

    

    
The problem is that people want an easy way of entering data and often end up focusing on ease of entry instead of ease of analysis. It is easy to understand how this format grows by adding more readings as extra columns on the spreadsheet and having one date per row. 
    

    

    

    
From a normalisation perspective, the problems with this format are that there is data in the column name: Tap4.Fe is actually encoding two fields – a position and a compound. Our first job is therefore to tidy up this data.
    

    

    

    
The tidyverse supplies a lot of useful functions to do just that. Here is how I tidied the data:
    

    

    

    

    
measurements %
    

    
  gather(measurement, value, -Date) %>% 
    

    
  separate(measurement, c(“position”, “compound”), sep=”\.”) %>%
    

    
  drop_na()
    

    

    

    

    
Now measurements is much “taller” and less wide:
    

    

    

    

    

    

    
The key points here are that we have a single row for a single observation. This is the essence of the tidy data philosophy and it makes downstream processing very easy.
    

    Here is how I recreated his plots:
    

    filter(measurements, compound == “Fe”) %>% 
    
  ggplot + 
    
  aes(x = Date, y = value, color = position) + geom_line()
    

    
That’s really nice. I can filter to find the Iron measurements and I can have ggplot handle the colours automatically by looking at the position.
    

    The resulting plot looks like this (this is just with all default settings):
    

    

    

    

    
Python
    

    
I was very happy with how succinct the R version of the solution was, but I use Python more than R, so I was interested to see how Pandas handles this.
    

    

    

    
The “tidyfication” of the data requires a lot more ceremony in Pandas:
    

    

    

    

    
df = (pandas.read_csv(‘data.csv’, parse_dates=[‘Date’])
    

    
      .melt(id_vars=’Date’)
    

    
      .dropna())
    

    
df[[‘position’, ‘compound’]] = df.variable.str.split(‘.’, expand=True)
    

    
del df[‘variable’]
    

    

    

    

    
I’m especially critical of how hard it is to replace a compound column with its split version (the last two lines). Wickham clearly understands that this is a common problem in data files, where the Pandas version feels “hacky”.
    

    

    

    
Now the code to recreate the graphics:
    

    

    

    

    
df.query(‘compound==”Fe”‘).pivot(index=’Date’, columns=’position’, values=’value’).plot()
    

    

    

    

    

    

    

    

    
Again, I am using the default settings. Although that might look a bit simpler than the R code, it took me ages to write. There is a bit of counter-intuitive (to my mind) pivoting so that I can use the built-in Pandas plot command and I have much less flexibility about the mapping.
    

    

    

    
I suppose the take-home here is that tidying your data is a win regardless of your platform, but that R and the tidyverse is really a force to be reckoned with. I’m hoping the Pandas API will end up providing some of the powerful verbs from tidyr et al eventually.