How to separate your data from your code

The wrong way

You’ve started writing some code which reads from data files, and does some expensive computation which is written to some different data files. Perhaps you’re doing this in a Jupyter notebook which contains the following cell:

import pandas

input_data = pandas.read_csv('input.csv')
results_data = expensive_operation(input_data)
results_data.to_csv('results.csv')

Now, you want to put your code on GitHub, so you add Calculation.ipynb, input.csv and results.csv to the repository.

The advantage of this method is that everything is in one place and so is easy to share with other people. If we weren’t using version control, this would be a reasonable method. However, there are some problems with this method when using version control to collaborate:

• If the expensive_operation doesn’t produce exactly the same output every time, results.csv will change with every run of your code. This will clog up the change log with lots of meaningless changes.
• If the expensive_operation produces large output files, your repository will also fill up with a large number of changes (remember tools like Git store all the versions of files)
• If you use binary formats for these files, the version control tracking is less useful.

For these reasons I believe both input data and output data should be kept separately from the code repository. I propose using a file syncing service like Dropbox, Google Drive, Box, etc to handle the task of managing and storing shared data.
“But”, I can hear you ask, “why not have the code on one of these platforms as well?”. Well, because version control handles source code much better and uses delayed sync so that you can tell the system when your code is ready to be shared, while data files are usually edited with external programs and uploaded all at once.

Every solution brings a new problem

Different people have different paths

So let’s say you’ve understood this, so you move those files into your Dropbox under a folder called “Data files”, which you can share with your colleagues. Now you need to rewrite your program to read from your Dropbox folder. Your first problem is finding the right filenames to put in to the code, which involves finding the path for the files. For Windows users the path to your folder may be C:UsersPeterDropboxData files. The C:UsersPeter part is called your “home directory” or “home folder” which we’ll get back to later. For Mac users this is /Users/Peter/Dropbox/Data files and for Linux users it’s often /home/Peter/Dropbox/Data files. Let’s assume you’re using windows, so you write the code as follows:

import pandas

input_data = pandas.read_csv(r'C:UsersPeterDropboxData filesinput.csv')
results_data = expensive_operation(input_data)
results_data.to_csv(r'C:UsersPeterDropboxData filesresults.csv')

The r in front of those strings stands for “raw” and stops Python from interpreting the backslashes as escapes.
The problem here is that every user has a different username, so we can no longer just download this code and run it. Furthermore, you will notice that the different operating systems have different conventions for which character separates the parts of paths (Windows uses , Unix-like OSs use /).

Pathlib to the rescue

Both of these problems (different user directories and different separators) can be solved with the pathlib library:

import pathlib
import pandas

# find this user's Dropbox folder
datadir = pathlib.Path('~/Dropbox').expanduser()

input_data = pandas.read_csv(datadir / 'input.csv')
results_data = expensive_operation(input_data)
results_data.to_csv(datadir / 'results.csv')

Pathlib allows us to expand the ~ character into the user directory correctly on every platform. It also allows us to use the division operator to join file parts correctly. The above code will run correctly on every major operating system and work for the default settings of Dropbox. Also notice how we didn’t repeat the same folder twice in the code.

One size fits all?

But of course, not everyone accepts the default settings. Also, what if you don’t use Dropbox, but some other service which syncs folders? How can we accommodate all the differences between people’s setups?
The answer is local configuration. This requires a couple of things to be in place, but is actually quite easy. I’ll start with the improved code first:

import configparser
import pathlib
import pandas

config = configparser.ConfigParser()
config.read('config.ini')

datadir = pathlib.Path(config['paths']['datadir']).expanduser()

input_data = pandas.read_csv(datadir / 'input.csv')
results_data = expensive_operation(input_data)
results_data.to_csv(datadir / 'results.csv')

For this code to run, there needs to be a file called config.ini in the current directory when it runs. This file looks like follows:

[paths]
datadir=~/Dropbox

“But”, I hear you say again, “aren’t we back to the same problem as before with local filenames?”. The answer is “No”. Because this filename is the same for all users of the code. It’s the contents of the file which changes per user. For this system to work, you need to avoid checking in the config.ini file into your repository (you can do this by adding it to .gitignore). And since you might want to use this configuration file in a lots of different places, I recommend creating a config.py module like this:

import configparser
import pathlib

config = configparser.ConfigParser()
config.read('config.ini')
datadir = pathlib.Path(config['paths']['datadir']).expanduser()

Now you can simply do this in the parts where you need to access your data:

from config import datadir

import pandas

input_data = pandas.read_csv(datadir / 'input.csv')
results_data = expensive_operation(input_data)
results_data.to_csv(datadir / 'results.csv')

It’s not just for data

The cool thing about having this configuration file is that you can use it for any local configuration you need. Think about all the things that are in your code right now which should be local.

Decide between user-local and directory local

I’ve made a particular design choice in this solution which is to have the config file in the directory where the code is. It might suit your program better to have a configuration file in each user’s User directory or perhaps a global one. What I’ve presented here is already much better than hard-coding your paths, though.

Late edit: It turns out that there is a nice package in PyPI which can help you to avoid writing your own config file reader. It’s called decouple.