You’ve started writing some code which reads from data files, and does some expensive computation which is written to some different data files. Perhaps you’re doing this in a Jupyter notebook which contains the following cell:<\/p>\n
import<\/span> pandas\n\ninput_data =<\/span> pandas.read_csv('input.csv')\nresults_data =<\/span> expensive_operation(input_data)\nresults_data.to_csv('results.csv')<\/code><\/pre>\n\n<\/code><\/pre>\n<\/div>\nNow, you want to put your code on GitHub, so you add Calculation.ipynb<\/code>, input.csv<\/code> and results.csv<\/code> to the repository.<\/p>\nThe advantage of this method is that everything is in one place and so is easy to share with other people. If we weren’t using version control, this would be a reasonable method. However, there are some problems with this method when using version control to collaborate:<\/p>\n
\n- If the
expensive_operation<\/code> doesn’t produce exactly<\/em> the same output every time, results.csv<\/code> will change with every run of your code. This will clog up the change log with lots of meaningless changes.<\/li>\n- If the
expensive_operation<\/code> produces large output files, your repository will also fill up with a large number of changes (remember tools like Git store all the versions of files)<\/li>\n- If you use binary formats for these files, the version control tracking is less useful.<\/li>\n<\/ul>\n
For these reasons I believe both input data and output data should be kept separately from the code repository. I propose using a file syncing service like Dropbox, Google Drive, Box, etc to handle the task of managing and storing shared data.
\n\u201cBut\u201d, I can hear you ask, \u201cwhy not have the code on one of these platforms as well?\u201d. Well, because version control handles source code much better and uses delayed sync so that you can tell the system when your code is ready to be shared, while data files are usually edited with external programs and uploaded all at once.<\/p>\n
\nEvery solution brings a new problem<\/h2>\n\nDifferent people have different paths<\/h3>\n
So let’s say you’ve understood this, so you move those files into your Dropbox under a folder called \u201cData files\u201d, which you can share with your colleagues. Now you need to rewrite your program to read from your Dropbox folder. Your first problem is finding the right filenames to put in to the code, which involves finding the path for the files. For Windows users the path to your folder may be C:UsersPeterDropboxData files<\/code>. The C:UsersPeter<\/code> part is called your \u201chome directory\u201d or \u201chome folder\u201d which we’ll get back to later. For Mac users this is \/Users\/Peter\/Dropbox\/Data files<\/code> and for Linux users it’s often \/home\/Peter\/Dropbox\/Data files<\/code>. Let’s assume you’re using windows, so you write the code as follows:<\/p>\n\nimport<\/span> pandas\n\ninput_data =<\/span> pandas.read_csv(r'C:<\/span>Users<\/span>Peter<\/span>Dropbox<\/span>Data files<\/span>input<\/span>.csv')\nresults_data =<\/span> expensive_operation(input_data)\nresults_data.to_csv(r'C:<\/span>Users<\/span>Peter<\/span>Dropbox<\/span>Data files<\/span>results.csv')<\/code><\/pre>\n<\/div>\n\nThe r<\/code> in front of those strings stands for \u201craw\u201d and stops Python from interpreting the backslashes as escapes.
\nThe problem here is that every user has a different username, so we can no longer just download this code and run it. Furthermore, you will notice that the different operating systems have different conventions for which character separates the parts of paths (Windows uses <\/code>, Unix-like OSs use \/<\/code>).<\/p>\n\nPathlib to the rescue<\/h3>\n
Both of these problems (different user directories and different separators) can be solved with the pathlib<\/code> library:<\/p>\n\nimport<\/span> pathlib\nimport<\/span> pandas\n\n# find this user's Dropbox folder<\/span>\ndatadir =<\/span> pathlib.Path('~\/<\/span>Dropbox').expanduser()\n\ninput_data =<\/span> pandas.read_csv(datadir \/<\/span> 'input.csv')\nresults_data =<\/span> expensive_operation(input_data)\nresults_data.to_csv(datadir \/<\/span> 'results.csv')<\/code><\/pre>\n<\/div>\n\nPathlib allows us to expand the ~<\/code> character into the user directory correctly on every platform. It also allows us to use the division operator to join file parts correctly. The above code will run correctly on every major operating system and work for the default settings of Dropbox. Also notice how we didn’t repeat the same folder twice in the code.<\/p>\n\nOne size fits all?<\/h2>\n
But of course, not everyone accepts the default settings. Also, what if you don’t use Dropbox, but some other service which syncs folders? How can we accommodate all the differences between people’s setups?
\nThe answer is local configuration. This requires a couple of things to be in place, but is actually quite easy. I’ll start with the improved code first:<\/p>\n
\nimport<\/span> configparser\nimport<\/span> pathlib\nimport<\/span> pandas\n\nconfig =<\/span> configparser.ConfigParser()\nconfig.read('config.ini')\n\ndatadir =<\/span> pathlib.Path(config['paths']['datadir']).expanduser()\n\ninput_data =<\/span> pandas.read_csv(datadir \/<\/span> 'input.csv')\nresults_data =<\/span> expensive_operation(input_data)\nresults_data.to_csv(datadir \/<\/span> 'results.csv')<\/code><\/pre>\n<\/div>\n\nFor this code to run, there needs to be a file called config.ini<\/code> in the current directory when it runs. This file looks like follows:<\/p>\n\n\n<\/span><\/code><\/pre>\n[paths]<\/span>\ndatadir<\/span>=<\/span>~\/Dropbox<\/span><\/code><\/pre>\n<\/div>\n\n\u201cBut\u201d, I hear you say again, \u201caren’t we back to the same problem as before with local filenames?\u201d. The answer is \u201cNo\u201d. Because this filename is the same for all users of the code. It’s the contents<\/em> of the file which changes per user. For this system to work, you need to avoid checking in the config.ini<\/code> file into your repository (you can do this by adding it to .gitignore<\/code>). And since you might want to use this configuration file in a lots of different places, I recommend creating a config.py<\/code> module like this:<\/p>\n\nimport<\/span> configparser\nimport<\/span> pathlib\n\nconfig =<\/span> configparser.ConfigParser()\nconfig.read('config.ini')\ndatadir =<\/span> pathlib.Path(config['paths']['datadir']).expanduser()<\/code><\/pre>\n<\/div>\n\nNow you can simply do this in the parts where you need to access your data:<\/p>\n
\n\n<\/span><\/code><\/pre>\nfrom<\/span> config import<\/span> datadir<\/code><\/pre>\nimport<\/span> pandas\n<\/code><\/pre>\n\ninput_data =<\/span> pandas.read_csv(datadir \/<\/span> 'input.csv')\nresults_data =<\/span> expensive_operation(input_data)\nresults_data.to_csv(datadir \/<\/span> 'results.csv')<\/code><\/pre>\n<\/div>\n\nIt’s not just for data<\/h2>\n
The cool thing about having this configuration file is that you can use it for any local configuration you need. Think about all the things that are in your code right now which should be local.<\/p>\n
\nDecide between user-local and directory local<\/h2>\n
I’ve made a particular design choice in this solution which is to have the config file in the directory where the code is. It might suit your program better to have a configuration file in each user’s User directory or perhaps a global one. What I’ve presented here is already much better than hard-coding your paths, though.<\/p>\n