Getting Conda Envs (And Environment Variables!) To Play Nicely With Cron

Code Snippet Corner

This isn't really a tutorial on cron in general; Better people at Linux have written way better ones than I could write.  Here's one: http://mediatemple.net/blog/news/complete-beginners-guide-cron-part-1/  This is more of a code journaling exercise for a problem that I didn't find a neat-and-tidy answer to online when I was looking for it, and that I presume at least one person will encounter at some point between now and the heat death of the universe.

Let's say you've got two different conda envs:  production and development.  Let's say that, in addition to having different packages installed, they each use a seperate database - development writes to one that you can wipe and reconstruct to your heart's content, while production gets used for stuff that actually affects customers.

Not a problem!  Just a teensy bit of fiddling and you're there.  From the official docs: https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux

So, for our purposes, let's say we want to set the variables from the development environment.  First we create two folders to hold a simple .sh script to  activate and deactivate the relevant environment variables:

cd /home/matt/anaconda3/envs/development
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Then let's add the info for the database we want to access.  Edit ./etc/conda/activate.d/env_vars.sh to say:

#!/bin/sh

export db-string='mysql+pymysql://dev:[email protected]/dev-db'

Now edit ./etc/conda/deactivate.d/env_vars.sh to say:

#!/bin/sh

unset db-string

Presto!  Now if you source activate development and run a script it'll all go great!

Cron Craziness

If you're running scripts by hand, this is all great.  It wouldn't even have warranted a blog post!  However, a little hiccup occurs if you try to have cron (or something that wraps cron - I was using the cool Ruby gem whenever because that was what was running stuff on the job server my stuff was running on) run the script.

I naively tried to have cron activate the env, run it, then deactivate it.

source activate production; python /home/matta/python_workspace/dbUpdater.py; source deactivate

This did not work.  Research lead me to believe the reason involved rules about spawning subshells, a concept I vaguely understand.  What I understood more concretely, however, was that this was not going to work.

Run It With The Packages You Want

Eventually I discovered that I at least had access to the environment's packages if, instead of running the script by calling python, I ran it by calling the python in the env itself.  Sooo...

/home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py

Sadly, we still have a problem - it's not loading our environment variables!  This makes sense - Python doesn't know where those variables are, they get exported when we call source activate development.

Sourcing the Env Variables

Buuut, since exporting the variables just happens in a .sh script, we can run it directly!

source /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py

Runs in the terminal!  Should run in the cron, right?

Well, in the moral sense it "should" - unfortunately, it does not.

It turns out that source is actually an alias for a command whose True Name is ..  For reasons I will not pretend to understand, cron does not know this, and will not be taught.  It is we who shall have to adjust to the machine's preferences.

Putting It All Together

The command we want is

. /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py

Let's say we want it to run every day at 3AM UTC:

* 3 * * * . /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py

And to add it directly to your crontab

crontab -l | { /bin/cat; /bin/echo "* 3 * * * . /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py"; } | crontab -
Author image
Center of the Universe Website
Super villain in somebody's action hero movie. Experienced a radioactive freak accident at a young age, which rendered him part-snake and strangely adept at Python.
Author image
Center of the Universe

Super villain in somebody's action hero movie. Experienced a radioactive freak accident at a young age, which rendered him part-snake and strangely adept at Python.