This isn't really a tutorial on cron
in general; Better people at Linux have written way better ones than I could write. Here's one:
This is more of a code journaling exercise for a problem that I didn't find a neat-and-tidy answer to online when I was looking for it, and that I presume at least one person will encounter at some point between now and the heat death of the universe.
Let's say you've got two different Conda envs: production
and development
. Let's say that, in addition to having different packages installed, they each use a separate database - development
writes to one that you can wipe and reconstruct to your heart's content, while production
gets used for stuff that actually affects customers.
Not a problem! Just a teensy bit of fiddling and you're there. From the official docs:
https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux
So, for our purposes, let's say we want to set the variables from the development
environment. First we create two folders to hold a simple .sh
script to activate
and deactivate
the relevant environment variables:
$ cd /home/matt/anaconda3/envs/development
$ mkdir -p ./etc/conda/activate.d
$ mkdir -p ./etc/conda/deactivate.d
$ touch ./etc/conda/activate.d/env_vars.sh
$ touch ./etc/conda/deactivate.d/env_vars.sh
Then let's add the info for the database we want to access. Edit ./etc/conda/activate.d/env_vars.sh
to say:
#!/bin/sh
export db-string='mysql+pymysql://dev:password@localhost/dev-db'
Now edit ./etc/conda/deactivate.d/env_vars.sh
to say:
#!/bin/sh
unset db-string
Presto! Now if you source activate development
and run a script it'll all go great!
Cron Craziness
If you're running scripts by hand, this is all great. It wouldn't even have warranted a blog post! However, a little hiccup occurs if you try to have cron
(or something that wraps cron
- I was using the cool Ruby gem whenever
because that was what was running stuff on the job server my stuff was running on) run the script.
I naively tried to have cron
activate the env, run it, then deactivate it.
source activate production; python /home/matta/python_workspace/dbUpdater.py; source deactivate
This did not work. Research lead me to believe the reason involved rules about spawning subshells
, a concept I vaguely understand. What I understood more concretely, however, was that this was not going to work.
Run It With The Packages You Want
Eventually I discovered that I at least had access to the environment's packages if, instead of running the script by calling python
, I ran it by calling the python
in the env itself. Sooo...
/home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py
Sadly, we still have a problem - it's not loading our environment variables! This makes sense - Python doesn't know where those variables are, they get exported when we call source activate development
.
Sourcing the Env Variables
Buuut, since exporting the variables just happens in a .sh
script, we can run it directly!
source /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py
Runs in the terminal! Should run in the cron
, right?
Well, in the moral sense it "should" - unfortunately, it does not.
It turns out that source
is actually an alias for a command whose True Name is .
. For reasons I will not pretend to understand, cron
does not know this, and will not be taught. It is we who shall have to adjust to the machine's preferences.
Putting It All Together
The command we want is:
. /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py
Let's say we want it to run every day at 3AM UTC:
* 3 * * * . /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py
And to add it directly to your crontab
crontab -l | { /bin/cat; /bin/echo "* 3 * * * . /home/matt/anaconda3/envs/development/etc/conda/activate.d/env_vars.sh ; /home/matt/anaconda3/envs/development/bin/python /home/matta/python_workspace/dbUpdater.py"; } | crontab -