Prevent cronjobs from overlapping in Linux

Mattias Geniar, Tuesday, July 24, 2012 - last modified: Saturday, December 24, 2016

It's an unfortunate common problem on many systems: you have scheduled tasks defined in cronjobs and for some reason, they take longer to execute than anticipated. This eventually means that they start to overlap and run at the same time. If those are cronjobs that are acting on the same data from a database, it may mean data corruption.

If they're doing heavy processing of data, it could mean the server load is rising too high. Because of that high load, those cronjobs are taking longer than usual and before you know it there's a vicious circle in which cronjobs keep launching and overlapping eachother.

Obviously, you don't want this. The good news is, this is fairly easy to prevent.

Using flock

Flock is a very interesting tool for managing lock files. Those lock files are used to determine if a script or application is already running (comparable to a PID file that contains the Process ID of the running script). If the lock exists, the cronjob won't start. If the lock doesn't exist, it's safe to launch the cron.

Take the following common example, where a cron is run every minute on the server.

$ crontab -l
* * * * * /usr/bin/php /path/to/cron.php

If the script takes longer than a minute to execute, they'll begin to overlap. To prevent it, you can change it with the flock example below.

$ crontab -l
* * * * * /usr/bin/flock -w 0 /path/to/cron.lock /usr/bin/php /path/to/cron.php

The example above requires flock to manage those lock files. If it does not yet exist on your system, installation should be as simple as a yum install util-linux or apt-get install flock, depending on your Linux Distribution (see: how to find your current Linux Distribution).

The moment flock starts, it locks the lock-file you specify in the command. You can see that by requesting the user/script that is having the lock on that file.

$ fuser -v /path/to/cron.lock
                     USER        PID ACCESS COMMAND
cron.lock:           root       7836 f.... flock
                     root       7837 f.... php

It will show you the Process IDs (PIDs) of the script that is holding the lock. If no script is holding the lock, the fuser command will simply return nothing.

$ fuser -v /path/to/cron.lock

So flock is a pretty good way to prevent cronjobs from overlapping by using an extra Command Line tool.

If flock isn't installed on your system yet, install the utils package which includes flock.

$ yum install util-linux

And you're set.

Using pgrep

Another method, without using lock files, is using a rather simple bash-one liner that checks for the current running file and executes it if it's not running. The trick is to wrap your crontask in a uniquely-named bash-script, as such.

$ cat /path/to/cron.sh
#!/bin/bash
/usr/bin/php /path/to/cron.php

$ chmod +x /path/to/cron.sh

In your crontab, it should be listed as such now.

$ crontab -l
* * * * * /path/to/cron.sh

The command above will, just as the first example, execute our PHP script every minute through a bash script. To prevent it from overlapping, it can also be changed to this.

$ crontab -l
* * * * * /usr/bin/pgrep -f /path/to/cron.sh > /dev/null 2> /dev/null || /path/to/cron.sh

The pgrep command will return false if it does not find a running process matching the first argument, /path/to/cron.sh. If it returns false, it'll process the second part of the OR comparison (the double vertical line, ||). If the running process was found, pgrep will return the Process ID (PID) and Bash will not continue to the second part of the OR statement since the first already returned true.

The trick here is to use very unique scriptnames. If the name is too generic (such as "cron.sh"), pgrep may return Process IDs from other running cron jobs and not execute the cron you wanted.

Using lock-files within the script

If the examples above are not available to you, you can still use the concept of lock files in your application. One of the first commands in your script could be to check for the existance of a lock-file. If it exists, the script would simply exit(1) out of the application and stop running. If the lock-file does not exist, the script could create it and prevent the next job from executing.

As a last step in your script you remove the lock file to indicate that the script has finished and allowing the next run to continue.



Hi! My name is Mattias Geniar. I'm a Support Manager at Nucleus Hosting in Belgium, a general web geek, public speaker and podcaster. Currently working on DNS Spy. Follow me on Twitter as @mattiasgeniar.

I respect your privacy and you won't get spam. Ever.
Just a weekly newsletter about Linux and open source.

SysCast podcast

In the SysCast podcast I talk about Linux & open source projects, interview sysadmins or developers and discuss web-related technologies. A show by and for geeks!

cron.weekly newsletter

A weekly newsletter - delivered every Sunday - for Linux sysadmins and open source users. It helps keeps you informed about open source projects, Linux guides & tutorials and the latest news.

Share this post

Did you like this post? Will you help me share it on social media? Thanks!

Comments

Bokac Tuesday, February 12, 2013 at 16:58 (permalink)

Hi,
nice post. I am using this first version with flock. And also I am using cakePHP. Script is working ok, but sometimes it just releases the lock from file and the script runs again.
The third method is not very clear to me. What happens with PHP script that is already running if another instance starts? Does the script stop and starts again or what?

Cheers,
Bokac

Reply


Coder of Salvation Thursday, July 24, 2014 at 11:25 (permalink)

Thanks, the /usr/bin/flock method is *very* portable.
I just replaced many lines of bashscript by your oneliner :)

In return I would like to show you my humble wrapper:

https://gist.github.com/coderofsalvation/8356298

next to the flock-overlap protection it also restricts cron to *only* send emails when things go wrong (instead of the bulky default emailbehaviour).

Also, it starts the given command using a seperate, custom-named binary, which makes processes more readable when using utilities like ‘top’ or ‘ps’.

Hope it helps!

Reply


Dag Wieerd Wednesday, November 11, 2015 at 20:06 (permalink)

You can use flock in your script as well. It’s not only useful for a oneliner, it integrates nicely with shell-scripts and you can have multiple types of locks (read/write, wait/timeout or nonblocking) within a single script.

The manual page shows you how to do that as part of a block. But you can use { } blocks rather than a child-process ( ) block.

Reply


    Zach Jacobs Tuesday, September 19, 2017 at 17:34 (permalink)

    The pgrep solution doesn’t work for me as written in this tutorial. I had to omit the path for the first reference. The following works:

    * * * * * /usr/bin/pgrep -f cron.sh > /dev/null 2> /dev/null || /path/to/cron.sh

    Reply


Jonathan Wednesday, February 10, 2016 at 20:34 (permalink)

Does the pgrep solution actually work for you? A co-worker and I tried it on a FEW different machines and found that the pgrep method was NOT preventing simultaneous execution.

A script with the following results in a new row in “log” every minute ….

#!/bin/bash
DATE=`date`
echo $DATE >> /home/oracle/scripts/log
sleep 70

Reply


    Mattias Geniar Sunday, February 14, 2016 at 16:31 (permalink)

    Yup, works fine here, I use it on a couple of servers to prevent rsync’s from overlapping. For instance, here’s one that works for me (which syncs the Apache mirror of open source projects):

    #!/bin/bash
    pgrep -f rsync.apache.org || rsync -aqz --delete --safe-links rsync.apache.org::apache-dist /data/apache/
    

    Reply


Madushanka Sampath Friday, December 23, 2016 at 08:49 (permalink)

#yum install flock
->No package flock available.

is not working.
plz help

Reply


Mattias Geniar Saturday, December 24, 2016 at 10:25 (permalink)

Try:
$ yum install util-linux

The ‘flock’ binary is part of that util package.

Reply


Joshua Barratt Monday, January 30, 2017 at 19:41 (permalink)

A quick noteI: The version of flock that ships on at least the current version of Amazon Linux (‘flock from util-linux 2.23.2’) fails with the `-w 0` argument, ‘timeout cannot be zero’. Setting it to any non-zero value, such as ‘0.1’, works just fine.

Reply


Leave a Reply

Your email address will not be published. Required fields are marked *

Inbound links