Celery 4 tasks with very long timeout
Delaying tasks is not obvious and as always when Celery comes in we must take care about few things.
This article is about Celery 4.0 and 4.1. If you come from the future, this may also apply to you.
Countdown
First and the easiest way for task delaying is to use countdown
argument. Countdown takes Int and stands for the delay time expressed in seconds.
my_task.apply_async(countdown=10)
Pros
- easy to use
- readable
Cons
- doesn’t work when
enable_utc
is set to False and timezone is defined
Estimated Time of Arrival
Second way is to use eta
argument, which takes exact date and time of execution. Works perfectly with native datetime object, date as String or even Pendulum instance.
my_task.apply_async(eta=datetime.now(pytz.timezone("Europe/Warsaw"))
my_task.apply_async(eta="2018-02-19 13:41:14+01:00")
my_task.apply_async(eta=pendulum.now("Europe/Warsaw"))
Pros
- readable
- precise
- works with timezones
Cons
- require more work
Visibility timeout
Sometimes there is a need for very long timeout for example 8 hours or more. For such long timeouts Celery require additional configuration.
app = Celery("project_name", broker="redis://localhost:6379", backend="redis://localhost:6379")
max_timeout_in_seconds = 21600 # 6h this is arbitrary
app.conf.broker_transport_options = {"visibility_timeout": max_timeout_in_seconds}
Without visibility_timeout
, tasks with very long timeout may be dropped or will be executed multiple times. Once or twice for every active worker.