Transaction timeout in InboxCleanupWork
When running calendar server 7.0 on a system with many calendar homes, error messages are logged from the worker system: 2016-02-02 14:18:44+0000 [-] [caldav-1] [-] [txdav.common.datastore.sql#error] Transaction abort too long: PG-TXN<ultimatelyPerform: 4041391 <INBOX_CLEANUP_WORK>>, Statements: 214963, IUDs: 0, Statement: insert into JOB (NOT_BEFORE, PRIORITY, WEIGHT, WORK_TYPE) values (%s, %s, %s, %s) returning FAILED, JOB_ID, PAUSE, OVERDUE, ASSIGNED 2016-02-02 14:18:44+0000 [-] [caldav-1] [-] [twext.enterprise.jobs.jobitem#error] JobItem: 4041391, WorkItem: 4041374 failed: [Failure instance: Traceback: <class 'twext.enterprise.ienterpri se.AlreadyFinishedError'>: ... The problem seems to be that InboxCleanupWork within one single transaction tries to schedule a CleanupOneInboxWork for each calendar home. Any suggestions for how we can get around this problem? Best regards, Kristoffer Møllerhøj.
On Feb 2, 2016, at 11:22 PM, Kristoffer <kmo@b-one.net> wrote:
When running calendar server 7.0 on a system with many calendar homes, error messages are logged from the worker system:
2016-02-02 14:18:44+0000 [-] [caldav-1] [-] [txdav.common.datastore.sql#error] Transaction abort too long: PG-TXN<ultimatelyPerform: 4041391 <INBOX_CLEANUP_WORK>>, Statements: 214963, IUDs: 0, Statement: insert into JOB (NOT_BEFORE, PRIORITY, WEIGHT, WORK_TYPE) values (%s, %s, %s, %s) returning FAILED, JOB_ID, PAUSE, OVERDUE, ASSIGNED 2016-02-02 14:18:44+0000 [-] [caldav-1] [-] [twext.enterprise.jobs.jobitem#error] JobItem: 4041391, WorkItem: 4041374 failed: [Failure instance: Traceback: <class 'twext.enterprise.ienterpri se.AlreadyFinishedError'>: ...
The problem seems to be that InboxCleanupWork within one single transaction tries to schedule a CleanupOneInboxWork for each calendar home.
Any suggestions for how we can get around this problem?
I don't believe we've seen this one yet... If that's what is really happening, the only options I can think of are to enhance the scheduling of that work to happen in reasonably sized chunks, or else bump up the transaction timeout (TransactionTimeoutSeconds). Roughly how many accounts are there? -dre
On Feb 2, 2016, at 11:22 PM, Kristoffer <kmo@b-one.net> wrote:
When running calendar server 7.0 on a system with many calendar homes, error messages are logged from the worker system:
2016-02-02 14:18:44+0000 [-] [caldav-1] [-] [txdav.common.datastore.sql#error] Transaction abort too long: PG-TXN<ultimatelyPerform: 4041391 <INBOX_CLEANUP_WORK>>, Statements: 214963, IUDs: 0, Statement: insert into JOB (NOT_BEFORE, PRIORITY, WEIGHT, WORK_TYPE) values (%s, %s, %s, %s) returning FAILED, JOB_ID, PAUSE, OVERDUE, ASSIGNED 2016-02-02 14:18:44+0000 [-] [caldav-1] [-] [twext.enterprise.jobs.jobitem#error] JobItem: 4041391, WorkItem: 4041374 failed: [Failure instance: Traceback: <class 'twext.enterprise.ienterpri se.AlreadyFinishedError'>: ...
The problem seems to be that InboxCleanupWork within one single transaction tries to schedule a CleanupOneInboxWork for each calendar home.
Any suggestions for how we can get around this problem?
For now, to stop the errors, you can disable the top-level key InboxCleanup <http://trac.calendarserver.org/browser/CalendarServer/trunk/conf/caldavd-stdconfig.plist#L1055> in your caldavd.plist (add this key and set it to enabled: false). This will turn off the InboxCleanupWork scheduler. As the name suggest, this work deletes old / stale items from user inboxes. It is a relatively new feature, and is not required for correct server operation in the short to medium term. Over the long term, it's a useful element of a data retention policy with the goal of steady-state operation over time, instead of accruing data indefinitely. Other elements to such a policy might include RevisionCleanup <http://trac.calendarserver.org/browser/CalendarServer/trunk/conf/caldavd-stdconfig.plist#L1041> and purging of data for deleted users (both of these behaviors are automatic and enabled by default), and the various 'purge <http://trac.calendarserver.org/browser/CalendarServer/trunk/calendarserver/tools/purge.py>' command line tools that can be run manually: calendarserver_purge_attachments calendarserver_purge_principals calendarserver_purge_events -dre
participants (2)
-
Andre LaBranche
-
Kristoffer