<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">There is some recent work that hopes to avoid a deadlock situation we have encountered recently in CalendarServer's work queue (twext.enterprise.jobs) by using the <a href="http://michael.otacoo.com/postgresql-2/postgres-9-5-feature-highlight-skip-locked-row-level/" class="">skip-locked</a> feature in Postgres >= 9.5.</div><div class=""><br class=""></div><div class=""><a href="http://trac.calendarserver.org/changeset/15735/CalendarServer/trunk" class="">http://trac.calendarserver.org/changeset/15735/CalendarServer/trunk</a> - if you take this, you might as well go all the way top latest for a small handful of other fixes and enhancements.</div><div class=""><br class=""></div><div class="">For CS in 'dev' mode, this may require nuking the build root to pull down the newer version automatically. Also, when using Postgres older than 9.5, take special note of this portion of conf/caldavd-stdconfig.plist - this plist is composed of the defaults for all parameters. With older postgres, you need to define DBFeatures with an empty array.</div><div class=""><br class=""></div><div class=""> <!-- Features supported by the database<br class=""><br class=""> 'skip-locked': SKIP LOCKED available with SELECT (<font color="#ff2600" class="">remove if using postgres<br class=""> &lt; v9.5</font>) --><br class=""> <key>DBFeatures</key><br class=""> <array><br class=""> <string>skip-locked</string><br class=""> </array></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">If you don't take the above fix in the short term, as a bandaid you can enable the transaction timeout, set to some reasonably high-ish value like 10 minutes. This would go in DatabaseConnection:txnTimeoutSeconds, for example:</div><div class=""><br class=""></div><div class=""><div class=""> <key>DatabaseConnection</key></div><div class=""> <dict></div><div class=""> <key>endpoint</key></div><div class=""> <string>tcp:localhost</string></div><div class=""> <key>database</key></div><div class=""> <string>caldav</string></div><div class=""> <key>user</key></div><div class=""> <string>caldav</string></div><div class=""> <key>password</key></div><div class=""> <string></string></div><div class=""> <!-- needed when DBType = postgres --></div><div class=""> <key>txnTimeoutSeconds</key></div><div class=""> <integer>600</integer></div><div class=""> </dict></div></div><div class=""><br class=""></div><div class="">Attentive readers may note that DatabaseConnection:txnTimeoutSeconds is not mentioned in either twistedcaldavd/stdconfig.py or conf/caldavd-stdconfig.plist. This sort of thing loosely falls under the category of "params we pass to someone else's code", and we make no effort to expose in our config all features of other software we use. However, this one probably deserves a mention, so I'll add that.</div><div class=""><br class=""></div><div class="">We might want additional detail about the error you are hitting, as I can't say for sure if your problem is the one addressed by r15735. The authority on this part of CS is currently on holiday, so I will get back to you if we need more info. If you just want to throw a wide net to collect something before trying to get out of the problem state, you could simply turn up DefaultLogLevel to debug and reproduce the problem, then save off the log - this may only be practical if the error reproduces fairly easily / quickly, otherwise the log grows rather large!</div><div class=""><br class=""></div><div class="">-dre</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 23, 2016, at 4:06 AM, Axel Rau <<a href="mailto:Axel.Rau@chaos1.de" class="">Axel.Rau@Chaos1.DE</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi,<br class=""><br class="">This error is reproducable on my development server:<br class="">- - -<br class="">2016-07-23T10:07:08+0000 [caldav-0] [calendarserver.tools.purge#warn] Cleaning up future events for principal 54371764-438E-4D2A-8E25-A1A15E8CB14B since they are no longer in directory<br class="">2016-07-23T10:07:40+0000 [caldav-0] [twext.enterprise.jobs.jobitem#error] JobItem: 4699, WorkItem: 74300 failed: [Failure instance: Traceback: <class 'pg8000.core.ProgrammingError'>: (u'ERROR', u'57014', u'canceling statement due to statement timeout', u'while locking tuple (456,4) in relation "calendar_object"\nSQL statement "SELECT 1 FROM ONLY "caldavd"."calendar_object" x WHERE "resource_id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x“', u'postgres.c', u'2967', u'ProcessInterrupts', u'', u'')<br class="">- - -<br class="">Shall I try upgrading to 8.1 or could it help to collect more info on 8.0?<br class=""><br class="">Thanks, Axel<br class="">---<br class="">PGP-Key:29E99DD6 ☀ computing @ chaos claudius<br class=""><br class="">_______________________________________________<br class="">calendarserver-dev mailing list<br class=""><a href="mailto:calendarserver-dev@lists.macosforge.org" class="">calendarserver-dev@lists.macosforge.org</a><br class="">https://lists.macosforge.org/mailman/listinfo/calendarserver-dev<br class=""></div></div></blockquote></div><br class=""></div></body></html>