[CalendarServer-changes] [10229] CalendarServer/branches/users/glyph/queue-locking-and-timing

[source_changes at macosforge.org]

Revision: 10229
          http://trac.calendarserver.org//changeset/10229
Author:   glyph at apple.com
Date:     2013-01-04 16:38:41 -0800 (Fri, 04 Jan 2013)
Log Message:
-----------
Better explanation of the 'group' attribute.

Modified Paths:
--------------
    CalendarServer/branches/users/glyph/queue-locking-and-timing/twext/enterprise/queue.py

Property Changed:
----------------
    CalendarServer/branches/users/glyph/queue-locking-and-timing/

Modified: CalendarServer/branches/users/glyph/queue-locking-and-timing/twext/enterprise/queue.py
===================================================================
--- CalendarServer/branches/users/glyph/queue-locking-and-timing/twext/enterprise/queue.py	2013-01-05 00:38:40 UTC (rev 10228)
+++ CalendarServer/branches/users/glyph/queue-locking-and-timing/twext/enterprise/queue.py	2013-01-05 00:38:41 UTC (rev 10229)
@@ -275,16 +275,37 @@
     execution, to control concurrency and for performance reasons repsectively.
 
     Although all the usual database mutual-exclusion rules apply to work
-    executed in L{WorkItem.doWork}, locking is not always the best way to
-    manage concurrency.  Quite often the L{WorkItem}s that affect a particular
-    logical object (such as a user, or an account), and those that affect the
-    same logical object should not be run concurrently with each other, but
-    those that affect different objects can be run with any level of
-    concurrency without interfering.
+    executed in L{WorkItem.doWork}, implicit database row locking is not always
+    the best way to manage concurrency.  They have some problems, including:
 
-    To accomplish this sort of isolation, simply set the C{group} attribute on
-    your L{WorkItem} class.
+        - implicit locks are easy to accidentally acquire out of order, which
+          can lead to deadlocks
 
+        - implicit locks are easy to forget to acquire correctly - for example,
+          any read operation which subsequently turns into a write operation
+          must have been acquired with C{Select(..., ForUpdate=True)}, but it
+          is difficult to consistently indicate that methods which abstract out
+          read operations must pass this flag in certain cases and not others.
+
+        - implicit locks are held until the transaction ends, which means that
+          if expensive (long-running) queue operations share the same lock with
+          cheap (short-running) queue operations or user interactions, the
+          cheap operations all have to wait for the expensive ones to complete,
+          but continue to consume whatever database resources they were using.
+
+    In order to ameliorate these problems with potentiallly concurrent work
+    that uses the same resources, L{WorkItem} provides a database-wide mutex
+    that is automatically acquired at the beginning of the transaction and
+    released at the end.  To use it, simply L{align
+    <twext.enterprise.dal.record.Record.namingConvention>} the C{group}
+    attribute on your L{WorkItem} subclass with a column holding a string
+    (varchar).  L{WorkItem} subclasses with the same value for C{group} will
+    not execute their C{doWork} methods concurrently.  Furthermore, if the lock
+    cannot be quickly acquired, database resources associated with the
+    transaction attempting it will be released, and the transaction rolled back
+    until a future transaction I{can} can acquire it quickly.  If you do not
+    want any limits to concurrency, simply leave it set to C{None}.
+
     In some applications it's possible to coalesce work together; to grab
     multiple L{WorkItem}s in one C{doWork} transaction.  All you need to do is
     to delete the rows which back other L{WorkItem}s from the database, and
@@ -319,11 +340,9 @@
     @type notBefore: L{datetime.datetime} on instance,
         L{twext.enterprise.dal.syntax.ColumnSyntax} on class.
 
-    @cvar group: If not C{None}.
-    @type group: any type on the instance,
-        L{twext.enterprise.dal.syntax.ExpressionSyntax} naming one or more
-        L{twext.enterprise.dal.syntax.ColumnSyntax}es on the class (in other
-        words,: a L{ColumnSyntax} or a L{Tuple}).
+    @ivar group: If specified, the name of the mutual exclusion.
+    @type group: L{unicode} on instance,
+        L{twext.enterprise.dal.syntax.ColumnSyntax} on class.
     """
 
     @abstract
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/calendarserver-changes/attachments/20130104/13d5af42/attachment.html>