[CalendarServer-changes] [15010] twext/branches/users/cdaboo/cfod

source_changes at macosforge.org source_changes at macosforge.org
Mon Jul 27 19:31:38 PDT 2015 

Revision: 15010
          http://trac.calendarserver.org//changeset/15010
Author:   cdaboo at apple.com
Date:     2015-07-27 19:31:38 -0700 (Mon, 27 Jul 2015)
Log Message:
-----------
Merge from trunk.

Modified Paths:
--------------
    twext/branches/users/cdaboo/cfod/twext/enterprise/adbapi2.py
    twext/branches/users/cdaboo/cfod/twext/enterprise/jobqueue.py
    twext/branches/users/cdaboo/cfod/twext/enterprise/test/test_adbapi2.py
    twext/branches/users/cdaboo/cfod/twext/python/launchd.py
    twext/branches/users/cdaboo/cfod/twext/python/log.py

Property Changed:
----------------
    twext/branches/users/cdaboo/cfod/
    twext/branches/users/cdaboo/cfod/twext/


Property changes on: twext/branches/users/cdaboo/cfod
___________________________________________________________________
Modified: svn:mergeinfo
   - /twext/branches/users/cdaboo/jobqueue-3:13444-13471
/twext/branches/users/cdaboo/jobs:12742-12780
/twext/branches/users/cdaboo/pod2pod-migration:14416-14520
/twext/branches/users/sagen/recordtypes:13647-13658
/twext/branches/users/sagen/recordtypes-2:13659
   + /twext/branches/users/cdaboo/jobqueue-3:13444-13471
/twext/branches/users/cdaboo/jobs:12742-12780
/twext/branches/users/cdaboo/pod2pod-migration:14416-14520
/twext/branches/users/sagen/recordtypes:13647-13658
/twext/branches/users/sagen/recordtypes-2:13659
/twext/trunk:14960-15005


Property changes on: twext/branches/users/cdaboo/cfod/twext
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalDAVTester/trunk/twext:11193-11198
/CalendarServer/branches/config-separation/twext:4379-4443
/CalendarServer/branches/egg-info-351/twext:4589-4625
/CalendarServer/branches/generic-sqlstore/twext:6167-6191
/CalendarServer/branches/new-store/twext:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/twext:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/twext:5936-5981
/CalendarServer/branches/release/CalendarServer-4.3-dev/twext:10180-10190,10192
/CalendarServer/branches/release/CalendarServer-5.1-dev/twext:11846
/CalendarServer/branches/users/cdaboo/batchupload-6699/twext:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/twext:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/twext:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/twext:3628-3644
/CalendarServer/branches/users/cdaboo/fix-no-ischedule/twext:11607-11871
/CalendarServer/branches/users/cdaboo/implicituidrace/twext:8137-8141
/CalendarServer/branches/users/cdaboo/ischedule-dkim/twext:9747-9979
/CalendarServer/branches/users/cdaboo/json/twext:11622-11912
/CalendarServer/branches/users/cdaboo/managed-attachments/twext:9985-10145
/CalendarServer/branches/users/cdaboo/more-sharing-5591/twext:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/twext:4465-4957
/CalendarServer/branches/users/cdaboo/performance-tweaks/twext:11824-11836
/CalendarServer/branches/users/cdaboo/pods/twext:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/twext:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/twext:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/twext:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/twext:5071-5105
/CalendarServer/branches/users/cdaboo/reverse-proxy-pods/twext:11875-11900
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/twext:5188-5440
/CalendarServer/branches/users/cdaboo/sharing-in-the-store/twext:11935-12016
/CalendarServer/branches/users/cdaboo/store-scheduling/twext:10876-11129
/CalendarServer/branches/users/cdaboo/timezones/twext:7443-7699
/CalendarServer/branches/users/cdaboo/txn-debugging/twext:8730-8743
/CalendarServer/branches/users/gaya/sharedgroups-3/twext:11088-11204
/CalendarServer/branches/users/glyph/always-abort-txn-on-error/twext:9958-9969
/CalendarServer/branches/users/glyph/case-insensitive-uid/twext:8772-8805
/CalendarServer/branches/users/glyph/conn-limit/twext:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/twext:4971-5080
/CalendarServer/branches/users/glyph/dalify/twext:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect/twext:6824-6876
/CalendarServer/branches/users/glyph/deploybuild/twext:7563-7572
/CalendarServer/branches/users/glyph/digest-auth-redux/twext:10624-10635
/CalendarServer/branches/users/glyph/disable-quota/twext:7718-7727
/CalendarServer/branches/users/glyph/dont-start-postgres/twext:6592-6614
/CalendarServer/branches/users/glyph/enforce-max-requests/twext:11640-11643
/CalendarServer/branches/users/glyph/hang-fix/twext:11465-11491
/CalendarServer/branches/users/glyph/imip-and-admin-html/twext:7866-7984
/CalendarServer/branches/users/glyph/ipv6-client/twext:9054-9105
/CalendarServer/branches/users/glyph/launchd-wrapper-bis/twext:11413-11436
/CalendarServer/branches/users/glyph/linux-tests/twext:6893-6900
/CalendarServer/branches/users/glyph/log-cleanups/twext:11691-11731
/CalendarServer/branches/users/glyph/migrate-merge/twext:8690-8713
/CalendarServer/branches/users/glyph/misc-portability-fixes/twext:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/twext:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7/twext:6369-6445
/CalendarServer/branches/users/glyph/multiget-delete/twext:8321-8330
/CalendarServer/branches/users/glyph/new-export/twext:7444-7485
/CalendarServer/branches/users/glyph/one-home-list-api/twext:10048-10073
/CalendarServer/branches/users/glyph/oracle/twext:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/twext:7340-7351
/CalendarServer/branches/users/glyph/other-html/twext:8062-8091
/CalendarServer/branches/users/glyph/parallel-sim/twext:8240-8251
/CalendarServer/branches/users/glyph/parallel-upgrade/twext:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/twext:8571-8583
/CalendarServer/branches/users/glyph/q/twext:9560-9688
/CalendarServer/branches/users/glyph/queue-locking-and-timing/twext:10204-10289
/CalendarServer/branches/users/glyph/quota/twext:7604-7637
/CalendarServer/branches/users/glyph/sendfdport/twext:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/twext:8436-8443
/CalendarServer/branches/users/glyph/shared-pool-take2/twext:8155-8174
/CalendarServer/branches/users/glyph/sharedpool/twext:6490-6550
/CalendarServer/branches/users/glyph/sharing-api/twext:9192-9205
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/twext:8524-8535
/CalendarServer/branches/users/glyph/sql-store/twext:5929-6073
/CalendarServer/branches/users/glyph/start-service-start-loop/twext:11060-11065
/CalendarServer/branches/users/glyph/subtransactions/twext:7248-7258
/CalendarServer/branches/users/glyph/table-alias/twext:8651-8664
/CalendarServer/branches/users/glyph/uidexport/twext:7673-7676
/CalendarServer/branches/users/glyph/unshare-when-access-revoked/twext:10562-10595
/CalendarServer/branches/users/glyph/use-system-twisted/twext:5084-5149
/CalendarServer/branches/users/glyph/uuid-normalize/twext:9268-9296
/CalendarServer/branches/users/glyph/warning-cleanups/twext:11347-11357
/CalendarServer/branches/users/glyph/whenNotProposed/twext:11881-11897
/CalendarServer/branches/users/glyph/xattrs-from-files/twext:7757-7769
/CalendarServer/branches/users/sagen/applepush/twext:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/twext:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/twext:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/twext:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/twext:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/twext:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/twext:4068-4075
/CalendarServer/branches/users/sagen/resources-2/twext:5084-5093
/CalendarServer/branches/users/sagen/testing/twext:10827-10851,10853-10855
/CalendarServer/branches/users/wsanchez/transations/twext:5515-5593
/twext/branches/users/cdaboo/jobqueue-3/twext:13444-13471
/twext/branches/users/cdaboo/jobs/twext:12742-12780
/twext/branches/users/cdaboo/pod2pod-migration/twext:14416-14520
/twext/branches/users/sagen/recordtypes/twext:13647-13658
/twext/branches/users/sagen/recordtypes-2/twext:13659
   + /CalDAVTester/trunk/twext:11193-11198
/CalendarServer/branches/config-separation/twext:4379-4443
/CalendarServer/branches/egg-info-351/twext:4589-4625
/CalendarServer/branches/generic-sqlstore/twext:6167-6191
/CalendarServer/branches/new-store-no-caldavfile-2/twext:5936-5981
/CalendarServer/branches/new-store-no-caldavfile/twext:5911-5935
/CalendarServer/branches/new-store/twext:5594-5934
/CalendarServer/branches/release/CalendarServer-4.3-dev/twext:10180-10190,10192
/CalendarServer/branches/release/CalendarServer-5.1-dev/twext:11846
/CalendarServer/branches/users/cdaboo/batchupload-6699/twext:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/twext:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/twext:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/twext:3628-3644
/CalendarServer/branches/users/cdaboo/fix-no-ischedule/twext:11607-11871
/CalendarServer/branches/users/cdaboo/implicituidrace/twext:8137-8141
/CalendarServer/branches/users/cdaboo/ischedule-dkim/twext:9747-9979
/CalendarServer/branches/users/cdaboo/json/twext:11622-11912
/CalendarServer/branches/users/cdaboo/managed-attachments/twext:9985-10145
/CalendarServer/branches/users/cdaboo/more-sharing-5591/twext:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/twext:4465-4957
/CalendarServer/branches/users/cdaboo/performance-tweaks/twext:11824-11836
/CalendarServer/branches/users/cdaboo/pods/twext:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/twext:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/twext:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/twext:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/twext:5071-5105
/CalendarServer/branches/users/cdaboo/reverse-proxy-pods/twext:11875-11900
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/twext:5188-5440
/CalendarServer/branches/users/cdaboo/sharing-in-the-store/twext:11935-12016
/CalendarServer/branches/users/cdaboo/store-scheduling/twext:10876-11129
/CalendarServer/branches/users/cdaboo/timezones/twext:7443-7699
/CalendarServer/branches/users/cdaboo/txn-debugging/twext:8730-8743
/CalendarServer/branches/users/gaya/sharedgroups-3/twext:11088-11204
/CalendarServer/branches/users/glyph/always-abort-txn-on-error/twext:9958-9969
/CalendarServer/branches/users/glyph/case-insensitive-uid/twext:8772-8805
/CalendarServer/branches/users/glyph/conn-limit/twext:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/twext:4971-5080
/CalendarServer/branches/users/glyph/dalify/twext:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect/twext:6824-6876
/CalendarServer/branches/users/glyph/deploybuild/twext:7563-7572
/CalendarServer/branches/users/glyph/digest-auth-redux/twext:10624-10635
/CalendarServer/branches/users/glyph/disable-quota/twext:7718-7727
/CalendarServer/branches/users/glyph/dont-start-postgres/twext:6592-6614
/CalendarServer/branches/users/glyph/enforce-max-requests/twext:11640-11643
/CalendarServer/branches/users/glyph/hang-fix/twext:11465-11491
/CalendarServer/branches/users/glyph/imip-and-admin-html/twext:7866-7984
/CalendarServer/branches/users/glyph/ipv6-client/twext:9054-9105
/CalendarServer/branches/users/glyph/launchd-wrapper-bis/twext:11413-11436
/CalendarServer/branches/users/glyph/linux-tests/twext:6893-6900
/CalendarServer/branches/users/glyph/log-cleanups/twext:11691-11731
/CalendarServer/branches/users/glyph/migrate-merge/twext:8690-8713
/CalendarServer/branches/users/glyph/misc-portability-fixes/twext:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/twext:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7/twext:6369-6445
/CalendarServer/branches/users/glyph/multiget-delete/twext:8321-8330
/CalendarServer/branches/users/glyph/new-export/twext:7444-7485
/CalendarServer/branches/users/glyph/one-home-list-api/twext:10048-10073
/CalendarServer/branches/users/glyph/oracle-nulls/twext:7340-7351
/CalendarServer/branches/users/glyph/oracle/twext:7106-7155
/CalendarServer/branches/users/glyph/other-html/twext:8062-8091
/CalendarServer/branches/users/glyph/parallel-sim/twext:8240-8251
/CalendarServer/branches/users/glyph/parallel-upgrade/twext:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/twext:8571-8583
/CalendarServer/branches/users/glyph/q/twext:9560-9688
/CalendarServer/branches/users/glyph/queue-locking-and-timing/twext:10204-10289
/CalendarServer/branches/users/glyph/quota/twext:7604-7637
/CalendarServer/branches/users/glyph/sendfdport/twext:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/twext:8436-8443
/CalendarServer/branches/users/glyph/shared-pool-take2/twext:8155-8174
/CalendarServer/branches/users/glyph/sharedpool/twext:6490-6550
/CalendarServer/branches/users/glyph/sharing-api/twext:9192-9205
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/twext:8524-8535
/CalendarServer/branches/users/glyph/sql-store/twext:5929-6073
/CalendarServer/branches/users/glyph/start-service-start-loop/twext:11060-11065
/CalendarServer/branches/users/glyph/subtransactions/twext:7248-7258
/CalendarServer/branches/users/glyph/table-alias/twext:8651-8664
/CalendarServer/branches/users/glyph/uidexport/twext:7673-7676
/CalendarServer/branches/users/glyph/unshare-when-access-revoked/twext:10562-10595
/CalendarServer/branches/users/glyph/use-system-twisted/twext:5084-5149
/CalendarServer/branches/users/glyph/uuid-normalize/twext:9268-9296
/CalendarServer/branches/users/glyph/warning-cleanups/twext:11347-11357
/CalendarServer/branches/users/glyph/whenNotProposed/twext:11881-11897
/CalendarServer/branches/users/glyph/xattrs-from-files/twext:7757-7769
/CalendarServer/branches/users/sagen/applepush/twext:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/twext:7380-7381
/CalendarServer/branches/users/sagen/locations-resources-2/twext:5052-5061
/CalendarServer/branches/users/sagen/locations-resources/twext:5032-5051
/CalendarServer/branches/users/sagen/purge_old_events/twext:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/twext:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/twext:4068-4075
/CalendarServer/branches/users/sagen/resources-2/twext:5084-5093
/CalendarServer/branches/users/sagen/testing/twext:10827-10851,10853-10855
/CalendarServer/branches/users/wsanchez/transations/twext:5515-5593
/twext/branches/users/cdaboo/jobqueue-3/twext:13444-13471
/twext/branches/users/cdaboo/jobs/twext:12742-12780
/twext/branches/users/cdaboo/pod2pod-migration/twext:14416-14520
/twext/branches/users/sagen/recordtypes-2/twext:13659
/twext/branches/users/sagen/recordtypes/twext:13647-13658
/twext/trunk/twext:14960-15005

Modified: twext/branches/users/cdaboo/cfod/twext/enterprise/adbapi2.py
===================================================================
--- twext/branches/users/cdaboo/cfod/twext/enterprise/adbapi2.py	2015-07-28 01:16:55 UTC (rev 15009)
+++ twext/branches/users/cdaboo/cfod/twext/enterprise/adbapi2.py	2015-07-28 02:31:38 UTC (rev 15010)
@@ -164,6 +164,10 @@
         self._label = label
 
 
+    def __repr__(self):
+        return "_ConnectedTxn({})".format(self._label)
+
+
     @_forward
     def paramstyle(self):
         """
@@ -425,6 +429,10 @@
         self._label = label
 
 
+    def __repr__(self):
+        return "_NoTxn({})".format(self._label)
+
+
     def _everything(self, *a, **kw):
         """
         Everything fails with a L{ConnectionError}.
@@ -459,6 +467,10 @@
         self._label = label
 
 
+    def __repr__(self):
+        return "_WaitingTxn({})".format(self._label)
+
+
     def _enspool(self, cmd, a=(), kw={}):
         d = Deferred()
         self._spool.append((d, cmd, a, kw))
@@ -912,6 +924,7 @@
         """
         self._pool = pool
         self._holder = holder
+        self._connection = False
         self._aborted = False
 
 
@@ -924,6 +937,13 @@
         self._aborted = True
         if self._retry is not None:
             self._retry.cancel()
+
+        def _reallyClose():
+            if self._connection:
+                self._connection.close()
+
+        self._holder.submit(_reallyClose)
+
         d = self._holder.stop()
 
         def removeme(ignored):
@@ -1121,7 +1141,10 @@
 
     def _startOneMore(self):
         """
-        Start one more _ConnectedTxn.
+        Start one more L{_ConnectedTxn}. What happens here is that we first create a
+        L{_ConnectingPseudoTxn} to hold a busy slot whilst the connection is starting
+        up. Once the connection is up, we replace the L{_ConnectingPseudoTxn} with the
+        L{_ConnectedTxn} that will do the actual DB work.
         """
         holder = self._createHolder()
         holder.start()
@@ -1132,9 +1155,9 @@
         def initCursor():
             # support threadlevel=1; we can't necessarily cursor() in a
             # different thread than we do transactions in.
-            connection = self.connectionFactory()
-            cursor = connection.cursor()
-            return (connection, cursor)
+            txn._connection = self.connectionFactory()
+            cursor = txn._connection.cursor()
+            return (txn._connection, cursor)
 
         def finishInit((connection, cursor)):
             if txn._aborted:

Modified: twext/branches/users/cdaboo/cfod/twext/enterprise/jobqueue.py
===================================================================
--- twext/branches/users/cdaboo/cfod/twext/enterprise/jobqueue.py	2015-07-28 01:16:55 UTC (rev 15009)
+++ twext/branches/users/cdaboo/cfod/twext/enterprise/jobqueue.py	2015-07-28 02:31:38 UTC (rev 15010)
@@ -1987,6 +1987,7 @@
         self._lastMinPriority = WORK_PRIORITY_LOW
         self._timeOfLastWork = time.time()
         self._actualPollInterval = self.queuePollInterval
+        self._inWorkCheck = False
 
 
     def addPeerConnection(self, peer):
@@ -2114,6 +2115,8 @@
             if not self.running or self.disableWorkProcessing:
                 returnValue(None)
 
+            self._inWorkCheck = True
+
             # Check the overall service load - if overloaded skip this poll cycle.
             # FIXME: need to include capacity of other nodes. For now we only check
             # our own capacity and stop processing if too busy. Other nodes that
@@ -2228,6 +2231,7 @@
                 if txn:
                     yield txn.commit()
                     txn = None
+                self._inWorkCheck = False
 
             if nextJob is not None:
                 try:
@@ -2382,7 +2386,13 @@
         for peer in self.peers:
             peer.transport.abortConnection()
 
+        # Wait for any active work check to finish
+        while self._inWorkCheck:
+            d = Deferred()
+            self.reactor.callLater(0.5, lambda : d.callback(None))
+            yield d
 
+
     def activeNodes(self, txn):
         """
         Load information about all other nodes.

Modified: twext/branches/users/cdaboo/cfod/twext/enterprise/test/test_adbapi2.py
===================================================================
--- twext/branches/users/cdaboo/cfod/twext/enterprise/test/test_adbapi2.py	2015-07-28 01:16:55 UTC (rev 15009)
+++ twext/branches/users/cdaboo/cfod/twext/enterprise/test/test_adbapi2.py	2015-07-28 02:31:38 UTC (rev 15010)
@@ -334,6 +334,24 @@
         self.assertEquals(holder.stopped, True)
 
 
+    def test_stopServicePseudoTxn(self):
+        """
+        When L{ConnectionPool.stopService} is called with a pending
+        L{_ConnectingPseudoTxn} active, the DB connection being created
+        is closed.
+        """
+        self.pauseHolders()
+        self.createTransaction()
+        stopResult = self.resultOf(self.pool.stopService())
+        self.assertEquals(stopResult, [])
+        self.flushHolders()
+        [holder] = self.holders
+        self.assertEquals(holder.started, True)
+        self.assertEquals(holder.stopped, True)
+        self.assertEquals(len(self.factory.connections), 1)
+        self.assertEquals(self.factory.connections[0].closed, True)
+
+
     def test_stopServiceMidAbort(self):
         """
         When L{ConnectionPool.stopService} is called with deferreds from

Modified: twext/branches/users/cdaboo/cfod/twext/python/launchd.py
===================================================================
--- twext/branches/users/cdaboo/cfod/twext/python/launchd.py	2015-07-28 01:16:55 UTC (rev 15009)
+++ twext/branches/users/cdaboo/cfod/twext/python/launchd.py	2015-07-28 02:31:38 UTC (rev 15010)
@@ -42,6 +42,7 @@
     raise ImportError(ve)
 
 
+
 def launchActivateSocket(name):
     fdList = []
 

Modified: twext/branches/users/cdaboo/cfod/twext/python/log.py
===================================================================
--- twext/branches/users/cdaboo/cfod/twext/python/log.py	2015-07-28 01:16:55 UTC (rev 15009)
+++ twext/branches/users/cdaboo/cfod/twext/python/log.py	2015-07-28 02:31:38 UTC (rev 15010)
@@ -997,7 +997,7 @@
             output = stdout
 
         if not text:
-            text = " ".join([str(m) for m in eventDict["message"]]) + "\n"
+            text = formatEvent(eventDict) + "\n"
 
         output.write(text)
         output.flush()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.macosforge.org/pipermail/calendarserver-changes/attachments/20150727/f0763296/attachment-0001.html>

More information about the calendarserver-changes mailing list