[CalendarServer-changes] [8584] CalendarServer/trunk

source_changes at macosforge.org source_changes at macosforge.org
Tue Jan 24 01:11:15 PST 2012 

Revision: 8584
          http://trac.macosforge.org/projects/calendarserver/changeset/8584
Author:   glyph at apple.com
Date:     2012-01-24 01:11:15 -0800 (Tue, 24 Jan 2012)
Log Message:
-----------
Merge parallel upgrade_to_1 branch.

Modified Paths:
--------------
    CalendarServer/trunk/calendarserver/tap/caldav.py
    CalendarServer/trunk/calendarserver/tap/cfgchild.py
    CalendarServer/trunk/calendarserver/tools/util.py
    CalendarServer/trunk/twext/internet/spawnsvc.py
    CalendarServer/trunk/twistedcaldav/config.py
    CalendarServer/trunk/twistedcaldav/test/test_upgrade.py
    CalendarServer/trunk/twistedcaldav/upgrade.py
    CalendarServer/trunk/txdav/common/datastore/upgrade/migrate.py
    CalendarServer/trunk/txdav/common/datastore/upgrade/test/test_migrate.py

Added Paths:
-----------
    CalendarServer/trunk/twext/python/parallel.py
    CalendarServer/trunk/twext/python/test/test_parallel.py

Property Changed:
----------------
    CalendarServer/trunk/
    CalendarServer/trunk/support/build.sh
    CalendarServer/trunk/txdav/caldav/datastore/index_file.py
    CalendarServer/trunk/txdav/caldav/datastore/test/test_index_file.py
    CalendarServer/trunk/txdav/carddav/datastore/index_file.py
    CalendarServer/trunk/txdav/carddav/datastore/test/test_index_file.py


Property changes on: CalendarServer/trunk
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalendarServer/branches/config-separation:4379-4443
/CalendarServer/branches/egg-info-351:4589-4625
/CalendarServer/branches/generic-sqlstore:6167-6191
/CalendarServer/branches/new-store:5594-5934
/CalendarServer/branches/new-store-no-caldavfile:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464:4465-4957
/CalendarServer/branches/users/cdaboo/pods:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar:7085-7206
/CalendarServer/branches/users/cdaboo/pycard:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187:5188-5440
/CalendarServer/branches/users/cdaboo/timezones:7443-7699
/CalendarServer/branches/users/glyph/conn-limit:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge:4971-5080
/CalendarServer/branches/users/glyph/dalify:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect:6824-6876
/CalendarServer/branches/users/glyph/deploybuild:7563-7572
/CalendarServer/branches/users/glyph/disable-quota:7718-7727
/CalendarServer/branches/users/glyph/dont-start-postgres:6592-6614
/CalendarServer/branches/users/glyph/imip-and-admin-html:7866-7984
/CalendarServer/branches/users/glyph/linux-tests:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7:6369-6445
/CalendarServer/branches/users/glyph/multiget-delete:8321-8330
/CalendarServer/branches/users/glyph/new-export:7444-7485
/CalendarServer/branches/users/glyph/oracle:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls:7340-7351
/CalendarServer/branches/users/glyph/other-html:8062-8091
/CalendarServer/branches/users/glyph/parallel-sim:8240-8251
/CalendarServer/branches/users/glyph/parallel-upgrade:8376-8400
/CalendarServer/branches/users/glyph/quota:7604-7637
/CalendarServer/branches/users/glyph/sendfdport:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes:8436-8443
/CalendarServer/branches/users/glyph/shared-pool-take2:8155-8174
/CalendarServer/branches/users/glyph/sharedpool:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones:8524-8535
/CalendarServer/branches/users/glyph/sql-store:5929-6073
/CalendarServer/branches/users/glyph/subtransactions:7248-7258
/CalendarServer/branches/users/glyph/uidexport:7673-7676
/CalendarServer/branches/users/glyph/use-system-twisted:5084-5149
/CalendarServer/branches/users/glyph/xattrs-from-files:7757-7769
/CalendarServer/branches/users/sagen/applepush:8126-8184
/CalendarServer/branches/users/sagen/inboxitems:7380-7381
/CalendarServer/branches/users/sagen/locations-resources:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066:4068-4075
/CalendarServer/branches/users/sagen/resources-2:5084-5093
/CalendarServer/branches/users/wsanchez/transations:5515-5593
   + /CalendarServer/branches/config-separation:4379-4443
/CalendarServer/branches/egg-info-351:4589-4625
/CalendarServer/branches/generic-sqlstore:6167-6191
/CalendarServer/branches/new-store:5594-5934
/CalendarServer/branches/new-store-no-caldavfile:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464:4465-4957
/CalendarServer/branches/users/cdaboo/pods:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar:7085-7206
/CalendarServer/branches/users/cdaboo/pycard:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187:5188-5440
/CalendarServer/branches/users/cdaboo/timezones:7443-7699
/CalendarServer/branches/users/glyph/conn-limit:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge:4971-5080
/CalendarServer/branches/users/glyph/dalify:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect:6824-6876
/CalendarServer/branches/users/glyph/deploybuild:7563-7572
/CalendarServer/branches/users/glyph/disable-quota:7718-7727
/CalendarServer/branches/users/glyph/dont-start-postgres:6592-6614
/CalendarServer/branches/users/glyph/imip-and-admin-html:7866-7984
/CalendarServer/branches/users/glyph/linux-tests:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7:6369-6445
/CalendarServer/branches/users/glyph/multiget-delete:8321-8330
/CalendarServer/branches/users/glyph/new-export:7444-7485
/CalendarServer/branches/users/glyph/oracle:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls:7340-7351
/CalendarServer/branches/users/glyph/other-html:8062-8091
/CalendarServer/branches/users/glyph/parallel-sim:8240-8251
/CalendarServer/branches/users/glyph/parallel-upgrade:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1:8571-8583
/CalendarServer/branches/users/glyph/quota:7604-7637
/CalendarServer/branches/users/glyph/sendfdport:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes:8436-8443
/CalendarServer/branches/users/glyph/shared-pool-take2:8155-8174
/CalendarServer/branches/users/glyph/sharedpool:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones:8524-8535
/CalendarServer/branches/users/glyph/sql-store:5929-6073
/CalendarServer/branches/users/glyph/subtransactions:7248-7258
/CalendarServer/branches/users/glyph/uidexport:7673-7676
/CalendarServer/branches/users/glyph/use-system-twisted:5084-5149
/CalendarServer/branches/users/glyph/xattrs-from-files:7757-7769
/CalendarServer/branches/users/sagen/applepush:8126-8184
/CalendarServer/branches/users/sagen/inboxitems:7380-7381
/CalendarServer/branches/users/sagen/locations-resources:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066:4068-4075
/CalendarServer/branches/users/sagen/resources-2:5084-5093
/CalendarServer/branches/users/wsanchez/transations:5515-5593

Modified: CalendarServer/trunk/calendarserver/tap/caldav.py
===================================================================
--- CalendarServer/trunk/calendarserver/tap/caldav.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/calendarserver/tap/caldav.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -918,17 +918,16 @@
                     parallel = config.MultiProcess.ProcessCount
                 else:
                     parallel = 0
+                spawner = ConfiguredChildSpawner(self, dispenser, config)
                 upgradeSvc = UpgradeFileSystemFormatService(
-                    config,
+                    config, spawner, parallel,
                     UpgradeDatabaseSchemaService.wrapService(
                         UpgradeDatabaseDataService.wrapService(
                             UpgradeToDatabaseService.wrapService(
                                 CachingFilePath(config.DocumentRoot),
                                 PostDBImportService(config, store, mainService),
                                 store, uid=overrideUID, gid=overrideGID,
-                                spawner=ConfiguredChildSpawner(
-                                    self, dispenser, config
-                                ),
+                                spawner=spawner,
                                 parallel=parallel
                             ),
                             store, uid=overrideUID, gid=overrideGID,

Modified: CalendarServer/trunk/calendarserver/tap/cfgchild.py
===================================================================
--- CalendarServer/trunk/calendarserver/tap/cfgchild.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/calendarserver/tap/cfgchild.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -29,7 +29,7 @@
 from calendarserver.tools.util import setupMemcached
 from twisted.python.reflect import namedAny, qual
 from twisted.internet.defer import inlineCallbacks, returnValue
-from twisted.protocols.amp import AMP, Command, String, Integer#, ListOf
+from twisted.protocols.amp import AMP, Command, String, Integer, Boolean
 from txdav.common.datastore.upgrade.migrate import (
     StoreSpawnerService, swapAMP
 )
@@ -61,6 +61,7 @@
 
         ## shared connection pool!
         ("connectionPoolFD", Integer(optional=True)),
+        ("withStore", Boolean()),
     ]
 
 
@@ -83,7 +84,7 @@
 
     @ConfigureChild.responder
     def conf(self, delegateTo, pidFile, logID, configFile, processCount,
-             connectionPoolFD=None):
+             connectionPoolFD=None, withStore=True):
         """
         Load the current config file into this child process, create a store
         based on it, and delegate to the upgrade logic.
@@ -107,19 +108,21 @@
             changedConfig.update(DBAMPFD=connectionPoolFD)
         self.config.updateDefaults(changedConfig)
 
-        # Construct and start database pool and store.
-        pool, txnf = getDBPool(self.config)
-        if pool is not None:
-            from twisted.internet import reactor
-            pool.startService()
-            reactor.addSystemEventTrigger(
-                "before", "shutdown", pool.stopService
-            )
-        dbstore = storeFromConfig(self.config, txnf)
-
+        if withStore:
+            # Construct and start database pool and store.
+            pool, txnf = getDBPool(self.config)
+            if pool is not None:
+                from twisted.internet import reactor
+                pool.startService()
+                reactor.addSystemEventTrigger(
+                    "before", "shutdown", pool.stopService
+                )
+            delegateArg = storeFromConfig(self.config, txnf)
+        else:
+            delegateArg = self.config
         # Finally, construct the class we're supposed to delegate to.
         delegateClass = namedAny(delegateTo)
-        swapAMP(self, delegateClass(dbstore))
+        swapAMP(self, delegateClass(delegateArg))
         return {}
 
 
@@ -148,11 +151,25 @@
         self.config = config
 
 
-    @inlineCallbacks
+    def spawnWithConfig(self, config, here, there):
+        """
+        Spawn the child with a configuration.
+        """
+        return self._doSpawn(config, here, there, False)
+
+
     def spawnWithStore(self, here, there):
         """
         Spawn the child with a store based on a configuration.
         """
+        return self._doSpawn(self.config, here, there, True)
+
+
+    @inlineCallbacks
+    def _doSpawn(self, config, here, there, withStore):
+        """
+        Common implementation of L{spawnWithStore} and L{spawnWithConfig}.
+        """
         thisID = self.nextID
         self.nextID += 1
         if self.dispenser is not None:
@@ -178,8 +195,9 @@
             delegateTo=qual(there),
             pidFile="%s-migrator-%s" % (self.maker.tapname, thisID),
             logID="migrator-%s" % (thisID,),
-            configFile=self.config.getProvider().getConfigFileName(),
-            processCount=self.config.MultiProcess.ProcessCount,
+            configFile=config.getProvider().getConfigFileName(),
+            processCount=config.MultiProcess.ProcessCount,
+            withStore=withStore,
             **extra
         )
         returnValue(swapAMP(controller, here))

Modified: CalendarServer/trunk/calendarserver/tools/util.py
===================================================================
--- CalendarServer/trunk/calendarserver/tools/util.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/calendarserver/tools/util.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -70,7 +70,7 @@
 
     return config
 
-def getDirectory():
+def getDirectory(config=config):
 
     class MyDirectoryService (AggregateDirectoryService):
         def getPrincipalCollection(self):


Property changes on: CalendarServer/trunk/support/build.sh
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalendarServer/branches/config-separation/support/build.sh:4379-4443
/CalendarServer/branches/egg-info-351/support/build.sh:4589-4615
/CalendarServer/branches/generic-sqlstore/support/build.sh:6167-6191
/CalendarServer/branches/new-store/support/build.sh:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/support/build.sh:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/support/build.sh:5936-5981
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/support/build.sh:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/support/build.sh:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/support/build.sh:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace/support/build.sh:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591/support/build.sh:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/support/build.sh:4465-4957
/CalendarServer/branches/users/cdaboo/pods/support/build.sh:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/support/build.sh:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/support/build.sh:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/support/build.sh:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/support/build.sh:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/support/build.sh:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/support/build.sh:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/support/build.sh:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/support/build.sh:4971-5080
/CalendarServer/branches/users/glyph/dalify/support/build.sh:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect/support/build.sh:6824-6876
/CalendarServer/branches/users/glyph/deploybuild/support/build.sh:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/support/build.sh:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/support/build.sh:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/support/build.sh:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/support/build.sh:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7/support/build.sh:6369-6445
/CalendarServer/branches/users/glyph/new-export/support/build.sh:7444-7485
/CalendarServer/branches/users/glyph/oracle-nulls/support/build.sh:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/support/build.sh:8376-8400
/CalendarServer/branches/users/glyph/sendfdport/support/build.sh:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/support/build.sh:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/support/build.sh:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/support/build.sh:8524-8535
/CalendarServer/branches/users/glyph/sql-store/support/build.sh:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/support/build.sh:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/support/build.sh:5084-5149
/CalendarServer/branches/users/sagen/applepush/support/build.sh:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/support/build.sh:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/support/build.sh:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/support/build.sh:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/support/build.sh:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/support/build.sh:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/support/build.sh:4068-4075
/CalendarServer/branches/users/sagen/resources-2/support/build.sh:5084-5093
/CalendarServer/branches/users/wsanchez/transations/support/build.sh:5515-5593
   + /CalendarServer/branches/config-separation/support/build.sh:4379-4443
/CalendarServer/branches/egg-info-351/support/build.sh:4589-4615
/CalendarServer/branches/generic-sqlstore/support/build.sh:6167-6191
/CalendarServer/branches/new-store/support/build.sh:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/support/build.sh:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/support/build.sh:5936-5981
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/support/build.sh:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/support/build.sh:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/support/build.sh:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace/support/build.sh:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591/support/build.sh:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/support/build.sh:4465-4957
/CalendarServer/branches/users/cdaboo/pods/support/build.sh:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/support/build.sh:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/support/build.sh:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/support/build.sh:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/support/build.sh:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/support/build.sh:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/support/build.sh:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/support/build.sh:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/support/build.sh:4971-5080
/CalendarServer/branches/users/glyph/dalify/support/build.sh:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect/support/build.sh:6824-6876
/CalendarServer/branches/users/glyph/deploybuild/support/build.sh:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/support/build.sh:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/support/build.sh:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/support/build.sh:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/support/build.sh:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7/support/build.sh:6369-6445
/CalendarServer/branches/users/glyph/new-export/support/build.sh:7444-7485
/CalendarServer/branches/users/glyph/oracle-nulls/support/build.sh:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/support/build.sh:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/support/build.sh:8571-8583
/CalendarServer/branches/users/glyph/sendfdport/support/build.sh:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/support/build.sh:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/support/build.sh:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/support/build.sh:8524-8535
/CalendarServer/branches/users/glyph/sql-store/support/build.sh:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/support/build.sh:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/support/build.sh:5084-5149
/CalendarServer/branches/users/sagen/applepush/support/build.sh:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/support/build.sh:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/support/build.sh:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/support/build.sh:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/support/build.sh:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/support/build.sh:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/support/build.sh:4068-4075
/CalendarServer/branches/users/sagen/resources-2/support/build.sh:5084-5093
/CalendarServer/branches/users/wsanchez/transations/support/build.sh:5515-5593

Modified: CalendarServer/trunk/twext/internet/spawnsvc.py
===================================================================
--- CalendarServer/trunk/twext/internet/spawnsvc.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/twext/internet/spawnsvc.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -222,6 +222,7 @@
         super(SpawnerService, self).startService()
         for spawn in self.pendingSpawns:
             self.spawn(*spawn)
+        self.pendingSpawns = []
 
 
     def addBridge(self, bridge):
@@ -242,7 +243,7 @@
         if self._stopAllDeferred is not None:
             if len(self.bridges) == 0:
                 self._stopAllDeferred.callback(None)
-                self._stopAllDeferred = False
+                self._stopAllDeferred = None
 
 
     def stopService(self):

Copied: CalendarServer/trunk/twext/python/parallel.py (from rev 8583, CalendarServer/branches/users/glyph/parallel-upgrade_to_1/twext/python/parallel.py)
===================================================================
--- CalendarServer/trunk/twext/python/parallel.py	                        (rev 0)
+++ CalendarServer/trunk/twext/python/parallel.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -0,0 +1,94 @@
+# -*- test-case-name: twext.python.test.test_parallel -*-
+##
+# Copyright (c) 2012 Apple Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+##
+
+"""
+Utilities for parallelizing tasks.
+"""
+
+from twisted.internet.defer import inlineCallbacks, DeferredList, returnValue
+
+class Parallelizer(object):
+    """
+    Do some operation with a degree of parallelism, using a set of resources
+    which may each only be used for one task at a time, given some underlying
+    API that returns L{Deferreds}.
+
+    @ivar available: A list of available resources from the C{resources}
+        constructor parameter.
+
+    @ivar busy: A list of resources which are currently being used by
+        operations.
+    """
+
+    def __init__(self, resources):
+        """
+        Initialize a L{Parallelizer} with a list of objects that will be passed
+        to the callables sent to L{Parallelizer.do}.
+
+        @param resources: objects which may be of any arbitrary type.
+        @type resources: C{list}
+        """
+        self.available = list(resources)
+        self.busy = []
+        self.activeDeferreds = []
+
+
+    @inlineCallbacks
+    def do(self, operation):
+        """
+        Call C{operation} with one of the resources in C{self.available},
+        removing that value for use by other callers of C{do} until the task
+        performed by C{operation} is complete (in other words, the L{Deferred}
+        returned by C{operation} has fired).
+
+        @param operation: a 1-argument callable taking a resource from
+            C{self.active} and returning a L{Deferred} when it's done using
+            that resource.
+        @type operation: C{callable}
+
+        @return: a L{Deferred} that fires as soon as there are resources
+            available such that this task can be I{started} - not completed.
+        """
+        if not self.available:
+            yield DeferredList(self.activeDeferreds, fireOnOneCallback=True,
+                               fireOnOneErrback=True)
+        active = self.available.pop(0)
+        self.busy.append(active)
+        o = operation(active)
+        def andFinally(whatever):
+            self.activeDeferreds.remove(o)
+            self.busy.remove(active)
+            self.available.append(active)
+            return whatever
+        self.activeDeferreds.append(o)
+        o.addBoth(andFinally)
+        returnValue(None)
+
+
+    def done(self):
+        """
+        Wait until all operations started by L{Parallelizer.do} are completed.
+
+        @return: a L{Deferred} that fires (with C{None}) when all the currently
+            pending work on this L{Parallelizer} is completed and C{busy} is
+            empty again.
+        """
+        return (DeferredList(self.activeDeferreds)
+                .addCallback(lambda ignored: None))
+
+
+

Copied: CalendarServer/trunk/twext/python/test/test_parallel.py (from rev 8583, CalendarServer/branches/users/glyph/parallel-upgrade_to_1/twext/python/test/test_parallel.py)
===================================================================
--- CalendarServer/trunk/twext/python/test/test_parallel.py	                        (rev 0)
+++ CalendarServer/trunk/twext/python/test/test_parallel.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -0,0 +1,63 @@
+##
+# Copyright (c) 2012 Apple Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+##
+
+"""
+Tests for L{twext.python.parallel}.
+"""
+
+from twisted.internet.defer import Deferred
+
+from twext.python.parallel import Parallelizer
+
+from twisted.trial.unittest import TestCase
+
+class ParallelizerTests(TestCase):
+    """
+    Tests for L{Parallelizer}.
+    """
+
+    def test_doAndDone(self):
+        """
+        Blanket catch-all test.  (TODO: split this up into more nice
+        fine-grained tests.)
+        """
+        d1 = Deferred()
+        d2 = Deferred()
+        d3 = Deferred()
+        d4 = Deferred()
+        doing = []
+        done = []
+        allDone = []
+        p = Parallelizer(['a', 'b', 'c'])
+        p.do(lambda a: doing.append(a) or d1).addCallback(done.append)
+        p.do(lambda b: doing.append(b) or d2).addCallback(done.append)
+        p.do(lambda c: doing.append(c) or d3).addCallback(done.append)
+        p.do(lambda b1: doing.append(b1) or d4).addCallback(done.append)
+        p.done().addCallback(allDone.append)
+        self.assertEqual(allDone, [])
+        self.assertEqual(doing, ['a', 'b', 'c'])
+        self.assertEqual(done, [None, None, None])
+        d2.callback(1)
+        self.assertEqual(doing, ['a', 'b', 'c', 'b'])
+        self.assertEqual(done, [None, None, None, None])
+        self.assertEqual(allDone, [])
+        d3.callback(2)
+        d4.callback(3)
+        d1.callback(4)
+        self.assertEqual(done, [None, None, None, None])
+        self.assertEqual(allDone, [None])
+
+

Modified: CalendarServer/trunk/twistedcaldav/config.py
===================================================================
--- CalendarServer/trunk/twistedcaldav/config.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/twistedcaldav/config.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -152,6 +152,8 @@
         if not attr.startswith("_"):
             self.__dict__["_dirty"] = True
 
+    _dirty = False
+    _data = ()
     def __getattr__(self, attr):
         if self._dirty:
             self.update()

Modified: CalendarServer/trunk/twistedcaldav/test/test_upgrade.py
===================================================================
--- CalendarServer/trunk/twistedcaldav/test/test_upgrade.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/twistedcaldav/test/test_upgrade.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -66,7 +66,13 @@
         config.ResourceService.params.xmlFile = resourceFile
 
 
+    def doUpgrade(self, config):
+        """
+        Perform the actual upgrade.  (Hook for parallel tests.)
+        """
+        return upgradeData(config)
 
+
     def setUpInitialStates(self):
         self.setUpXMLDirectory()
 
@@ -77,6 +83,7 @@
         self.setUpNewDataRoot()
         self.setUpDataRootWithProxyDB()
 
+
     def setUpOldDocRoot(self):
         
         # Set up doc root
@@ -155,7 +162,7 @@
         self.assertTrue(os.path.exists(os.path.join(config.DocumentRoot, "principals", OLDPROXYFILE)))
         self.assertFalse(os.path.exists(os.path.join(config.DataRoot, NEWPROXYFILE)))
 
-        (yield upgradeData(config))
+        (yield self.doUpgrade(config))
         
         # Check post-conditions
         self.assertFalse(os.path.exists(os.path.join(config.DocumentRoot, "principals",)))
@@ -177,7 +184,7 @@
         self.assertFalse(os.path.exists(os.path.join(config.DocumentRoot, "principals")))
         self.assertTrue(os.path.exists(os.path.join(config.DataRoot, NEWPROXYFILE)))
 
-        (yield upgradeData(config))
+        (yield self.doUpgrade(config))
         
         # Check post-conditions
         self.assertFalse(os.path.exists(os.path.join(config.DocumentRoot, "principals",)))
@@ -273,12 +280,12 @@
         config.DocumentRoot = root
         config.DataRoot = root
 
-        (yield upgradeData(config))
+        (yield self.doUpgrade(config))
         self.assertTrue(self.verifyHierarchy(root, after))
 
         if reverify:
             # Ensure that repeating the process doesn't change anything
-            (yield upgradeData(config))
+            (yield self.doUpgrade(config))
             self.assertTrue(self.verifyHierarchy(root, after))
 
 
@@ -1316,7 +1323,7 @@
         config.DataRoot = root
 
         try:
-            (yield upgradeData(config))
+            (yield self.doUpgrade(config))
         except UpgradeError:
             pass
         else:
@@ -1379,7 +1386,7 @@
         config.DocumentRoot = root
         config.DataRoot = root
 
-        (yield upgradeData(config))
+        (yield self.doUpgrade(config))
         self.assertTrue(self.verifyHierarchy(root, after))
 
         proxydbClass = namedClass(config.ProxyDBService.type)
@@ -1590,3 +1597,15 @@
         return True
     except ValueError:
         return False
+
+
+class ParallelUpgradeTests(UpgradeTests):
+    """
+    Tests for upgradeData in parallel.
+    """
+
+    def doUpgrade(self, config):
+        from txdav.common.datastore.upgrade.test.test_migrate import StubSpawner
+        spawner = StubSpawner(config)
+        return upgradeData(config, spawner, 2)
+

Modified: CalendarServer/trunk/twistedcaldav/upgrade.py
===================================================================
--- CalendarServer/trunk/twistedcaldav/upgrade.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/twistedcaldav/upgrade.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -17,7 +17,7 @@
 
 from __future__ import with_statement
 
-import xattr, os, zlib, hashlib, datetime, pwd, grp, shutil, errno
+import xattr, os, zlib, hashlib, datetime, pwd, grp, shutil, errno, operator
 from zlib import compress
 from cPickle import loads as unpickle, UnpicklingError
 
@@ -40,16 +40,21 @@
 
 from twisted.application.service import Service
 from twisted.internet import reactor
-from twisted.internet.defer import inlineCallbacks, succeed, returnValue
+from twisted.internet.defer import (
+    inlineCallbacks, succeed, returnValue, gatherResults
+)
 from twisted.python.reflect import namedAny
 from twisted.python.reflect import namedClass
 
 from txdav.caldav.datastore.index_file import db_basename
 
+from twisted.protocols.amp import AMP, Command, String, Boolean
+
 from calendarserver.tap.util import getRootResource, FakeRequest, directoryFromConfig
 from calendarserver.tools.resources import migrateResources
 from calendarserver.tools.util import getDirectory
 
+from twext.python.parallel import Parallelizer
 
 deadPropertyXattrPrefix = namedAny(
     "txdav.base.propertystore.xattr.PropertyStore.deadPropertyXattrPrefix"
@@ -81,102 +86,150 @@
 
     return uid, gid
 
-#
-# upgrade_to_1
-#
-# Upconverts data from any calendar server version prior to data format 1
-#
 
- at inlineCallbacks
-def upgrade_to_1(config, directory):
+def fixBadQuotes(data):
+    if (
+        data.find('\\"') != -1 or
+        data.find('\\\r\n "') != -1 or
+        data.find('\r\n \r\n "') != -1
+    ):
+        # Fix by continuously replacing \" with " until no more
+        # replacements occur
+        while True:
+            newData = data.replace('\\"', '"').replace('\\\r\n "', '\r\n "').replace('\r\n \r\n "', '\r\n "')
+            if newData == data:
+                break
+            else:
+                data = newData
 
-    errorOccurred = False
+        return data, True
+    else:
+        return data, False
 
-    def fixBadQuotes(data):
-        if (
-            data.find('\\"') != -1 or
-            data.find('\\\r\n "') != -1 or
-            data.find('\r\n \r\n "') != -1
-        ):
-            # Fix by continuously replacing \" with " until no more
-            # replacements occur
-            while True:
-                newData = data.replace('\\"', '"').replace('\\\r\n "', '\r\n "').replace('\r\n \r\n "', '\r\n "')
-                if newData == data:
-                    break
-                else:
-                    data = newData
 
-            return data, True
-        else:
-            return data, False
 
+def upgradeCalendarCollection(calPath, directory, cuaCache):
+    errorOccurred = False
+    collectionUpdated = False
 
+    for resource in os.listdir(calPath):
 
+        if resource.startswith("."):
+            continue
 
+        resPath = os.path.join(calPath, resource)
 
-    def upgradeCalendarCollection(calPath, directory, cuaCache):
+        if os.path.isdir(resPath):
+            # Skip directories
+            continue
 
-        errorOccurred = False
-        collectionUpdated = False
+        log.debug("Processing: %s" % (resPath,))
+        needsRewrite = False
+        with open(resPath) as res:
+            data = res.read()
 
-        for resource in os.listdir(calPath):
+            try:
+                data, fixed = fixBadQuotes(data)
+                if fixed:
+                    log.warn("Fixing bad quotes in %s" % (resPath,))
+                    needsRewrite = True
+            except Exception, e:
+                log.error("Error while fixing bad quotes in %s: %s" %
+                    (resPath, e))
+                errorOccurred = True
+                continue
 
-            if resource.startswith("."):
+            try:
+                data, fixed = removeIllegalCharacters(data)
+                if fixed:
+                    log.warn("Removing illegal characters in %s" % (resPath,))
+                    needsRewrite = True
+            except Exception, e:
+                log.error("Error while removing illegal characters in %s: %s" %
+                    (resPath, e))
+                errorOccurred = True
                 continue
 
-            resPath = os.path.join(calPath, resource)
-
-            if os.path.isdir(resPath):
-                # Skip directories
+            try:
+                data, fixed = normalizeCUAddrs(data, directory, cuaCache)
+                if fixed:
+                    log.debug("Normalized CUAddrs in %s" % (resPath,))
+                    needsRewrite = True
+            except Exception, e:
+                log.error("Error while normalizing %s: %s" %
+                    (resPath, e))
+                errorOccurred = True
                 continue
 
-            log.debug("Processing: %s" % (resPath,))
-            needsRewrite = False
-            with open(resPath) as res:
-                data = res.read()
+        if needsRewrite:
+            with open(resPath, "w") as res:
+                res.write(data)
 
-                try:
-                    data, fixed = fixBadQuotes(data)
-                    if fixed:
-                        log.warn("Fixing bad quotes in %s" % (resPath,))
-                        needsRewrite = True
-                except Exception, e:
-                    log.error("Error while fixing bad quotes in %s: %s" %
-                        (resPath, e))
-                    errorOccurred = True
-                    continue
+            md5value = "<?xml version='1.0' encoding='UTF-8'?>\r\n<getcontentmd5 xmlns='http://twistedmatrix.com/xml_namespace/dav/'>%s</getcontentmd5>\r\n" % (hashlib.md5(data).hexdigest(),)
+            md5value = zlib.compress(md5value)
+            try:
+                xattr.setxattr(resPath, xattrname("{http:%2F%2Ftwistedmatrix.com%2Fxml_namespace%2Fdav%2F}getcontentmd5"), md5value)
+            except IOError, ioe:
+                if ioe.errno == errno.EOPNOTSUPP:
+                    # On non-native xattr systems we cannot do this,
+                    # but those systems will typically not be migrating
+                    # from pre-v1
+                    pass
+            except:
+                raise
 
-                try:
-                    data, fixed = removeIllegalCharacters(data)
-                    if fixed:
-                        log.warn("Removing illegal characters in %s" % (resPath,))
-                        needsRewrite = True
-                except Exception, e:
-                    log.error("Error while removing illegal characters in %s: %s" %
-                        (resPath, e))
-                    errorOccurred = True
-                    continue
+            collectionUpdated = True
 
-                try:
-                    data, fixed = normalizeCUAddrs(data, directory, cuaCache)
-                    if fixed:
-                        log.debug("Normalized CUAddrs in %s" % (resPath,))
-                        needsRewrite = True
-                except Exception, e:
-                    log.error("Error while normalizing %s: %s" %
-                        (resPath, e))
-                    errorOccurred = True
-                    continue
 
-            if needsRewrite:
-                with open(resPath, "w") as res:
-                    res.write(data)
+    if collectionUpdated:
+        ctagValue = "<?xml version='1.0' encoding='UTF-8'?>\r\n<getctag xmlns='http://calendarserver.org/ns/'>%s</getctag>\r\n" % (str(datetime.datetime.now()),)
+        ctagValue = zlib.compress(ctagValue)
+        try:
+            xattr.setxattr(calPath, xattrname("{http:%2F%2Fcalendarserver.org%2Fns%2F}getctag"), ctagValue)
+        except IOError, ioe:
+            if ioe.errno == errno.EOPNOTSUPP:
+                # On non-native xattr systems we cannot do this,
+                # but those systems will typically not be migrating
+                # from pre-v1
+                pass
+        except:
+            raise
 
-                md5value = "<?xml version='1.0' encoding='UTF-8'?>\r\n<getcontentmd5 xmlns='http://twistedmatrix.com/xml_namespace/dav/'>%s</getcontentmd5>\r\n" % (hashlib.md5(data).hexdigest(),)
-                md5value = zlib.compress(md5value)
+    return errorOccurred
+
+
+
+def upgradeCalendarHome(homePath, directory, cuaCache):
+
+    errorOccurred = False
+
+    log.debug("Upgrading calendar home: %s" % (homePath,))
+
+    try:
+        for cal in os.listdir(homePath):
+            calPath = os.path.join(homePath, cal)
+            if not os.path.isdir(calPath):
+                # Skip non-directories; these might have been uploaded by a
+                # random DAV client, they can't be calendar collections.
+                continue
+            if cal == 'notifications':
+                # Delete the old, now obsolete, notifications directory.
+                rmdir(calPath)
+                continue
+            log.debug("Upgrading calendar: %s" % (calPath,))
+            if not upgradeCalendarCollection(calPath, directory, cuaCache):
+                errorOccurred = True
+
+            # Change the calendar-free-busy-set xattrs of the inbox to the
+            # __uids__/<guid> form
+            if cal == "inbox":
                 try:
-                    xattr.setxattr(resPath, xattrname("{http:%2F%2Ftwistedmatrix.com%2Fxml_namespace%2Fdav%2F}getcontentmd5"), md5value)
+                    for attr, value in xattr.xattr(calPath).iteritems():
+                        if attr == xattrname("{urn:ietf:params:xml:ns:caldav}calendar-free-busy-set"):
+                            value = updateFreeBusySet(value, directory)
+                            if value is not None:
+                                # Need to write the xattr back to disk
+                                xattr.setxattr(calPath, attr, value)
                 except IOError, ioe:
                     if ioe.errno == errno.EOPNOTSUPP:
                         # On non-native xattr systems we cannot do this,
@@ -185,75 +238,61 @@
                         pass
                 except:
                     raise
+    except Exception, e:
+        log.error("Failed to upgrade calendar home %s: %s" % (homePath, e))
+        raise
 
-                collectionUpdated = True
+    return errorOccurred
 
 
-        if collectionUpdated:
-            ctagValue = "<?xml version='1.0' encoding='UTF-8'?>\r\n<getctag xmlns='http://calendarserver.org/ns/'>%s</getctag>\r\n" % (str(datetime.datetime.now()),)
-            ctagValue = zlib.compress(ctagValue)
-            try:
-                xattr.setxattr(calPath, xattrname("{http:%2F%2Fcalendarserver.org%2Fns%2F}getctag"), ctagValue)
-            except IOError, ioe:
-                if ioe.errno == errno.EOPNOTSUPP:
-                    # On non-native xattr systems we cannot do this,
-                    # but those systems will typically not be migrating
-                    # from pre-v1
-                    pass
-            except:
-                raise
 
-        return errorOccurred
+class UpgradeOneHome(Command):
+    arguments = [('path', String())]
+    response = [('succeeded', Boolean())]
 
 
-    def upgradeCalendarHome(homePath, directory, cuaCache):
 
-        errorOccurred = False
+class To1Driver(AMP):
+    """
+    Upgrade driver which runs in the parent process.
+    """
 
-        log.debug("Upgrading calendar home: %s" % (homePath,))
+    def upgradeHomeInHelper(self, path):
+        return self.callRemote(UpgradeOneHome, path=path).addCallback(
+            operator.itemgetter("succeeded")
+        )
 
-        try:
-            for cal in os.listdir(homePath):
-                calPath = os.path.join(homePath, cal)
-                if not os.path.isdir(calPath):
-                    # Skip non-directories; these might have been uploaded by a
-                    # random DAV client, they can't be calendar collections.
-                    continue
-                if cal == 'notifications':
-                    # Delete the old, now obsolete, notifications directory.
-                    rmdir(calPath)
-                    continue
-                log.debug("Upgrading calendar: %s" % (calPath,))
-                if not upgradeCalendarCollection(calPath, directory, cuaCache):
-                    errorOccurred = True
 
-                # Change the calendar-free-busy-set xattrs of the inbox to the
-                # __uids__/<guid> form
-                if cal == "inbox":
-                    try:
-                        for attr, value in xattr.xattr(calPath).iteritems():
-                            if attr == xattrname("{urn:ietf:params:xml:ns:caldav}calendar-free-busy-set"):
-                                value = updateFreeBusySet(value, directory)
-                                if value is not None:
-                                    # Need to write the xattr back to disk
-                                    xattr.setxattr(calPath, attr, value)
-                    except IOError, ioe:
-                        if ioe.errno == errno.EOPNOTSUPP:
-                            # On non-native xattr systems we cannot do this,
-                            # but those systems will typically not be migrating
-                            # from pre-v1
-                            pass
-                    except:
-                        raise
 
+class To1Home(AMP):
+    """
+    Upgrade worker which runs in dedicated subprocesses.
+    """
 
-        except Exception, e:
-            log.error("Failed to upgrade calendar home %s: %s" % (homePath, e))
-            raise
+    def __init__(self, config):
+        super(To1Home, self).__init__()
+        self.directory = getDirectory(config)
+        self.cuaCache = {}
 
-        return errorOccurred
 
+    @UpgradeOneHome.responder
+    def upgradeOne(self, path):
+        result = upgradeCalendarHome(path, self.directory, self.cuaCache)
+        return dict(succeeded=result)
 
+
+
+ at inlineCallbacks
+def upgrade_to_1(config, spawner, parallel, directory):
+    """
+    Upconvert data from any calendar server version prior to data format 1.
+    """
+    errorOccurred = []
+    def setError(f=None):
+        if f is not None:
+            log.err(f)
+        errorOccurred.append(True)
+
     def doProxyDatabaseMoveUpgrade(config, uid=-1, gid=-1):
         # See if the new one is already present
         oldFilename = ".db.calendaruserproxy"
@@ -467,6 +506,12 @@
                 os.chown(inboxItemsFile, uid, gid)
 
             if total:
+                if parallel:
+                    spawner.startService()
+                    parallelizer = Parallelizer((yield gatherResults(
+                        [spawner.spawnWithConfig(config, To1Driver(), To1Home)
+                         for x in xrange(parallel)]
+                    )))
                 log.warn("Processing %d calendar homes in %s" % (total, uidHomes))
 
                 # Upgrade calendar homes in the new location:
@@ -484,15 +529,30 @@
                                         # Skip non-directories
                                         continue
 
-                                    if not upgradeCalendarHome(homePath,
-                                        directory, cuaCache):
-                                        errorOccurred = True
+                                    if parallel:
+                                        def doIt(driver, hp=homePath):
+                                            d = driver.upgradeHomeInHelper(hp)
+                                            def itWorked(succeeded):
+                                                if not succeeded:
+                                                    setError()
+                                                return succeeded
+                                            d.addCallback(itWorked)
+                                            d.addErrback(setError)
+                                            return d
+                                        yield parallelizer.do(doIt)
+                                    else:
+                                        if not upgradeCalendarHome(
+                                            homePath, directory, cuaCache
+                                        ):
+                                            setError()
 
                                     count += 1
                                     if count % 10 == 0:
                                         log.warn("Processed calendar home %d of %d"
                                             % (count, total))
-
+                if parallel:
+                    yield parallelizer.done()
+                    yield spawner.stopService()
                 log.warn("Done processing calendar homes")
 
     yield migrateResourceInfo(config, directory, uid, gid)
@@ -522,7 +582,6 @@
     cal = Component.fromString(data)
 
     def lookupFunction(cuaddr):
-
         # Return cached results, if any.
         if cuaCache.has_key(cuaddr):
             return cuaCache[cuaddr]
@@ -559,7 +618,7 @@
 
 
 @inlineCallbacks
-def upgrade_to_2(config, directory):
+def upgrade_to_2(config, spawner, parallel, directory):
     
     errorOccurred = False
 
@@ -675,7 +734,7 @@
 ]
 
 @inlineCallbacks
-def upgradeData(config):
+def upgradeData(config, spawner=None, parallel=0):
 
     directory = getDirectory()
 
@@ -707,7 +766,7 @@
     for version, method in upgradeMethods:
         if onDiskVersion < version:
             log.warn("Upgrading to version %d" % (version,))
-            (yield method(config, directory))
+            (yield method(config, spawner, parallel, directory))
             log.warn("Upgraded to version %d" % (version,))
             with open(versionFilePath, "w") as verFile:
                 verFile.write(str(version))
@@ -899,12 +958,14 @@
     Upgrade filesystem from previous versions.
     """
 
-    def __init__(self, config, service):
+    def __init__(self, config, spawner, parallel, service):
         """
         Initialize the service.
         """
         self.wrappedService = service
         self.config = config
+        self.spawner = spawner
+        self.parallel = parallel
 
 
     @inlineCallbacks
@@ -921,7 +982,7 @@
         memcacheEnabled = self.config.Memcached.Pools.Default.ClientEnabled
         self.config.Memcached.Pools.Default.ClientEnabled = False
 
-        yield upgradeData(self.config)
+        yield upgradeData(self.config, self.spawner, self.parallel)
 
         # Restore memcached client setting
         self.config.Memcached.Pools.Default.ClientEnabled = memcacheEnabled


Property changes on: CalendarServer/trunk/txdav/caldav/datastore/index_file.py
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalendarServer/branches/config-separation/txdav/caldav/datastore/index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/caldav/datastore/index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/caldav/datastore/index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/caldav/datastore/index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/caldav/datastore/index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/caldav/datastore/index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/caldav/datastore/index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/caldav/datastore/index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/caldav/datastore/index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/caldav/datastore/index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace/txdav/caldav/datastore/index_file.py:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/caldav/datastore/index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/caldav/datastore/index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/caldav/datastore/index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/caldav/datastore/index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/caldav/datastore/index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/caldav/datastore/index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/caldav/datastore/index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/caldav/datastore/index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/caldav/datastore/index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/caldav/datastore/index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/caldav/datastore/index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/caldav/datastore/index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/caldav/datastore/index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/caldav/datastore/index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/caldav/datastore/index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/caldav/datastore/index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/caldav/datastore/index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/caldav/datastore/index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/caldav/datastore/index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/caldav/datastore/index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/caldav/datastore/index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/caldav/datastore/index_file.py:8376-8400
/CalendarServer/branches/users/glyph/sendfdport/txdav/caldav/datastore/index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/caldav/datastore/index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/caldav/datastore/index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/caldav/datastore/index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/caldav/datastore/index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/caldav/datastore/index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/caldav/datastore/index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/caldav/datastore/index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/caldav/datastore/index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/caldav/datastore/index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/caldav/datastore/index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/caldav/datastore/index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/caldav/datastore/index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/caldav/datastore/index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/caldav/datastore/index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/caldav/datastore/index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/index.py:6322-6394
   + /CalendarServer/branches/config-separation/txdav/caldav/datastore/index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/caldav/datastore/index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/caldav/datastore/index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/caldav/datastore/index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/caldav/datastore/index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/caldav/datastore/index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/caldav/datastore/index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/caldav/datastore/index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/caldav/datastore/index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/caldav/datastore/index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace/txdav/caldav/datastore/index_file.py:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/caldav/datastore/index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/caldav/datastore/index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/caldav/datastore/index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/caldav/datastore/index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/caldav/datastore/index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/caldav/datastore/index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/caldav/datastore/index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/caldav/datastore/index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/caldav/datastore/index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/caldav/datastore/index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/caldav/datastore/index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/caldav/datastore/index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/caldav/datastore/index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/caldav/datastore/index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/caldav/datastore/index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/caldav/datastore/index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/caldav/datastore/index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/caldav/datastore/index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/caldav/datastore/index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/caldav/datastore/index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/caldav/datastore/index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/caldav/datastore/index_file.py:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/txdav/caldav/datastore/index_file.py:8571-8583
/CalendarServer/branches/users/glyph/sendfdport/txdav/caldav/datastore/index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/caldav/datastore/index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/caldav/datastore/index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/caldav/datastore/index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/caldav/datastore/index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/caldav/datastore/index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/caldav/datastore/index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/caldav/datastore/index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/caldav/datastore/index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/caldav/datastore/index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/caldav/datastore/index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/caldav/datastore/index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/caldav/datastore/index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/caldav/datastore/index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/caldav/datastore/index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/caldav/datastore/index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/index.py:6322-6394


Property changes on: CalendarServer/trunk/txdav/caldav/datastore/test/test_index_file.py
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalendarServer/branches/config-separation/txdav/caldav/datastore/test/test_index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/caldav/datastore/test/test_index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/caldav/datastore/test/test_index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/caldav/datastore/test/test_index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/caldav/datastore/test/test_index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/caldav/datastore/test/test_index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/caldav/datastore/test/test_index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/caldav/datastore/test/test_index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/caldav/datastore/test/test_index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/caldav/datastore/test/test_index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/caldav/datastore/test/test_index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/caldav/datastore/test/test_index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/caldav/datastore/test/test_index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/caldav/datastore/test/test_index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/caldav/datastore/test/test_index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/caldav/datastore/test/test_index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/caldav/datastore/test/test_index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/caldav/datastore/test/test_index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/caldav/datastore/test/test_index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/caldav/datastore/test/test_index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/caldav/datastore/test/test_index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/caldav/datastore/test/test_index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/caldav/datastore/test/test_index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/caldav/datastore/test/test_index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/caldav/datastore/test/test_index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/caldav/datastore/test/test_index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/caldav/datastore/test/test_index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/caldav/datastore/test/test_index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/caldav/datastore/test/test_index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/caldav/datastore/test/test_index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/caldav/datastore/test/test_index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/caldav/datastore/test/test_index_file.py:8376-8400
/CalendarServer/branches/users/glyph/sendfdport/txdav/caldav/datastore/test/test_index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/caldav/datastore/test/test_index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/caldav/datastore/test/test_index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/caldav/datastore/test/test_index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/caldav/datastore/test/test_index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/caldav/datastore/test/test_index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/caldav/datastore/test/test_index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/caldav/datastore/test/test_index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/caldav/datastore/test/test_index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/caldav/datastore/test/test_index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/caldav/datastore/test/test_index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/caldav/datastore/test/test_index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/caldav/datastore/test/test_index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/caldav/datastore/test/test_index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/caldav/datastore/test/test_index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/caldav/datastore/test/test_index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/test/test_index.py:6322-6394
   + /CalendarServer/branches/config-separation/txdav/caldav/datastore/test/test_index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/caldav/datastore/test/test_index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/caldav/datastore/test/test_index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/caldav/datastore/test/test_index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/caldav/datastore/test/test_index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/caldav/datastore/test/test_index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/caldav/datastore/test/test_index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/caldav/datastore/test/test_index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/caldav/datastore/test/test_index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/caldav/datastore/test/test_index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/caldav/datastore/test/test_index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/caldav/datastore/test/test_index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/caldav/datastore/test/test_index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/caldav/datastore/test/test_index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/caldav/datastore/test/test_index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/caldav/datastore/test/test_index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/caldav/datastore/test/test_index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/caldav/datastore/test/test_index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/caldav/datastore/test/test_index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/caldav/datastore/test/test_index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/caldav/datastore/test/test_index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/caldav/datastore/test/test_index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/caldav/datastore/test/test_index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/caldav/datastore/test/test_index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/caldav/datastore/test/test_index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/caldav/datastore/test/test_index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/caldav/datastore/test/test_index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/caldav/datastore/test/test_index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/caldav/datastore/test/test_index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/caldav/datastore/test/test_index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/caldav/datastore/test/test_index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/caldav/datastore/test/test_index_file.py:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/txdav/caldav/datastore/test/test_index_file.py:8571-8583
/CalendarServer/branches/users/glyph/sendfdport/txdav/caldav/datastore/test/test_index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/caldav/datastore/test/test_index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/caldav/datastore/test/test_index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/caldav/datastore/test/test_index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/caldav/datastore/test/test_index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/caldav/datastore/test/test_index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/caldav/datastore/test/test_index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/caldav/datastore/test/test_index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/caldav/datastore/test/test_index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/caldav/datastore/test/test_index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/caldav/datastore/test/test_index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/caldav/datastore/test/test_index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/caldav/datastore/test/test_index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/caldav/datastore/test/test_index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/caldav/datastore/test/test_index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/caldav/datastore/test/test_index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/test/test_index.py:6322-6394


Property changes on: CalendarServer/trunk/txdav/carddav/datastore/index_file.py
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalendarServer/branches/config-separation/txdav/carddav/datastore/index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/carddav/datastore/index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/carddav/datastore/index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/carddav/datastore/index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/carddav/datastore/index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/carddav/datastore/index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/carddav/datastore/index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/carddav/datastore/index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/carddav/datastore/index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/carddav/datastore/index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace/txdav/carddav/datastore/index_file.py:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/carddav/datastore/index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/carddav/datastore/index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/carddav/datastore/index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/carddav/datastore/index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/carddav/datastore/index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/carddav/datastore/index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/carddav/datastore/index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/carddav/datastore/index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/carddav/datastore/index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/carddav/datastore/index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/carddav/datastore/index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/carddav/datastore/index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/carddav/datastore/index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/carddav/datastore/index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/carddav/datastore/index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/carddav/datastore/index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/carddav/datastore/index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/carddav/datastore/index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/carddav/datastore/index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/carddav/datastore/index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/carddav/datastore/index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/carddav/datastore/index_file.py:8376-8400
/CalendarServer/branches/users/glyph/sendfdport/txdav/carddav/datastore/index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/carddav/datastore/index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/carddav/datastore/index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/carddav/datastore/index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/carddav/datastore/index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/carddav/datastore/index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/carddav/datastore/index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/carddav/datastore/index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/carddav/datastore/index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/carddav/datastore/index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/carddav/datastore/index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/carddav/datastore/index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/carddav/datastore/index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/carddav/datastore/index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/carddav/datastore/index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/carddav/datastore/index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/vcardindex.py:6322-6394
   + /CalendarServer/branches/config-separation/txdav/carddav/datastore/index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/carddav/datastore/index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/carddav/datastore/index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/carddav/datastore/index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/carddav/datastore/index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/carddav/datastore/index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/carddav/datastore/index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/carddav/datastore/index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/carddav/datastore/index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/carddav/datastore/index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/implicituidrace/txdav/carddav/datastore/index_file.py:8137-8141
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/carddav/datastore/index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/carddav/datastore/index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/carddav/datastore/index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/carddav/datastore/index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/carddav/datastore/index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/carddav/datastore/index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/carddav/datastore/index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/carddav/datastore/index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/carddav/datastore/index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/carddav/datastore/index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/carddav/datastore/index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/carddav/datastore/index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/carddav/datastore/index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/carddav/datastore/index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/carddav/datastore/index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/carddav/datastore/index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/carddav/datastore/index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/carddav/datastore/index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/carddav/datastore/index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/carddav/datastore/index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/carddav/datastore/index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/carddav/datastore/index_file.py:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/txdav/carddav/datastore/index_file.py:8571-8583
/CalendarServer/branches/users/glyph/sendfdport/txdav/carddav/datastore/index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/carddav/datastore/index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/carddav/datastore/index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/carddav/datastore/index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/carddav/datastore/index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/carddav/datastore/index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/carddav/datastore/index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/carddav/datastore/index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/carddav/datastore/index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/carddav/datastore/index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/carddav/datastore/index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/carddav/datastore/index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/carddav/datastore/index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/carddav/datastore/index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/carddav/datastore/index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/carddav/datastore/index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/vcardindex.py:6322-6394


Property changes on: CalendarServer/trunk/txdav/carddav/datastore/test/test_index_file.py
___________________________________________________________________
Modified: svn:mergeinfo
   - /CalendarServer/branches/config-separation/txdav/carddav/datastore/test/test_index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/carddav/datastore/test/test_index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/carddav/datastore/test/test_index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/carddav/datastore/test/test_index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/carddav/datastore/test/test_index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/carddav/datastore/test/test_index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/carddav/datastore/test/test_index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/carddav/datastore/test/test_index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/carddav/datastore/test/test_index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/carddav/datastore/test/test_index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/carddav/datastore/test/test_index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/carddav/datastore/test/test_index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/carddav/datastore/test/test_index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/carddav/datastore/test/test_index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/carddav/datastore/test/test_index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/carddav/datastore/test/test_index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/carddav/datastore/test/test_index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/carddav/datastore/test/test_index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/carddav/datastore/test/test_index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/carddav/datastore/test/test_index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/carddav/datastore/test/test_index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/carddav/datastore/test/test_index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/carddav/datastore/test/test_index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/carddav/datastore/test/test_index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/carddav/datastore/test/test_index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/carddav/datastore/test/test_index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/carddav/datastore/test/test_index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/carddav/datastore/test/test_index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/carddav/datastore/test/test_index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/carddav/datastore/test/test_index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/carddav/datastore/test/test_index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/carddav/datastore/test/test_index_file.py:8376-8400
/CalendarServer/branches/users/glyph/sendfdport/txdav/carddav/datastore/test/test_index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/carddav/datastore/test/test_index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/carddav/datastore/test/test_index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/carddav/datastore/test/test_index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/carddav/datastore/test/test_index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/carddav/datastore/test/test_index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/carddav/datastore/test/test_index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/carddav/datastore/test/test_index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/carddav/datastore/test/test_index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/carddav/datastore/test/test_index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/carddav/datastore/test/test_index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/carddav/datastore/test/test_index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/carddav/datastore/test/test_index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/carddav/datastore/test/test_index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/carddav/datastore/test/test_index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/carddav/datastore/test/test_index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/test/test_vcardindex.py:6322-6394
   + /CalendarServer/branches/config-separation/txdav/carddav/datastore/test/test_index_file.py:4379-4443
/CalendarServer/branches/egg-info-351/txdav/carddav/datastore/test/test_index_file.py:4589-4625
/CalendarServer/branches/generic-sqlstore/txdav/carddav/datastore/test/test_index_file.py:6167-6191
/CalendarServer/branches/new-store/txdav/carddav/datastore/test/test_index_file.py:5594-5934
/CalendarServer/branches/new-store-no-caldavfile/txdav/carddav/datastore/test/test_index_file.py:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2/txdav/carddav/datastore/test/test_index_file.py:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699/txdav/carddav/datastore/test/test_index_file.py:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692/txdav/carddav/datastore/test/test_index_file.py:5693-5702
/CalendarServer/branches/users/cdaboo/component-set-fixes/txdav/carddav/datastore/test/test_index_file.py:8130-8346
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627/txdav/carddav/datastore/test/test_index_file.py:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591/txdav/carddav/datastore/test/test_index_file.py:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464/txdav/carddav/datastore/test/test_index_file.py:4465-4957
/CalendarServer/branches/users/cdaboo/pods/txdav/carddav/datastore/test/test_index_file.py:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar/txdav/carddav/datastore/test/test_index_file.py:7085-7206
/CalendarServer/branches/users/cdaboo/pycard/txdav/carddav/datastore/test/test_index_file.py:7227-7237
/CalendarServer/branches/users/cdaboo/queued-attendee-refreshes/txdav/carddav/datastore/test/test_index_file.py:7740-8287
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070/txdav/carddav/datastore/test/test_index_file.py:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187/txdav/carddav/datastore/test/test_index_file.py:5188-5440
/CalendarServer/branches/users/cdaboo/timezones/txdav/carddav/datastore/test/test_index_file.py:7443-7699
/CalendarServer/branches/users/glyph/conn-limit/txdav/carddav/datastore/test/test_index_file.py:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge/txdav/carddav/datastore/test/test_index_file.py:4971-5080
/CalendarServer/branches/users/glyph/dalify/txdav/carddav/datastore/test/test_index_file.py:6932-7023
/CalendarServer/branches/users/glyph/deploybuild/txdav/carddav/datastore/test/test_index_file.py:7563-7572
/CalendarServer/branches/users/glyph/dont-start-postgres/txdav/carddav/datastore/test/test_index_file.py:6592-6614
/CalendarServer/branches/users/glyph/linux-tests/txdav/carddav/datastore/test/test_index_file.py:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes/txdav/carddav/datastore/test/test_index_file.py:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6/txdav/carddav/datastore/test/test_index_file.py:6322-6334
/CalendarServer/branches/users/glyph/more-deferreds-7/txdav/carddav/datastore/test/test_index_file.py:6369
/CalendarServer/branches/users/glyph/new-export/txdav/carddav/datastore/test/test_index_file.py:7444-7485
/CalendarServer/branches/users/glyph/oracle/txdav/carddav/datastore/test/test_index_file.py:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls/txdav/carddav/datastore/test/test_index_file.py:7340-7351
/CalendarServer/branches/users/glyph/parallel-upgrade/txdav/carddav/datastore/test/test_index_file.py:8376-8400
/CalendarServer/branches/users/glyph/parallel-upgrade_to_1/txdav/carddav/datastore/test/test_index_file.py:8571-8583
/CalendarServer/branches/users/glyph/sendfdport/txdav/carddav/datastore/test/test_index_file.py:5388-5424
/CalendarServer/branches/users/glyph/shared-pool-fixes/txdav/carddav/datastore/test/test_index_file.py:8436-8443
/CalendarServer/branches/users/glyph/sharedpool/txdav/carddav/datastore/test/test_index_file.py:6490-6550
/CalendarServer/branches/users/glyph/skip-lonely-vtimezones/txdav/carddav/datastore/test/test_index_file.py:8524-8535
/CalendarServer/branches/users/glyph/sql-store/txdav/carddav/datastore/test/test_index_file.py:5929-6073
/CalendarServer/branches/users/glyph/subtransactions/txdav/carddav/datastore/test/test_index_file.py:7248-7258
/CalendarServer/branches/users/glyph/use-system-twisted/txdav/carddav/datastore/test/test_index_file.py:5084-5149
/CalendarServer/branches/users/sagen/applepush/txdav/carddav/datastore/test/test_index_file.py:8126-8184
/CalendarServer/branches/users/sagen/inboxitems/txdav/carddav/datastore/test/test_index_file.py:7380-7381
/CalendarServer/branches/users/sagen/locations-resources/txdav/carddav/datastore/test/test_index_file.py:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2/txdav/carddav/datastore/test/test_index_file.py:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events/txdav/carddav/datastore/test/test_index_file.py:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038/txdav/carddav/datastore/test/test_index_file.py:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066/txdav/carddav/datastore/test/test_index_file.py:4068-4075
/CalendarServer/branches/users/sagen/resources-2/txdav/carddav/datastore/test/test_index_file.py:5084-5093
/CalendarServer/branches/users/wsanchez/transations/txdav/carddav/datastore/test/test_index_file.py:5515-5593
/CalendarServer/trunk/twistedcaldav/test/test_vcardindex.py:6322-6394

Modified: CalendarServer/trunk/txdav/common/datastore/upgrade/migrate.py
===================================================================
--- CalendarServer/trunk/txdav/common/datastore/upgrade/migrate.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/txdav/common/datastore/upgrade/migrate.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -32,9 +32,10 @@
 from twisted.application.service import Service
 from twisted.internet import reactor
 from twisted.internet.defer import inlineCallbacks, returnValue
-from twisted.internet.defer import maybeDeferred, DeferredList
+from twisted.internet.defer import maybeDeferred, gatherResults
 
 from twext.python.filepath import CachingFilePath
+from twext.python.parallel import Parallelizer
 from twext.internet.spawnsvc import SpawnerService
 
 from twisted.protocols.amp import AMP, Command, String
@@ -91,7 +92,16 @@
         raise NotImplementedError("subclasses must implement the specifics")
 
 
+    def spawnWithConfig(self, config, here, there):
+        """
+        Like L{SpawnerService.spawn}, but instead of instantiating C{there}
+        with 0 arguments, it instantiates it with the given
+        L{twistedcaldav.config.Config}.
+        """
+        raise NotImplementedError("subclasses must implement the specifics")
 
+
+
 class Configure(Command):
     """
     Configure the upgrade helper process.
@@ -161,8 +171,8 @@
         super(UpgradeHelperProcess, self).__init__()
         self.store = store
         self.store.setMigrating(True)
-        
 
+
     @Configure.responder
     def configure(self, filename, appropriateStoreClass):
         subsvc = None
@@ -339,12 +349,11 @@
                           (parallel,))
             spawner = self.spawner
             spawner.startService()
-            drivers = []
-            for value in xrange(parallel):
-                driver = yield spawner.spawnWithStore(UpgradeDriver(self),
-                                                      UpgradeHelperProcess)
-                drivers.append(driver)
-
+            drivers = yield gatherResults(
+                [spawner.spawnWithStore(UpgradeDriver(self),
+                                        UpgradeHelperProcess)
+                 for x in xrange(parallel)]
+            )
             # Wait for all subprocesses to be fully configured before
             # continuing, but let them configure in any order.
             self.log_warn("Configuring upgrade helper processes.")
@@ -356,13 +365,15 @@
             # know the intimate details of the fileStore implementation.
             # (Alternately, wrapService could just hold on to the details that
             # it used to construct the service in the first place.)
-            yield DeferredList([driver.configure(self.fileStore._path.path,
-                                                 self.fileStore._propertyStoreClass)
-                                for driver in drivers])
+            yield gatherResults(
+                [driver.configure(self.fileStore._path.path,
+                                  self.fileStore._propertyStoreClass)
+                 for driver in drivers]
+            )
             self.log_warn("Upgrade helpers ready.")
+            parallelizer = Parallelizer(drivers)
 
         self.log_warn("Beginning filesystem -> database upgrade.")
-        inParallel = []
         for homeType, eachFunc in [
                 ("calendar", self.fileStore.eachCalendarHome),
                 ("addressbook", self.fileStore.eachAddressbookHome),
@@ -374,27 +385,17 @@
                     # No-op transaction here: make sure everything's unlocked
                     # before asking the subprocess to handle it.
                     yield fileTxn.commit()
-                    if not drivers:
-                        # All the subprocesses are currently busy processing an
-                        # upgrade.  Wait for one to become available.
-                        yield DeferredList(inParallel, fireOnOneCallback=True,
-                                           fireOnOneErrback=True)
-                    busy = drivers.pop(0)
-                    d = busy.oneUpgrade(fileHome.uid(), homeType)
-                    inParallel.append(d)
-                    def freeUp(result, d=d, busy=busy, uid=uid,
-                               homeType=homeType):
-                        inParallel.remove(d)
-                        drivers.append(busy)
+                    @inlineCallbacks
+                    def doOneUpgrade(driver, fileUID=uid):
+                        yield driver.oneUpgrade(fileUID, homeType)
                         self.log_warn("Completed migration of %s uid %r" %
-                                      (homeType, uid))
-                        return result
-                    d.addBoth(freeUp)
+                                      (homeType, fileUID))
+                    yield parallelizer.do(doOneUpgrade)
                 else:
                     yield self.migrateOneHome(fileTxn, homeType, fileHome)
 
-        if inParallel:
-            yield DeferredList(inParallel)
+        if parallel:
+            yield parallelizer.done()
 
         for homeType in TOPPATHS:
             homesPath = self.fileStore._path.child(homeType)

Modified: CalendarServer/trunk/txdav/common/datastore/upgrade/test/test_migrate.py
===================================================================
--- CalendarServer/trunk/txdav/common/datastore/upgrade/test/test_migrate.py	2012-01-24 09:00:02 UTC (rev 8583)
+++ CalendarServer/trunk/txdav/common/datastore/upgrade/test/test_migrate.py	2012-01-24 09:11:15 UTC (rev 8584)
@@ -20,6 +20,7 @@
 
 from twext.python.filepath import CachingFilePath
 from twext.web2.http_headers import MimeType
+from twext.enterprise.adbapi2 import Pickle
 
 from twisted.python.modules import getModule
 from twisted.application.service import Service, MultiService
@@ -28,6 +29,8 @@
 from twisted.protocols.amp import AMP, Command, String
 from twisted.python.reflect import qual, namedAny
 from twisted.trial.unittest import TestCase
+
+
 from txdav.caldav.datastore.test.common import CommonTests
 from txdav.carddav.datastore.test.common import CommonTests as ABCommonTests
 from txdav.common.datastore.file import CommonDataStore
@@ -50,7 +53,15 @@
     arguments = [('delegateTo', String())]
 
 
+class PickleConfig(Command):
+    """
+    Unpickle some configuration in a subprocess.
+    """
+    arguments = [('delegateTo', String()),
+                 ('config', Pickle())]
 
+
+
 class StoreCreator(AMP):
     """
     Helper protocol.
@@ -65,25 +76,49 @@
         return {}
 
 
+    @PickleConfig.responder
+    def pickleConfig(self, config, delegateTo):
+        #from twistedcaldav.config import config as globalConfig
+        #globalConfig._data = config._data
+        swapAMP(self, namedAny(delegateTo)(config))
+        return {}
 
+
+
 class StubSpawner(StoreSpawnerService):
     """
     Stub spawner service which populates the store forcibly.
     """
 
+    def __init__(self, config=None):
+        super(StubSpawner, self).__init__()
+        self.config = config
+
+
     @inlineCallbacks
     def spawnWithStore(self, here, there):
         """
-        'here' and 'there' are the helper protocols; in a slight modification
-        of the signature, 'there' will expect to be created with an instance of
-        a store.
+        'here' and 'there' are the helper protocols 'there' will expect to be
+        created with an instance of a store.
         """
         master = yield self.spawn(AMP(), StoreCreator)
         yield master.callRemote(CreateStore, delegateTo=qual(there))
         returnValue(swapAMP(master, here))
 
 
+    @inlineCallbacks
+    def spawnWithConfig(self, config, here, there):
+        """
+        Similar to spawnWithStore except the child process gets a configuration
+        object instead.
+        """
+        master = yield self.spawn(AMP(), StoreCreator)
+        yield master.callRemote(PickleConfig, config=self.config,
+                                delegateTo=qual(there))
+        returnValue(swapAMP(master, here))
 
+
+
 class HomeMigrationTests(TestCase):
     """
     Tests for L{UpgradeToDatabaseService}.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/calendarserver-changes/attachments/20120124/2c75630b/attachment-0001.html>

More information about the calendarserver-changes mailing list