[CalendarServer-changes] [6067] CalendarServer/branches/users/glyph/sql-store
source_changes at macosforge.org
source_changes at macosforge.org
Wed Aug 11 17:05:15 PDT 2010
Revision: 6067
http://trac.macosforge.org/projects/calendarserver/changeset/6067
Author: glyph at apple.com
Date: 2010-08-11 17:05:13 -0700 (Wed, 11 Aug 2010)
Log Message:
-----------
trunk integration
Modified Paths:
--------------
CalendarServer/branches/users/glyph/sql-store/setup.py
CalendarServer/branches/users/glyph/sql-store/support/Makefile.Apple
CalendarServer/branches/users/glyph/sql-store/support/build.sh
CalendarServer/branches/users/glyph/sql-store/support/submit
CalendarServer/branches/users/glyph/sql-store/twext/web2/resource.py
Added Paths:
-----------
CalendarServer/branches/users/glyph/sql-store/contrib/performance/
CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh
CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py
CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d
CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py
Removed Paths:
-------------
CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh
CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py
CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d
CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py
Property Changed:
----------------
CalendarServer/branches/users/glyph/sql-store/
Property changes on: CalendarServer/branches/users/glyph/sql-store
___________________________________________________________________
Modified: svn:mergeinfo
- /CalendarServer/branches/config-separation:4379-4443
/CalendarServer/branches/egg-info-351:4589-4625
/CalendarServer/branches/new-store:5594-5934
/CalendarServer/branches/new-store-no-caldavfile:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2:5936-5981
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692:5693-5702
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464:4465-4957
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187:5188-5440
/CalendarServer/branches/users/glyph/contacts-server-merge:4971-5080
/CalendarServer/branches/users/glyph/sendfdport:5388-5424
/CalendarServer/branches/users/glyph/use-system-twisted:5084-5149
/CalendarServer/branches/users/sagen/locations-resources:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2:5052-5061
/CalendarServer/branches/users/sagen/resource-delegates-4038:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066:4068-4075
/CalendarServer/branches/users/sagen/resources-2:5084-5093
/CalendarServer/branches/users/wsanchez/transations:5515-5593
/CalendarServer/trunk:5929-6058
+ /CalendarServer/branches/config-separation:4379-4443
/CalendarServer/branches/egg-info-351:4589-4625
/CalendarServer/branches/new-store:5594-5934
/CalendarServer/branches/new-store-no-caldavfile:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2:5936-5981
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692:5693-5702
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464:4465-4957
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187:5188-5440
/CalendarServer/branches/users/glyph/contacts-server-merge:4971-5080
/CalendarServer/branches/users/glyph/sendfdport:5388-5424
/CalendarServer/branches/users/glyph/use-system-twisted:5084-5149
/CalendarServer/branches/users/sagen/locations-resources:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2:5052-5061
/CalendarServer/branches/users/sagen/resource-delegates-4038:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066:4068-4075
/CalendarServer/branches/users/sagen/resources-2:5084-5093
/CalendarServer/branches/users/wsanchez/transations:5515-5593
/CalendarServer/trunk:5929-6066
Deleted: CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh
===================================================================
--- CalendarServer/trunk/contrib/performance/benchmark.sh 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh 2010-08-12 00:05:13 UTC (rev 6067)
@@ -1,2 +0,0 @@
-#!/bin/bash
-sudo PYTHONPATH=$PYTHONPATH:../../../vobject/:../../../../CalDAVClientLibrary/trunk/src:../../../Twisted/ python mkcal.py 5466 5465
Copied: CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh (from rev 6066, CalendarServer/trunk/contrib/performance/benchmark.sh)
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh (rev 0)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/benchmark.sh 2010-08-12 00:05:13 UTC (rev 6067)
@@ -0,0 +1,2 @@
+#!/bin/bash
+sudo PYTHONPATH=$PYTHONPATH:../../../vobject/:../../../../CalDAVClientLibrary/trunk/src:../../../Twisted/ python mkcal.py 5466 5465
Deleted: CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py
===================================================================
--- CalendarServer/trunk/contrib/performance/httpauth.py 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py 2010-08-12 00:05:13 UTC (rev 6067)
@@ -1,86 +0,0 @@
-import shlex
-
-from twisted.web.http_headers import Headers
-
-class BasicChallenge(object):
- def __init__(self, realm):
- self.realm = realm
-
-
- def response(self, uri, keyring):
- username, password = keyring.passwd.find_user_password(self.realm, uri)
- credentials = ('%s:%s' % (username, password)).encode('base64').strip()
- authorization = 'basic ' + credentials
- return {'authorization': [authorization]}
-
-
-
-class AuthHandlerAgent(object):
- def __init__(self, agent, authinfo):
- self._agent = agent
- self._authinfo = authinfo
-
-
- def request(self, method, uri, headers=None, bodyProducer=None):
- d = self._agent.request(method, uri, headers, bodyProducer)
- d.addCallback(self._authenticate, method, uri, headers, bodyProducer)
- return d
-
-
- def _parse(self, authorization):
- parts = shlex.split(authorization)
- scheme = parts.pop(0)
- args = dict([p.split('=', 1) for p in parts])
- if scheme == 'basic':
- return BasicChallenge(**args)
- return None
-
-
- def _authenticate(self, response, method, uri, headers, bodyProducer):
- if response.code == 401:
- # Look for a challenge
- authorization = response.headers.getRawHeaders('www-authenticate')
- if authorization is None:
- raise Exception("401 response with no WWW-Authenticate header")
-
- for auth in authorization:
- challenge = self._parse(auth)
- if challenge is None:
- continue
-
- if headers is None:
- headers = Headers()
- else:
- headers = Headers(dict(headers.getAllRawHeaders()))
- for k, vs in challenge.response(uri, self._authinfo).iteritems():
- for v in vs:
- headers.addRawHeader(k, v)
-
- return self._agent.request(method, uri, headers, bodyProducer)
-
- return response
-
-
-if __name__ == '__main__':
- from urllib2 import HTTPDigestAuthHandler
- handler = HTTPDigestAuthHandler()
- handler.add_password(
- realm="Test Realm",
- uri="http://localhost:8008/",
- user="user01",
- passwd="user01")
-
- from twisted.web.client import Agent
- from twisted.internet import reactor
- from twisted.python.log import err
- agent = AuthHandlerAgent(Agent(reactor), handler)
- d = agent.request(
- 'DELETE', 'http://localhost:8008/calendars/users/user01/monkeys3/')
- def deleted(response):
- print response.code
- print response.headers
- reactor.stop()
- d.addCallback(deleted)
- d.addErrback(err)
- d.addCallback(lambda ign: reactor.stop())
- reactor.run()
Copied: CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py (from rev 6066, CalendarServer/trunk/contrib/performance/httpauth.py)
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py (rev 0)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/httpauth.py 2010-08-12 00:05:13 UTC (rev 6067)
@@ -0,0 +1,86 @@
+import shlex
+
+from twisted.web.http_headers import Headers
+
+class BasicChallenge(object):
+ def __init__(self, realm):
+ self.realm = realm
+
+
+ def response(self, uri, keyring):
+ username, password = keyring.passwd.find_user_password(self.realm, uri)
+ credentials = ('%s:%s' % (username, password)).encode('base64').strip()
+ authorization = 'basic ' + credentials
+ return {'authorization': [authorization]}
+
+
+
+class AuthHandlerAgent(object):
+ def __init__(self, agent, authinfo):
+ self._agent = agent
+ self._authinfo = authinfo
+
+
+ def request(self, method, uri, headers=None, bodyProducer=None):
+ d = self._agent.request(method, uri, headers, bodyProducer)
+ d.addCallback(self._authenticate, method, uri, headers, bodyProducer)
+ return d
+
+
+ def _parse(self, authorization):
+ parts = shlex.split(authorization)
+ scheme = parts.pop(0)
+ args = dict([p.split('=', 1) for p in parts])
+ if scheme == 'basic':
+ return BasicChallenge(**args)
+ return None
+
+
+ def _authenticate(self, response, method, uri, headers, bodyProducer):
+ if response.code == 401:
+ # Look for a challenge
+ authorization = response.headers.getRawHeaders('www-authenticate')
+ if authorization is None:
+ raise Exception("401 response with no WWW-Authenticate header")
+
+ for auth in authorization:
+ challenge = self._parse(auth)
+ if challenge is None:
+ continue
+
+ if headers is None:
+ headers = Headers()
+ else:
+ headers = Headers(dict(headers.getAllRawHeaders()))
+ for k, vs in challenge.response(uri, self._authinfo).iteritems():
+ for v in vs:
+ headers.addRawHeader(k, v)
+
+ return self._agent.request(method, uri, headers, bodyProducer)
+
+ return response
+
+
+if __name__ == '__main__':
+ from urllib2 import HTTPDigestAuthHandler
+ handler = HTTPDigestAuthHandler()
+ handler.add_password(
+ realm="Test Realm",
+ uri="http://localhost:8008/",
+ user="user01",
+ passwd="user01")
+
+ from twisted.web.client import Agent
+ from twisted.internet import reactor
+ from twisted.python.log import err
+ agent = AuthHandlerAgent(Agent(reactor), handler)
+ d = agent.request(
+ 'DELETE', 'http://localhost:8008/calendars/users/user01/monkeys3/')
+ def deleted(response):
+ print response.code
+ print response.headers
+ reactor.stop()
+ d.addCallback(deleted)
+ d.addErrback(err)
+ d.addCallback(lambda ign: reactor.stop())
+ reactor.run()
Deleted: CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d
===================================================================
--- CalendarServer/trunk/contrib/performance/io_measure.d 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d 2010-08-12 00:05:13 UTC (rev 6067)
@@ -1,56 +0,0 @@
-
-/*
- * Report current timestamp (nanoseconds) of io and SQLite3 events.
- */
-
-#pragma D option switchrate=10hz
-
-/*
- * Low-level I/O stuff
- */
-
-io:::start
-/args[0]->b_flags & B_READ/
-{
- printf("%d", args[0]->b_bcount);
-}
-
-io:::start
-/!(args[0]->b_flags & B_READ)/
-{
- printf("%d", args[0]->b_bcount);
-}
-
-/*
- * SQLite3 stuff
- */
-
-pid$target:_sqlite3.so:_pysqlite_query_execute:entry
-{
- self->executing = 1;
- self->sql = "";
- printf("%d", timestamp);
-}
-
-pid$target:_sqlite3.so:_pysqlite_query_execute:return
-{
- self->executing = 0;
- printf("%d %s", timestamp, self->sql);
-}
-
-pid$target::PyString_AsString:return
-/self->executing/
-{
- self->sql = copyinstr(arg1);
- self->executing = 0;
-}
-
-pid$target:_sqlite3.so:pysqlite_cursor_iternext:entry
-{
- printf("%d", timestamp);
-}
-
-pid$target:_sqlite3.so:pysqlite_cursor_iternext:return
-{
- printf("%d", timestamp);
-}
Copied: CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d (from rev 6066, CalendarServer/trunk/contrib/performance/io_measure.d)
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d (rev 0)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/io_measure.d 2010-08-12 00:05:13 UTC (rev 6067)
@@ -0,0 +1,56 @@
+
+/*
+ * Report current timestamp (nanoseconds) of io and SQLite3 events.
+ */
+
+#pragma D option switchrate=10hz
+
+/*
+ * Low-level I/O stuff
+ */
+
+io:::start
+/args[0]->b_flags & B_READ/
+{
+ printf("%d", args[0]->b_bcount);
+}
+
+io:::start
+/!(args[0]->b_flags & B_READ)/
+{
+ printf("%d", args[0]->b_bcount);
+}
+
+/*
+ * SQLite3 stuff
+ */
+
+pid$target:_sqlite3.so:_pysqlite_query_execute:entry
+{
+ self->executing = 1;
+ self->sql = "";
+ printf("%d", timestamp);
+}
+
+pid$target:_sqlite3.so:_pysqlite_query_execute:return
+{
+ self->executing = 0;
+ printf("%d %s", timestamp, self->sql);
+}
+
+pid$target::PyString_AsString:return
+/self->executing/
+{
+ self->sql = copyinstr(arg1);
+ self->executing = 0;
+}
+
+pid$target:_sqlite3.so:pysqlite_cursor_iternext:entry
+{
+ printf("%d", timestamp);
+}
+
+pid$target:_sqlite3.so:pysqlite_cursor_iternext:return
+{
+ printf("%d", timestamp);
+}
Deleted: CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py
===================================================================
--- CalendarServer/trunk/contrib/performance/mkcal.py 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py 2010-08-12 00:05:13 UTC (rev 6067)
@@ -1,352 +0,0 @@
-"""
-Make a new calendar using an existing user account on an already-running iCal
-server.
-"""
-
-import sys
-
-from subprocess import PIPE, Popen
-from signal import SIGINT
-
-from urllib2 import HTTPDigestAuthHandler
-from uuid import uuid1, uuid4
-from datetime import datetime, timedelta
-from time import time
-from StringIO import StringIO
-
-import vobject
-
-from client.account import CalDAVAccount
-from protocol.url import URL
-
-from zope.interface import implements
-
-from twisted.internet.protocol import ProcessProtocol, Protocol
-from twisted.internet.defer import (
- Deferred, inlineCallbacks, returnValue, succeed, gatherResults)
-from twisted.internet import reactor
-from twisted.web.iweb import IBodyProducer
-from twisted.web.client import Agent
-from twisted.web.http_headers import Headers
-
-from httpauth import AuthHandlerAgent
-
-
-# XXX Represent these as vobjects? Would make it easier to add more vevents.
-event = """\
-BEGIN:VCALENDAR
-VERSION:2.0
-CALSCALE:GREGORIAN
-PRODID:-//Apple Inc.//iCal 4.0.3//EN
-BEGIN:VTIMEZONE
-TZID:America/New_York
-BEGIN:STANDARD
-DTSTART:20071104T020000
-RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
-TZNAME:EST
-TZOFFSETFROM:-0400
-TZOFFSETTO:-0500
-END:STANDARD
-BEGIN:DAYLIGHT
-DTSTART:20070311T020000
-RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
-TZNAME:EDT
-TZOFFSETFROM:-0500
-TZOFFSETTO:-0400
-END:DAYLIGHT
-END:VTIMEZONE
-%(VEVENTS)s\
-END:VCALENDAR
-"""
-
-vfreebusy = """\
-BEGIN:VCALENDAR
-CALSCALE:GREGORIAN
-VERSION:2.0
-METHOD:REQUEST
-PRODID:-//Apple Inc.//iCal 4.0.3//EN
-BEGIN:VFREEBUSY
-UID:81F582C8-4E7F-491C-85F4-E541864BE0FA
-DTEND:20100730T150000Z
-ATTENDEE:urn:uuid:user02
-DTSTART:20100730T140000Z
-X-CALENDARSERVER-MASK-UID:EC75A61B-08A3-44FD-BFBB-2457BBD0D490
-DTSTAMP:20100729T174751Z
-ORGANIZER:mailto:user01 at example.com
-SUMMARY:Availability for urn:uuid:user02
-END:VFREEBUSY
-END:VCALENDAR
-"""
-
-def formatDate(d):
- return ''.join(filter(str.isalnum, d.isoformat()))
-
-def makeEvent(i):
- s = """\
-BEGIN:VEVENT
-UID:%(UID)s
-DTSTART;TZID=America/New_York:%(START)s
-DTEND;TZID=America/New_York:%(END)s
-CREATED:20100729T193912Z
-DTSTAMP:20100729T195557Z
-SEQUENCE:%(SEQUENCE)s
-SUMMARY:STUFF IS THINGS
-TRANSP:OPAQUE
-END:VEVENT
-"""
- base = datetime(2010, 7, 30, 11, 15, 00)
- interval = timedelta(0, 5)
- duration = timedelta(0, 3)
- return event % {
- 'VEVENTS': s % {
- 'UID': uuid4(),
- 'START': formatDate(base + i * interval),
- 'END': formatDate(base + i * interval + duration),
- 'SEQUENCE': i,
- },
- }
-
-
-def makeEvents(n):
- return [makeEvent(i) for i in range(n)]
-
-
-class _DiscardReader(Protocol):
- def __init__(self, finished):
- self.finished = finished
-
-
- def dataReceived(self, bytes):
- pass
-
-
- def connectionLost(self, reason):
- self.finished.callback(None)
-
-
-
-def readBody(response):
- finished = Deferred()
- response.deliverBody(_DiscardReader(finished))
- return finished
-
-
-
-class StringProducer(object):
- implements(IBodyProducer)
-
- def __init__(self, body):
- self._body = body
- self.length = len(self._body)
-
-
- def startProducing(self, consumer):
- consumer.write(self._body)
- return succeed(None)
-
-
- at inlineCallbacks
-def measure(pids, events, samples):
- # First set things up
- account = CalDAVAccount(
- "localhost:8008", user="user01", pswd="user01", root="/", principal="/")
- account.session.deleteResource(
- URL("/calendars/users/user01/monkeys3/"))
- account.session.makeCalendar(
- URL("/calendars/users/user01/monkeys3/"))
-
- for i, cal in enumerate(makeEvents(events)):
- account.session.writeData(
- URL("/calendars/users/user01/monkeys3/foo-%d.ics" % (i,)),
- cal,
- "text/calendar")
-
- # CalDAVClientLibrary can't seem to POST things.
- authinfo = HTTPDigestAuthHandler()
- authinfo.add_password(
- realm="Test Realm",
- uri="http://localhost:8008/",
- user="user01",
- passwd="user01")
-
- agent = AuthHandlerAgent(Agent(reactor), authinfo)
- method = 'POST'
- uri = 'http://localhost:8008/calendars/__uids__/user01/outbox/'
- headers = Headers({"content-type": ["text/calendar"]})
- body = StringProducer(vfreebusy)
-
- # Now sample it a bunch of times
- data = []
- with DTraceCollector(pids) as dtrace:
- for i in range(samples):
- before = time()
- response = yield agent.request(
- method, uri, headers, body)
- yield readBody(response)
- after = time()
- data.append(after - before)
- stats = {Duration('urlopen time'): data}
- stats.update((yield dtrace))
- returnValue(stats)
-
-
-class _Statistic(object):
- def __init__(self, name):
- self.name = name
-
-
- def summarize(self, data):
- print 'mean', self.name, mean(data)
- print 'median', self.name, median(data)
- print 'stddev', self.name, stddev(data)
- print 'sum', self.name, sum(data)
-
-
- def write(self, basename, data):
- fObj = file(basename % (self.name,), 'w')
- fObj.write('\n'.join(map(str, data)) + '\n')
- fObj.close()
-
-
-
-class Duration(_Statistic):
- pass
-
-
-
-class Bytes(_Statistic):
- pass
-
-
-
-class DTraceCollector(object):
- def __init__(self, pids):
- self.pids = pids
- self._read = []
- self._write = []
- self._execute = []
- self._iternext = []
-
-
- def stats(self):
- return {
- Bytes('read'): self._read,
- Bytes('write'): self._write,
- Duration('execute'): self._execute,
- Duration('iternext'): self._iternext,
- }
-
-
- def _parse(self, dtrace):
- file('dtrace.log', 'a').write(dtrace)
-
-
- start = None
- for L in dtrace.splitlines():
- parts = L.split(None)
- if len(parts) >= 4:
- event = parts[2]
- func, stage = event.split(':')
- value = int(parts[3])
- if stage == 'entry':
- start = value
- elif stage == 'return':
- if start is None:
- print func, 'return without entry at', parts[3]
- continue
- end = int(parts[3])
- diff = end - start
- if func == '_pysqlite_query_execute':
- accum = self._execute
- elif func == 'pysqlite_cursor_iternext':
- accum = self._iternext
- else:
- continue
- if diff < 0:
- print 'Completely bogus dealie', func, start, end
- else:
- accum.append(diff)
- start = None
- else:
- continue
-
-
- def __enter__(self):
- finished = []
- self.dtraces = {}
- for p in self.pids:
- d = Deferred()
- self.dtraces[p] = reactor.spawnProcess(
- IOMeasureConsumer(d),
- "/usr/sbin/dtrace",
- ["/usr/sbin/dtrace", "-p", str(p), "-s", "io_measure.d"])
- d.addCallback(self._cleanup, p)
- d.addCallback(self._parse)
- finished.append(d)
- return gatherResults(finished).addCallback(lambda ign: self.stats())
-
-
- def _cleanup(self, passthrough, pid):
- del self.dtraces[pid]
- return passthrough
-
-
- def __exit__(self, type, value, traceback):
- for proc in self.dtraces.itervalues():
- proc.signalProcess(SIGINT)
-
-
-
-class IOMeasureConsumer(ProcessProtocol):
- def __init__(self, done):
- self.done = done
-
-
- def connectionMade(self):
- self.out = StringIO()
-
-
- def errReceived(self, bytes):
- print bytes
-
- def outReceived(self, bytes):
- self.out.write(bytes)
-
-
- def processEnded(self, reason):
- self.done.callback(self.out.getvalue())
-
-
-def mean(samples):
- return sum(samples) / len(samples)
-
-
-def median(samples):
- return sorted(samples)[len(samples) / 2]
-
-
-def stddev(samples):
- m = mean(samples)
- variance = sum([(datum - m) ** 2 for datum in samples]) / len(samples)
- return variance ** 0.5
-
- at inlineCallbacks
-def main():
- # Figure out which pids we are benchmarking.
- pids = map(int, sys.argv[1:])
-
- for numEvents in [1, 100]: #, 1000]:#, 10000]:
- print 'Testing', numEvents, 'events'
- data = yield measure(pids, numEvents, 100)
- for k, v in data.iteritems():
- if v:
- k.summarize(v)
- k.write('vfreebusy.%%s.%d' % (numEvents,), v)
-
-
-if __name__ == '__main__':
- from twisted.python.log import err
- d = main()
- d.addErrback(err)
- d.addCallback(lambda ign: reactor.stop())
- reactor.run()
Copied: CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py (from rev 6066, CalendarServer/trunk/contrib/performance/mkcal.py)
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py (rev 0)
+++ CalendarServer/branches/users/glyph/sql-store/contrib/performance/mkcal.py 2010-08-12 00:05:13 UTC (rev 6067)
@@ -0,0 +1,352 @@
+"""
+Make a new calendar using an existing user account on an already-running iCal
+server.
+"""
+
+import sys
+
+from subprocess import PIPE, Popen
+from signal import SIGINT
+
+from urllib2 import HTTPDigestAuthHandler
+from uuid import uuid1, uuid4
+from datetime import datetime, timedelta
+from time import time
+from StringIO import StringIO
+
+import vobject
+
+from client.account import CalDAVAccount
+from protocol.url import URL
+
+from zope.interface import implements
+
+from twisted.internet.protocol import ProcessProtocol, Protocol
+from twisted.internet.defer import (
+ Deferred, inlineCallbacks, returnValue, succeed, gatherResults)
+from twisted.internet import reactor
+from twisted.web.iweb import IBodyProducer
+from twisted.web.client import Agent
+from twisted.web.http_headers import Headers
+
+from httpauth import AuthHandlerAgent
+
+
+# XXX Represent these as vobjects? Would make it easier to add more vevents.
+event = """\
+BEGIN:VCALENDAR
+VERSION:2.0
+CALSCALE:GREGORIAN
+PRODID:-//Apple Inc.//iCal 4.0.3//EN
+BEGIN:VTIMEZONE
+TZID:America/New_York
+BEGIN:STANDARD
+DTSTART:20071104T020000
+RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
+TZNAME:EST
+TZOFFSETFROM:-0400
+TZOFFSETTO:-0500
+END:STANDARD
+BEGIN:DAYLIGHT
+DTSTART:20070311T020000
+RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
+TZNAME:EDT
+TZOFFSETFROM:-0500
+TZOFFSETTO:-0400
+END:DAYLIGHT
+END:VTIMEZONE
+%(VEVENTS)s\
+END:VCALENDAR
+"""
+
+vfreebusy = """\
+BEGIN:VCALENDAR
+CALSCALE:GREGORIAN
+VERSION:2.0
+METHOD:REQUEST
+PRODID:-//Apple Inc.//iCal 4.0.3//EN
+BEGIN:VFREEBUSY
+UID:81F582C8-4E7F-491C-85F4-E541864BE0FA
+DTEND:20100730T150000Z
+ATTENDEE:urn:uuid:user02
+DTSTART:20100730T140000Z
+X-CALENDARSERVER-MASK-UID:EC75A61B-08A3-44FD-BFBB-2457BBD0D490
+DTSTAMP:20100729T174751Z
+ORGANIZER:mailto:user01 at example.com
+SUMMARY:Availability for urn:uuid:user02
+END:VFREEBUSY
+END:VCALENDAR
+"""
+
+def formatDate(d):
+ return ''.join(filter(str.isalnum, d.isoformat()))
+
+def makeEvent(i):
+ s = """\
+BEGIN:VEVENT
+UID:%(UID)s
+DTSTART;TZID=America/New_York:%(START)s
+DTEND;TZID=America/New_York:%(END)s
+CREATED:20100729T193912Z
+DTSTAMP:20100729T195557Z
+SEQUENCE:%(SEQUENCE)s
+SUMMARY:STUFF IS THINGS
+TRANSP:OPAQUE
+END:VEVENT
+"""
+ base = datetime(2010, 7, 30, 11, 15, 00)
+ interval = timedelta(0, 5)
+ duration = timedelta(0, 3)
+ return event % {
+ 'VEVENTS': s % {
+ 'UID': uuid4(),
+ 'START': formatDate(base + i * interval),
+ 'END': formatDate(base + i * interval + duration),
+ 'SEQUENCE': i,
+ },
+ }
+
+
+def makeEvents(n):
+ return [makeEvent(i) for i in range(n)]
+
+
+class _DiscardReader(Protocol):
+ def __init__(self, finished):
+ self.finished = finished
+
+
+ def dataReceived(self, bytes):
+ pass
+
+
+ def connectionLost(self, reason):
+ self.finished.callback(None)
+
+
+
+def readBody(response):
+ finished = Deferred()
+ response.deliverBody(_DiscardReader(finished))
+ return finished
+
+
+
+class StringProducer(object):
+ implements(IBodyProducer)
+
+ def __init__(self, body):
+ self._body = body
+ self.length = len(self._body)
+
+
+ def startProducing(self, consumer):
+ consumer.write(self._body)
+ return succeed(None)
+
+
+ at inlineCallbacks
+def measure(pids, events, samples):
+ # First set things up
+ account = CalDAVAccount(
+ "localhost:8008", user="user01", pswd="user01", root="/", principal="/")
+ account.session.deleteResource(
+ URL("/calendars/users/user01/monkeys3/"))
+ account.session.makeCalendar(
+ URL("/calendars/users/user01/monkeys3/"))
+
+ for i, cal in enumerate(makeEvents(events)):
+ account.session.writeData(
+ URL("/calendars/users/user01/monkeys3/foo-%d.ics" % (i,)),
+ cal,
+ "text/calendar")
+
+ # CalDAVClientLibrary can't seem to POST things.
+ authinfo = HTTPDigestAuthHandler()
+ authinfo.add_password(
+ realm="Test Realm",
+ uri="http://localhost:8008/",
+ user="user01",
+ passwd="user01")
+
+ agent = AuthHandlerAgent(Agent(reactor), authinfo)
+ method = 'POST'
+ uri = 'http://localhost:8008/calendars/__uids__/user01/outbox/'
+ headers = Headers({"content-type": ["text/calendar"]})
+ body = StringProducer(vfreebusy)
+
+ # Now sample it a bunch of times
+ data = []
+ with DTraceCollector(pids) as dtrace:
+ for i in range(samples):
+ before = time()
+ response = yield agent.request(
+ method, uri, headers, body)
+ yield readBody(response)
+ after = time()
+ data.append(after - before)
+ stats = {Duration('urlopen time'): data}
+ stats.update((yield dtrace))
+ returnValue(stats)
+
+
+class _Statistic(object):
+ def __init__(self, name):
+ self.name = name
+
+
+ def summarize(self, data):
+ print 'mean', self.name, mean(data)
+ print 'median', self.name, median(data)
+ print 'stddev', self.name, stddev(data)
+ print 'sum', self.name, sum(data)
+
+
+ def write(self, basename, data):
+ fObj = file(basename % (self.name,), 'w')
+ fObj.write('\n'.join(map(str, data)) + '\n')
+ fObj.close()
+
+
+
+class Duration(_Statistic):
+ pass
+
+
+
+class Bytes(_Statistic):
+ pass
+
+
+
+class DTraceCollector(object):
+ def __init__(self, pids):
+ self.pids = pids
+ self._read = []
+ self._write = []
+ self._execute = []
+ self._iternext = []
+
+
+ def stats(self):
+ return {
+ Bytes('read'): self._read,
+ Bytes('write'): self._write,
+ Duration('execute'): self._execute,
+ Duration('iternext'): self._iternext,
+ }
+
+
+ def _parse(self, dtrace):
+ file('dtrace.log', 'a').write(dtrace)
+
+
+ start = None
+ for L in dtrace.splitlines():
+ parts = L.split(None)
+ if len(parts) >= 4:
+ event = parts[2]
+ func, stage = event.split(':')
+ value = int(parts[3])
+ if stage == 'entry':
+ start = value
+ elif stage == 'return':
+ if start is None:
+ print func, 'return without entry at', parts[3]
+ continue
+ end = int(parts[3])
+ diff = end - start
+ if func == '_pysqlite_query_execute':
+ accum = self._execute
+ elif func == 'pysqlite_cursor_iternext':
+ accum = self._iternext
+ else:
+ continue
+ if diff < 0:
+ print 'Completely bogus dealie', func, start, end
+ else:
+ accum.append(diff)
+ start = None
+ else:
+ continue
+
+
+ def __enter__(self):
+ finished = []
+ self.dtraces = {}
+ for p in self.pids:
+ d = Deferred()
+ self.dtraces[p] = reactor.spawnProcess(
+ IOMeasureConsumer(d),
+ "/usr/sbin/dtrace",
+ ["/usr/sbin/dtrace", "-p", str(p), "-s", "io_measure.d"])
+ d.addCallback(self._cleanup, p)
+ d.addCallback(self._parse)
+ finished.append(d)
+ return gatherResults(finished).addCallback(lambda ign: self.stats())
+
+
+ def _cleanup(self, passthrough, pid):
+ del self.dtraces[pid]
+ return passthrough
+
+
+ def __exit__(self, type, value, traceback):
+ for proc in self.dtraces.itervalues():
+ proc.signalProcess(SIGINT)
+
+
+
+class IOMeasureConsumer(ProcessProtocol):
+ def __init__(self, done):
+ self.done = done
+
+
+ def connectionMade(self):
+ self.out = StringIO()
+
+
+ def errReceived(self, bytes):
+ print bytes
+
+ def outReceived(self, bytes):
+ self.out.write(bytes)
+
+
+ def processEnded(self, reason):
+ self.done.callback(self.out.getvalue())
+
+
+def mean(samples):
+ return sum(samples) / len(samples)
+
+
+def median(samples):
+ return sorted(samples)[len(samples) / 2]
+
+
+def stddev(samples):
+ m = mean(samples)
+ variance = sum([(datum - m) ** 2 for datum in samples]) / len(samples)
+ return variance ** 0.5
+
+ at inlineCallbacks
+def main():
+ # Figure out which pids we are benchmarking.
+ pids = map(int, sys.argv[1:])
+
+ for numEvents in [1, 100]: #, 1000]:#, 10000]:
+ print 'Testing', numEvents, 'events'
+ data = yield measure(pids, numEvents, 100)
+ for k, v in data.iteritems():
+ if v:
+ k.summarize(v)
+ k.write('vfreebusy.%%s.%d' % (numEvents,), v)
+
+
+if __name__ == '__main__':
+ from twisted.python.log import err
+ d = main()
+ d.addErrback(err)
+ d.addCallback(lambda ign: reactor.stop())
+ reactor.run()
Modified: CalendarServer/branches/users/glyph/sql-store/setup.py
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/setup.py 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/setup.py 2010-08-12 00:05:13 UTC (rev 6067)
@@ -108,6 +108,9 @@
"zoneinfo/*/*/*.ics",
"images/*/*.jpg",
],
+ "txcaldav.calendarstore": [
+ "*.sql",
+ ],
},
scripts = [
"bin/caldavd",
Modified: CalendarServer/branches/users/glyph/sql-store/support/Makefile.Apple
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/support/Makefile.Apple 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/support/Makefile.Apple 2010-08-12 00:05:13 UTC (rev 6067)
@@ -46,16 +46,17 @@
PyOpenDirectory:: $(BuildDirectory)/PyOpenDirectory
PyXML-0.8.4:: $(BuildDirectory)/PyXML-0.8.4
vobject:: $(BuildDirectory)/vobject
+PyGreSQL-4.0:: $(BuildDirectory)/PyGreSQL-4.0
$(Project):: $(BuildDirectory)/$(Project)
-build:: PyKerberos PyOpenDirectory PyXML-0.8.4 vobject $(Project)
+build:: PyKerberos PyOpenDirectory PyXML-0.8.4 vobject PyGreSQL-4.0 $(Project)
setup:
$(_v) ./run -g
-prep:: setup CalDAVTester.tgz PyKerberos.tgz PyOpenDirectory.tgz PyXML-0.8.4.tgz vobject.tgz
+prep:: setup CalDAVTester.tgz PyKerberos.tgz PyOpenDirectory.tgz PyXML-0.8.4.tgz vobject.tgz PyGreSQL-4.0.tgz
-PyKerberos PyOpenDirectory PyXML-0.8.4 vobject $(Project)::
+PyKerberos PyOpenDirectory PyXML-0.8.4 vobject PyGreSQL-4.0 $(Project)::
@echo "Building $@..."
$(_v) cd $(BuildDirectory)/$@ && $(Environment) $(PYTHON) setup.py build
@@ -70,6 +71,7 @@
$(_v) cd $(BuildDirectory)/PyOpenDirectory && $(Environment) $(PYTHON) setup.py install $(PY_INSTALL_FLAGS)
$(_v) cd $(BuildDirectory)/PyXML-0.8.4 && $(Environment) $(PYTHON) setup.py install $(PY_INSTALL_FLAGS)
$(_v) cd $(BuildDirectory)/vobject && $(Environment) $(PYTHON) setup.py install $(PY_INSTALL_FLAGS)
+ $(_v) cd $(BuildDirectory)/PyGreSQL-4.0 && $(Environment) $(PYTHON) setup.py install $(PY_INSTALL_FLAGS)
$(_v) for so in $$(find "$(DSTROOT)$(PY_HOME)/lib" -type f -name '*.so'); do $(STRIP) -Sx "$${so}"; done
$(_v) $(INSTALL_FILE) "$(Sources)/conf/caldavd-apple.plist" "$(DSTROOT)$(ETCDIR)/caldavd/caldavd.plist"
$(_v) $(INSTALL_FILE) "$(Sources)/conf/caldavd-partitioning-primary.plist" "$(DSTROOT)$(ETCDIR)/caldavd/caldavd-partitioning-primary.plist"
Modified: CalendarServer/branches/users/glyph/sql-store/support/build.sh
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/support/build.sh 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/support/build.sh 2010-08-12 00:05:13 UTC (rev 6067)
@@ -556,11 +556,20 @@
"vObject" "vobject" "vobject" \
"http://svn.osafoundation.org/vobject/trunk";
+ #
# Tool dependencies. The code itself doesn't depend on these, but
# they are useful to developers.
+ #
+
svn_get "CalDAVTester" "${top}/CalDAVTester" "${svn_uri_base}/CalDAVTester/trunk" HEAD;
+
svn_get "Pyflakes" "${top}/Pyflakes" http://divmod.org/svn/Divmod/trunk/Pyflakes HEAD;
+ local pd="pydoctor-0.3";
+ py_dependency \
+ "pydoctor" "pydoctor" "${pd}" \
+ "http://launchpadlibrarian.net/42323121/${pd}.tar.gz";
+
if "${do_setup}"; then
cd "${caldav}";
echo "Building our own extension modules...";
Modified: CalendarServer/branches/users/glyph/sql-store/support/submit
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/support/submit 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/support/submit 2010-08-12 00:05:13 UTC (rev 6067)
@@ -128,7 +128,7 @@
echo "Copying ${src}...";
ignores="$(mktemp -t CalendarServer_ignores)";
svn st --no-ignore | sed -n -e 's|^I......||p' > "${ignores}";
- rsync -av --exclude=".svn" --exclude="_trial_temp" --exclude-from="${ignores}" "${src}/" "${wc}";
+ rsync -av --exclude=".svn" --exclude="_trial_temp" --exclude="*.pyc" --exclude="*.so" --exclude-from="${ignores}" "${src}/" "${wc}";
rm "${ignores}";
else
echo "";
Modified: CalendarServer/branches/users/glyph/sql-store/twext/web2/resource.py
===================================================================
--- CalendarServer/branches/users/glyph/sql-store/twext/web2/resource.py 2010-08-11 23:32:02 UTC (rev 6066)
+++ CalendarServer/branches/users/glyph/sql-store/twext/web2/resource.py 2010-08-12 00:05:13 UTC (rev 6067)
@@ -117,13 +117,13 @@
response.headers.setHeader("allow", self.allowedMethods())
return response
- def http_TRACE(self, request):
- """
- Respond to a TRACE request.
- @param request: the request to process.
- @return: an object adaptable to L{iweb.IResponse}.
- """
- return server.doTrace(request)
+# def http_TRACE(self, request):
+# """
+# Respond to a TRACE request.
+# @param request: the request to process.
+# @return: an object adaptable to L{iweb.IResponse}.
+# """
+# return server.doTrace(request)
def http_HEAD(self, request):
"""
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/calendarserver-changes/attachments/20100811/2b5035a8/attachment-0001.html>
More information about the calendarserver-changes
mailing list