Performance and Parallel Requests to Calendarserver
hi, i'm running a local calendarserver (version: 2.4.dfsg-6) on my ubuntu machine (oneiric - 11.10). i'm developing a php/web-based calendar application which sends requests to the calendarserver and processes its answer. i am now confronted with two problems: the first problem: to request all available dates within the month of july via firefox, for example, it lasts round about 12 seconds. [there are approximately 600 *.ics files for july] the request is: <?xml version="1.0" encoding="utf-8" ?> <C:calendar-query xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> <D:prop> <D:getetag/> <C:calendar-data/> </D:prop> <C:filter> <C:comp-filter name="VCALENDAR"> <C:comp-filter name="VEVENT"> <C:time-range start="20120631T220000Z" end="20120731T215959Z"/> </C:comp-filter> </C:comp-filter> </C:filter> </C:calendar-query> are there any possibilities to optimize the response time? (requesting week by week for july results in the same amount of time since every request per week lasts 3 seconds...) the second problem: if i simulate two different users with independent sessions concurrently requesting the same timeframe (july) then indeed both requests arrive at the calendarserver at the same time, but "the first request wins" and the other one has to wait until the first request has been processed by the calendarserver. the calenderserver processes the second request only if it has finished processing the first one. so, the winner got the results after this delay of 12 seconds, the other user has to wait 24 seconds. if there would be more users, the response time would grow enormously. did i miss something while configuring the calendarserver? i have to admit that i use the default caldav.plist config file. i thought that the calendarserver could handle multiple request at the same time like the apache webserver, for instance. any help and advice/suggestion would be great. thanks, emport -- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
Le Jul 2, 2012 à 12:05 PM, emport <em@svenport.de> a écrit :
i'm running a local calendarserver (version: 2.4.dfsg-6) on my ubuntu machine (oneiric - 11.10).
(...)
are there any possibilities to optimize the response time? (requesting week
Yes. Please upgrade to CalendarServer 3.0 or later. These use a nicely indexed SQL database - PostgreSQL, specifically - to optimize things exactly like the time-range query you're asking about - rather than scanning through the filesystem with a few ad-hoc sqlite databases to help with some operations. Debian (and therefore ubuntu) have even been working on upgrading their packaging; if you look in the archives for this list you should see a couple of messages indicating where you can get newer pre-release packages for testing. -glyph
On Jul 2, 2012, at 12:05 PM, emport <em@svenport.de> wrote:
i thought that the calendarserver could handle multiple request at the same time like the apache webserver, for instance.
Verify that you have the server configured properly (e.g. enough daemons to handle concurrent requests - see ProcessCount in caldavd.plist). The larger problem, though, is that you are issuing one of the most expensive kinds of requests and expecting it to be fast. The service provides ctags / etags as change tokens on events and collections which can be used in conjunction with a client-side cache. In an expected client implementation, requests like the one you are issuing would only happen the first time the client comes online with the account, to cache everything. After that, you only download things that are new or changed. In general, the service is able to operate in a highly concurrent fashion, although scaling it beyond a single server does require some know-how. You should be able to support a small to medium size workgroup on an average server. Also, 2.4 is positively ancient. Debian just released a 3.1 package; hopefully Ubuntu will follow soon :) With 3.1, you can also use the sync report: http://tools.ietf.org/html/draft-daboo-webdav-sync-02 - although I don't think it will buy you any huge performance wins, though it may be more convenient. HTH, -dre
any help and advice/suggestion would be great.
thanks, emport -- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
hi, i tested your suggestion by increasing the ProcessCount to 3. Andre LaBranche-2 wrote:
Verify that you have the server configured properly (e.g. enough daemons to handle concurrent requests - see ProcessCount in caldavd.plist).
i discovered that the server opens up 3 "instances" which are accessible via http://localhost:8008 http://localhost:8009 http://localhost:8010 is this intended? [sorry for this clueless question...] but the result is: if i concurrently fire three requests i still send the request to http://localhost:8008, since my programm has no clue that there are other ports to use and thus the response time for all three requests is nearly the same, i.e. the triple time as if i would send only one request. uah, this is complicated to describe. i hope my description is intelligible, although. maybe someone could give me some more advise... thanks, emport -- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
On Jul 3, 2012, at 12:23 PM, emport wrote:
hi,
i tested your suggestion by increasing the ProcessCount to 3.
Andre LaBranche-2 wrote:
Verify that you have the server configured properly (e.g. enough daemons to handle concurrent requests - see ProcessCount in caldavd.plist).
i discovered that the server opens up 3 "instances" which are accessible via
http://localhost:8008 http://localhost:8009 http://localhost:8010
is this intended? [sorry for this clueless question...]
Yes, that is expected in your configuration, for this ancient version. In your version, there is a python software load balancer that distributes requests to available instances. You now have three available instances and not one, so it's expected to see additional listeners. There is, however, one additional variable that affects concurrency, the config variable MaxRequests. This defines how many requests can be handled in parallel by a single daemon, *but only when those requests are doing deferrable work*. You might just be bottlenecked in the filesystem. If your disk IO is railed while these requests are processing, that is the bottleneck. There's really no way around this other than upgrading. Just to set your expectation, doing such a large-distance upgrade successfully (when there is data to be maintained / upgraded) will be a lot of work. -dre
but the result is: if i concurrently fire three requests i still send the request to http://localhost:8008, since my programm has no clue that there are other ports to use and thus the response time for all three requests is nearly the same, i.e. the triple time as if i would send only one request.
uah, this is complicated to describe. i hope my description is intelligible, although.
maybe someone could give me some more advise...
thanks, emport -- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
hi, Andre LaBranche-2 wrote:
Yes, that is expected in your configuration, for this ancient version. In your version, there is a python software load balancer that distributes requests to available instances. You now have three available instances and not one, so it's expected to see additional listeners.
ok, this sounds reasonable. Andre LaBranche-2 wrote:
There is, however, one additional variable that affects concurrency, the config variable MaxRequests.
hm, where can i find this variable? i didn't see it in caldav.plist... Andre LaBranche-2 wrote:
Just to set your expectation, doing such a large-distance upgrade successfully (when there is data to be maintained / upgraded) will be a lot of work.
do you mean an upgrade from 2.4 to 3.x? i'm a little bit curios what you mean by *a lot of work*. my naive approach would be: set up the server, migrate all *.ics files and go to lunch :) kinds regards, emport -- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
On Jul 3, 2012, at 12:49 PM, emport wrote:
hi,
Andre LaBranche-2 wrote:
Yes, that is expected in your configuration, for this ancient version. In your version, there is a python software load balancer that distributes requests to available instances. You now have three available instances and not one, so it's expected to see additional listeners.
ok, this sounds reasonable.
Andre LaBranche-2 wrote:
There is, however, one additional variable that affects concurrency, the config variable MaxRequests.
hm, where can i find this variable? i didn't see it in caldav.plist...
http://trac.calendarserver.org/browser/CalendarServer/tags/release/CalendarS... If MaxRequests meant the same thing then (in 2.4) as I think it means, the default value of 600 is ... potentially terrifying; it should stop accepting new work far sooner than that. Anyway, this isn't the bottleneck for you :) -dre
Andre LaBranche-2 wrote:
Just to set your expectation, doing such a large-distance upgrade successfully (when there is data to be maintained / upgraded) will be a lot of work.
do you mean an upgrade from 2.4 to 3.x? i'm a little bit curios what you mean by *a lot of work*. my naive approach would be: set up the server, migrate all *.ics files and go to lunch :)
kinds regards, emport
-- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
hi, thank you for your quick answers. the main problem is the calendarserver at the production system. unfortunately, it can't be changed. after your suggestions and advises and my status as newbie regarding this topic :-( , i would like to ask you some more questions: Andre LaBranche-2 wrote:
Verify that you have the server configured properly (e.g. enough daemons to handle concurrent requests - see ProcessCount in caldavd.plist).
my local server has the following lines, actually: <key>ProcessType</key> <string>Combined</string> <key>MultiProcess</key> <dict> <key>ProcessCount</key> <integer>1</integer> <!-- 0 = larger of: 4 or (2 * CPU count) --> </dict> i don't understand the comment right beside the integer tag. if i would change this to X, would this imply that the calendarserver could handle X requests in parallel? or am i totally wrong? [is there any documentation about this config file? i didn't found anything....] the other question is: how could i determine the version of the production calendarserver? is there a special request which i could send to there server (via curl, e.g.) so that it responses with its version (number)? i didn't found anything via google nor in the webinterface of the calendarserver. i tried it by
curl -i --trace-ascii - --anyauth -u test http://localhost:8008/principals/users/test for my local machine but the only hint was Server: Twisted/8.2.0 TwistedWeb/8.2.0 but i don't see the connection to the version 2.4.dfsg-6... or is there any correlation?
again, thanks for you help -- View this message in context: http://old.nabble.com/Performance-and-Parallel-Requests-to-Calendarserver-tp... Sent from the Calendar Server - Users mailing list archive at Nabble.com.
participants (3)
-
Andre LaBranche
-
emport
-
Glyph