Cyrus Daboo wrote:

...

I'm considering using Varnish in front of CalendarServer to cache iCalendar data for calendar-collections.

Do you have a lot of users doing "subscriptions" to the calendar collection itself?

Enough for the current speed of GET on a collection to be too slow. It's not so much a question of number of users as a question of response time for a single "rolled-up" req. on a large calendar. But I have enough users that both could be a problem if I had tried it in production.

I know we have been meaning to look into improving the performance of calendar collections GETs - however strictly speaking that is not an "official" part of the spec though many CalDAV servers do support that capability.

Two options we have for improving performance are to cache the rolled up calendar data on disk, and/or cache in memcached.

Yes, That's actually what we try to do with an external cache.

If we are really smart we could include an index with that so that we could do partial updates to the cached data as individual calendar resources change within the collection itself - but that would be more complicated.

A key question here is what are the usage patterns for these calendars. Are they updated frequently? If so, how often?

How large are they?

Some very large - so large that a GET actively rolling up data, currently is too slow to be practical.

How many subscribers are likely?

Probably not many, though it might change if the feature grows in popularity.

Why not just use CalDAV which allows for more efficient partial updates...

Because publishing iCalendar data is still most often done via GET on an iCalendar resource.

...

As far as I can see the CTag is necessary because an ETag on a collection is not required by WebDAV to change when a member changes content.

Correct - ETags on collections are somewhat vague.

...

However... with the current calendarserver behavior returning rolled up iCalendar when doing a GET it seems that the Etag must change due to RFC4917 9,4 and RFC2616 13.3.4 (ETag must change when GET value changes).

Correct.

...

Also, it seems like this is enforced by CalendarServer storing the CTag in an attribute of the calendar collection - which in turn make Twisted update the ETag based on the modification time of the inode.

A nice side-effect!

Yes - but I'm somewhat reluctant to rely on it given the number of on-disk layout changes you have :)

...

So, am I right in concluding that CTag and ETag on a calendar collection will currently always be in sync. - partly because of the implementation detail of the on-disk storage that ctag is in an extended attribute. - and partly because it's an implicit requirement when GET on a calendar-collection returnes rolled up iCalendar

That is certainly true right now. To be more robust, what we might want to consider is explicitly having the etag value be the ctag. That way if we ever use a different type of property store (not xattrs) it will still work.

Thanks for the quick reply. /Peter