[CalendarServer-dev] CTag/ETag on calendar collections

[Peter Mogensen]

Cyrus Daboo wrote:
>> I'm considering using Varnish in front of CalendarServer to cache
>> iCalendar data for calendar-collections.
> 
> Do you have a lot of users doing "subscriptions" to the calendar 
> collection itself?

Enough for the current speed of GET on a collection to be too slow.
It's not so much a question of number of users as a question of response 
time for a single "rolled-up" req. on a large calendar.
But I have enough users that both could be a problem if I had tried it 
in production.

> I know we have been meaning to look into improving 
> the performance of calendar collections GETs - however strictly speaking 
> that is not an "official" part of the spec though many CalDAV servers do 
> support that capability.
> 
> Two options we have for improving performance are to cache the rolled up 
> calendar data on disk, and/or cache in memcached.

Yes, That's actually what we try to do with an external cache.

> If we are really smart 
> we could include an index with that so that we could do partial updates 
> to the cached data as individual calendar resources change within the 
> collection itself - but that would be more complicated.
> 
> A key question here is what are the usage patterns for these calendars. 
> Are they updated frequently?  If so, how often?

> How large are they?

Some very large - so large that a GET actively rolling up data, 
currently is too slow to be practical.

> How 
> many subscribers are likely?

Probably not many, though it might change if the feature grows in 
popularity.

> Why not just use CalDAV which allows for 
> more efficient partial updates...

Because publishing iCalendar data is still most often done via GET on an 
iCalendar resource.

>> As far as I can see the CTag is necessary because an ETag on a collection
>> is not required by WebDAV to change when a member changes content.
> 
> Correct - ETags on collections are somewhat vague.
> 
>> However... with the current calendarserver behavior returning rolled up
>> iCalendar when doing a GET it seems that the Etag must change due to
>> RFC4917 9,4 and RFC2616 13.3.4 (ETag must change when GET value changes).
> 
> Correct.
> 
>> Also, it seems like this is enforced by CalendarServer storing the CTag
>> in an attribute of the calendar collection - which in turn make Twisted
>> update the ETag based on the modification time of the inode.
> 
> A nice side-effect!

Yes - but I'm somewhat reluctant to rely on it given the number of 
on-disk layout changes you have :)

>> So, am I right in concluding that CTag and ETag on a calendar collection
>> will currently always be in sync.
>> - partly because of the implementation detail of the on-disk storage that
>> ctag is in an extended attribute.
>> - and partly because it's an implicit requirement when GET on a
>> calendar-collection returnes rolled up iCalendar
> 
> That is certainly true right now. To be more robust, what we might want 
> to consider is explicitly having the etag value be the ctag. That way if 
> we ever use a different type of property store (not xattrs) it will 
> still work.

Thanks for the quick reply.

/Peter