Unbound RRULE rewritten to COUNT=400
Hi, when I save an unbound RRULE, CalendarServer seems to rewrite it to a bound RRULE with a constant count of 400. Why is that? Does CalServer flatten RRULEs somewhere? I wonder whether its appropriate to convert a COUNT=400 to unlimited on import ... (should there be an X-CAL-SERVER-COUNT property instead?) Thanks, Helge
Hi Helge, --On July 1, 2009 4:20:31 PM +0200 Helge Heß <me@helgehess.eu> wrote:
when I save an unbound RRULE, CalendarServer seems to rewrite it to a bound RRULE with a constant count of 400.
Why is that? Does CalServer flatten RRULEs somewhere?
I wonder whether its appropriate to convert a COUNT=400 to unlimited on import ... (should there be an X-CAL-SERVER-COUNT property instead?)
We deliberately truncate unbounded RRULEs that would generate too many instances in our index for performance reasons. The basic policy is this: 1) For an unbounded RRULE with FREQ equal to anything other than YEARLY we add COUNT=400. 2) For a bounded RRULE with a COUNT greater than 400, we set COUNT=400. 3) For a bounded RRULE with an UNTIL we attempt to determine the number of instances up to the UNTIL value (using a fast, but not absolutely accurate, method based on frequency and period). If instances exceed 400 we remove the UNTIL and add COUNT=400. There are still some tweaks we are thinking about here. In particular it has been suggested that rather than doing a COUNT truncation, what we should do is set UNTIL to the end of the year after the year in the DTSTART - i.e. allow at least one whole year's worth of bookings. At some point we may also revise our indexing to remove the performance issue at which point truncation may not be needed - but for now it is better to do it. -- Cyrus Daboo
On 01.07.2009, at 16:31, Cyrus Daboo wrote:
We deliberately truncate unbounded RRULEs that would generate too many instances in our index for performance reasons.
OK. I still wonder if it would be better for the usability if we re- translate the 400 to unbound. Hm. Would it make sense to add a property which signals the truncation? (BTW: most clients will probably never use server side RRULE expansion? A webclient is the only thing which might make use of those REPORTs, hence the indices?)
At some point we may also revise our indexing to remove the performance issue at which point truncation may not be needed - but for now it is better to do it.
Can't you just keep 'small' events in the index and evaluate others in memory? I think thats what I originally did in SOGo. This DAViCal thing is also quite interesting: http://repo.or.cz/w/davical.git?a=blob;f=dba/rrule_functions.sql;hb=HEAD Greets, Helge -- Helge Hess http://zideone.com/
Hi Helge, --On July 1, 2009 4:44:24 PM +0200 Helge Heß <me@helgehess.eu> wrote:
We deliberately truncate unbounded RRULEs that would generate too many instances in our index for performance reasons.
OK.
I still wonder if it would be better for the usability if we re-translate the 400 to unbound. Hm. Would it make sense to add a property which signals the truncation?
I am not sure whether a X- parameter on the RRULE would be good or not. We now that clients are likely to throw away X- parameters.
(BTW: most clients will probably never use server side RRULE expansion? A webclient is the only thing which might make use of those REPORTs, hence the indices?)
Actually the server uses the indices for free-busy lookups and for room auto-accept processing.
At some point we may also revise our indexing to remove the performance issue at which point truncation may not be needed - but for now it is better to do it.
Can't you just keep 'small' events in the index and evaluate others in memory? I think thats what I originally did in SOGo.
As noted above we really want free busy lookups to be super fast as users will expect that to be done in "real-time". So indexing the time periods for each event index allows us to quickly find events that overlap a given period. It turns out that right now when we find those events we do go and read/parse them to extract information to determine what type of busy-time they represent (free, busy, busy-tentative etc). But we have found that users tend to do lots of free busy queries and even those read operations are slowing things down more than we would like, so we are working on putting enough information into the index that we can avoid having to read/parse the calendar data on disk.
This DAViCal thing is also quite interesting:
http://repo.or.cz/w/davical.git?a=blob;f=dba/rrule_functions.sql;hb=HEAD
Hrmm - looks like a stored function in the DB that does recurrence expansion. I still think that fully caching the instance time-ranges is better - free busy queries need to be done fast. -- Cyrus Daboo
On 01.07.2009, at 16:58, Cyrus Daboo wrote:
I still wonder if it would be better for the usability if we re- translate the 400 to unbound. Hm. Would it make sense to add a property which signals the truncation? I am not sure whether a X- parameter on the RRULE would be good or not. We now that clients are likely to throw away X- parameters.
We certainly don't and I'm still not sure why you assume that clients will? At least Evolution and Kontact use iCalendar as their internal data format and AFAIK preserve anything thrown at them. (would need more testing of course, would be good to do that at CalConnect)
(BTW: most clients will probably never use server side RRULE expansion? A webclient is the only thing which might make use of those REPORTs, hence the indices?) Actually the server uses the indices for free-busy lookups and for room auto-accept processing.
Hm, OK :-)
But we have found that users tend to do lots of free busy queries and even those read operations are slowing things down more than we would like, so we are working on putting enough information into the index that we can avoid having to read/parse the calendar data on disk.
Yes, parsing the V data is quite expensive. I'm just assuming that recurring events are the minority (<20%?) in a bigger personal calendar?
This DAViCal thing is also quite interesting:
http://repo.or.cz/w/davical.git?a=blob;f=dba/rrule_functions.sql;hb=HEAD Hrmm - looks like a stored function in the DB that does recurrence expansion. I still think that fully caching the instance time-ranges is better - free busy queries need to be done fast.
I agree. Because of this I think I would actually keep the freebusy data in a separate table/store and update that asynchronously. But anyways ;-) OK. This is getting a bit off-topic. I guess I need to ask the user whether they are OK with a 400 COUNT coming up in Outlook. At least our QA guy marked plugin RRULE generation as broken because his selection wasn't preserved ;-) I wonder whether it would be better if the server would reject RRULEs it can't store with a proper error code. (which tells the client that the user must specify a smaller COUNT, smaller UNTIL, etc - we could show that to the user in Outlook). Thanks, Helge -- Helge Hess http://zideone.com/
participants (2)
-
Cyrus Daboo
-
Helge Heß