[CalendarServer-dev] Fixing #207

[Peter Mogensen]

Hi,

I plan an attempt at fixing ticket #207 and would like to hear if 
there's any comments.

SUMMARY (of current situaion):
Events taking place more than 356 days from creation time will never be 
returned by indexed queries. Not even when you approach them.
This means that events like "birthdays" will not work in the calendar.
The iCal client is not affected by this, since it uses multiget REPORTS 
and ctag/etag to query the server. Other clients are though.

DIAGNOSIS:
* When events are store (PUT) they are expanded up to 356 days and any 
instances are put into the TIMESPAN index. A note (RECURRANCE_MAX) is 
made in the RESOURCE index of how far into the future TIMESPAN is valid.
* When a calendar-query REPORT is made it will use an indexed query 
unless the query-parser/generator throws an exception saying that it 
does not handle such a query. This exception is currently not thrown 
when a comp-filter has a time-range element.
* This means that indexed queries will still be used even if not all 
instances within the query time-range are in the TIMESPAN index.
RECURRANCE_MAX is never updated or used.

SOLUTION (wrt. the 1.2 code base):
The solution must consist of two parts:
1) Recognize that a query will need to expand RECURRANCE_MAX.
2) Actually do the expansion for at least those events which
    has RECURRANCE_MAX < "end" for the time-range specified in
    the query.

ad 1) The handling of filters is done in
   twistedcaldav/query/calendarquery.py:sqlcalendarquery()
   which is called from index.py:search() and searchValid()
   The code in twistedcaldav/query doesn't have access to the index, so
   any checking of RECURRANCE_MAX must be done in search()/searchValid()
   But only the code in twistedcaldav/query knows if there is a
   time-range and it is necessary to check RECURRANCE_MAX.
   So we need to change the prototype of sqlcalendarquery() to tell
   the index not only the SQL qualifiers, but also what it demands of
   the index to be able to use them. (like RECURRANCE_MAX > some-value -
   taken from time-range "end").
   If search()/searchValid() gets such a demand, it must check the
   RESOURCE table and maybe do further expansion of events before running
   the query.

ad 2) The easiest way here seems to call ical.py:expandTimeRanges() with
   a new limit and either replace the index values as if the resource has
   been PUT again, or filter out those instances which are before
   RECURRANCE_MAX and add the rest to the index.

/Peter