<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head><meta http-equiv="content-type" content="text/html; charset=utf-8" />
<title>[12459] CalendarServer/branches/users/cdaboo/scheduling-queue-refresh</title>
</head>
<body>

<style type="text/css"><!--
#msg dl.meta { border: 1px #006 solid; background: #369; padding: 6px; color: #fff; }
#msg dl.meta dt { float: left; width: 6em; font-weight: bold; }
#msg dt:after { content:':';}
#msg dl, #msg dt, #msg ul, #msg li, #header, #footer, #logmsg { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt;  }
#msg dl a { font-weight: bold}
#msg dl a:link    { color:#fc3; }
#msg dl a:active  { color:#ff0; }
#msg dl a:visited { color:#cc6; }
h3 { font-family: verdana,arial,helvetica,sans-serif; font-size: 10pt; font-weight: bold; }
#msg pre { overflow: auto; background: #ffc; border: 1px #fa0 solid; padding: 6px; }
#logmsg { background: #ffc; border: 1px #fa0 solid; padding: 1em 1em 0 1em; }
#logmsg p, #logmsg pre, #logmsg blockquote { margin: 0 0 1em 0; }
#logmsg p, #logmsg li, #logmsg dt, #logmsg dd { line-height: 14pt; }
#logmsg h1, #logmsg h2, #logmsg h3, #logmsg h4, #logmsg h5, #logmsg h6 { margin: .5em 0; }
#logmsg h1:first-child, #logmsg h2:first-child, #logmsg h3:first-child, #logmsg h4:first-child, #logmsg h5:first-child, #logmsg h6:first-child { margin-top: 0; }
#logmsg ul, #logmsg ol { padding: 0; list-style-position: inside; margin: 0 0 0 1em; }
#logmsg ul { text-indent: -1em; padding-left: 1em; }#logmsg ol { text-indent: -1.5em; padding-left: 1.5em; }
#logmsg > ul, #logmsg > ol { margin: 0 0 1em 0; }
#logmsg pre { background: #eee; padding: 1em; }
#logmsg blockquote { border: 1px solid #fa0; border-left-width: 10px; padding: 1em 1em 0 1em; background: white;}
#logmsg dl { margin: 0; }
#logmsg dt { font-weight: bold; }
#logmsg dd { margin: 0; padding: 0 0 0.5em 0; }
#logmsg dd:before { content:'\00bb';}
#logmsg table { border-spacing: 0px; border-collapse: collapse; border-top: 4px solid #fa0; border-bottom: 1px solid #fa0; background: #fff; }
#logmsg table th { text-align: left; font-weight: normal; padding: 0.2em 0.5em; border-top: 1px dotted #fa0; }
#logmsg table td { text-align: right; border-top: 1px dotted #fa0; padding: 0.2em 0.5em; }
#logmsg table thead th { text-align: center; border-bottom: 1px solid #fa0; }
#logmsg table th.Corner { text-align: left; }
#logmsg hr { border: none 0; border-top: 2px dashed #fa0; height: 1px; }
#header, #footer { color: #fff; background: #636; border: 1px #300 solid; padding: 6px; }
#patch { width: 100%; }
#patch h4 {font-family: verdana,arial,helvetica,sans-serif;font-size:10pt;padding:8px;background:#369;color:#fff;margin:0;}
#patch .propset h4, #patch .binary h4 {margin:0;}
#patch pre {padding:0;line-height:1.2em;margin:0;}
#patch .diff {width:100%;background:#eee;padding: 0 0 10px 0;overflow:auto;}
#patch .propset .diff, #patch .binary .diff  {padding:10px 0;}
#patch span {display:block;padding:0 10px;}
#patch .modfile, #patch .addfile, #patch .delfile, #patch .propset, #patch .binary, #patch .copfile {border:1px solid #ccc;margin:10px 0;}
#patch ins {background:#dfd;text-decoration:none;display:block;padding:0 10px;}
#patch del {background:#fdd;text-decoration:none;display:block;padding:0 10px;}
#patch .lines, .info {color:#888;background:#fff;}
--></style>
<div id="msg">
<dl class="meta">
<dt>Revision</dt> <dd><a href="http://trac.calendarserver.org//changeset/12459">12459</a></dd>
<dt>Author</dt> <dd>cdaboo@apple.com</dd>
<dt>Date</dt> <dd>2014-01-27 17:25:30 -0800 (Mon, 27 Jan 2014)</dd>
</dl>

<h3>Log Message</h3>
<pre>Checkpoint: changed iTIP message processing to treat the attendee partstat state as definitive when sequence has not changed. This
ensures out of sequence request/replies do not mess up valid attendee states.</pre>

<h3>Modified Paths</h3>
<ul>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshconfcaldavdtestplist">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/conf/caldavd-test.plist</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtwistedcaldavstdconfigpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/twistedcaldav/stdconfig.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingicaldiffpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/icaldiff.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingimplicitpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/implicit.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingitippy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/itip.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingprocessingpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/processing.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingtesttest_icaldiffpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_icaldiff.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingtesttest_implicitpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_implicit.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingtesttest_itippy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_itip.py</a></li>
<li><a href="#CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingworkpy">CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/work.py</a></li>
</ul>

</div>
<div id="patch">
<h3>Diff</h3>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshconfcaldavdtestplist"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/conf/caldavd-test.plist (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/conf/caldavd-test.plist        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/conf/caldavd-test.plist        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -792,6 +792,22 @@
</span><span class="cx">                 &lt;key&gt;DefaultMode&lt;/key&gt;
</span><span class="cx">                 &lt;string&gt;automatic&lt;/string&gt;
</span><span class="cx">                 &lt;/dict&gt;
</span><ins>+                
+                &lt;key&gt;WorkQueues&lt;/key&gt;
+                &lt;dict&gt;
+                        &lt;key&gt;Enabled&lt;/key&gt;
+                        &lt;false/&gt;
+            &lt;key&gt;RequestDelaySeconds&lt;/key&gt;
+            &lt;integer&gt;1&lt;/integer&gt;
+            &lt;key&gt;ReplyDelaySeconds&lt;/key&gt;
+            &lt;integer&gt;2&lt;/integer&gt;
+            &lt;key&gt;AutoReplyDelaySeconds&lt;/key&gt;
+            &lt;integer&gt;1&lt;/integer&gt;
+            &lt;key&gt;AttendeeRefreshBatchDelaySeconds&lt;/key&gt;
+            &lt;integer&gt;2&lt;/integer&gt;
+            &lt;key&gt;AttendeeRefreshBatchIntervalSeconds&lt;/key&gt;
+            &lt;integer&gt;2&lt;/integer&gt;
+                &lt;/dict&gt;
</ins><span class="cx">       &lt;/dict&gt;
</span><span class="cx">     &lt;/dict&gt;
</span><span class="cx"> 
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtwistedcaldavstdconfigpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/twistedcaldav/stdconfig.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/twistedcaldav/stdconfig.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/twistedcaldav/stdconfig.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -737,12 +737,7 @@
</span><span class="cx">             &quot;TrackUnscheduledResourceData&quot;        : True, # Track who the last modifier of an unscheduled resource event is
</span><span class="cx">             &quot;LimitFreeBusyAttendees&quot;              : 30, # Maximum number of attendees to request freebusy for
</span><span class="cx">             &quot;AttendeeRefreshBatch&quot;                : 5, # Number of attendees to do batched refreshes: 0 - no batching
</span><del>-            &quot;AttendeeRefreshBatchDelaySeconds&quot;    : 5, # Time after an iTIP REPLY for first batched attendee refresh
-            &quot;AttendeeRefreshBatchIntervalSeconds&quot; : 5, # Time between attendee batch refreshes
</del><span class="cx">             &quot;AttendeeRefreshCountLimit&quot;           : 50, # Number of attendees above which attendee refreshes are suppressed: 0 - no limit
</span><del>-            &quot;AutoReplyDelaySeconds&quot;               : 5, # Time delay for sending an auto reply iTIP message
-            &quot;QueuedRequestDelaySeconds&quot;           : 5, # Number of seconds delay for a queued scheduling request/cancel
-            &quot;QueuedReplyDelaySeconds&quot;             : 1, # Number of seconds delay for a queued scheduling reply
</del><span class="cx">             &quot;UIDLockTimeoutSeconds&quot;               : 60, # Time for implicit UID lock timeout
</span><span class="cx">             &quot;UIDLockExpirySeconds&quot;                : 300, # Expiration time for UID lock,
</span><span class="cx">             &quot;PrincipalHostAliases&quot;                : [], # Host names matched in http(s) CUAs
</span><span class="lines">@@ -765,6 +760,15 @@
</span><span class="cx">                 &quot;FutureFreeBusyDays&quot;              : 3 * 365,       # How far into the future to check for booking conflicts
</span><span class="cx">             },
</span><span class="cx"> 
</span><ins>+            &quot;WorkQueues&quot; : {
+                &quot;Enabled&quot;                             : False,       # Work queues for scheduling enabled
+                &quot;RequestDelaySeconds&quot;                 : 5,          # Number of seconds delay for a queued scheduling request/cancel
+                &quot;ReplyDelaySeconds&quot;                   : 1,          # Number of seconds delay for a queued scheduling reply
+                &quot;AutoReplyDelaySeconds&quot;               : 5,          # Time delay for sending an auto reply iTIP message
+                &quot;AttendeeRefreshBatchDelaySeconds&quot;    : 5,          # Time after an iTIP REPLY for first batched attendee refresh
+                &quot;AttendeeRefreshBatchIntervalSeconds&quot; : 5,          # Time between attendee batch refreshes
+            },
+
</ins><span class="cx">             &quot;Splitting&quot;: {
</span><span class="cx">                 &quot;Enabled&quot;                         : False,          # False for now whilst we experiment with this
</span><span class="cx">                 &quot;Size&quot;                            : 100 * 1024,     # Consider splitting when greater than 100KB
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingicaldiffpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/icaldiff.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/icaldiff.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/icaldiff.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -774,6 +774,75 @@
</span><span class="cx">         return rids
</span><span class="cx"> 
</span><span class="cx"> 
</span><ins>+    def attendeeNeedsAction(self, diffs):
+        &quot;&quot;&quot;
+        Given a set of results from L{whatIsDifferent}, determine which recurrence-id's
+        have changed in such a way that we need to force the attendee to NEEDS-ACTION
+        state. We need this because with async processing of iTIP messages, it is possible
+        that the incoming iTIP message does not gave the most recent partstat change an
+        attendee made, but we need to preserve the attendee partstat whenever possible.
+
+        @param diffs: mapping of rid to property changes
+        @type diffs: C{dict}
+        &quot;&quot;&quot;
+
+        date_changed_rids = set()
+        recurrence_reschedule = False
+
+        for rid, props in diffs.iteritems():
+            if any([testprop in props for testprop in (
+                &quot;DTSTART&quot;,
+                &quot;DTEND&quot;,
+                &quot;DURATION&quot;,
+                &quot;DUE&quot;,
+                &quot;RECURRENCE-ID&quot;,
+            )]):
+                date_changed_rids.add(rid)
+
+            # Check to see whether a change to R-ID's happened
+            if rid is None:
+
+                if &quot;DTSTART&quot; in props and self.newcalendar.masterComponent().hasProperty(&quot;RRULE&quot;):
+                    # DTSTART change with RRULE present is always a reschedule
+                    recurrence_reschedule = True
+
+                elif &quot;RRULE&quot; in props:
+
+                    # Need to see if the RRULE change is a simple truncation or expansion - i.e. a change to
+                    # COUNT or UNTIL only. If so we don't need to treat this as a complete re-schedule.
+
+                    # Start off assuming they are different
+                    recurrence_reschedule = True
+
+                    # Get each RRULE (can be only one in the master)
+                    oldrrule = tuple(self.oldcalendar.masterComponent().properties(&quot;RRULE&quot;))
+                    oldrrule = oldrrule[0].value() if len(oldrrule) else None
+                    newrrule = tuple(self.newcalendar.masterComponent().properties(&quot;RRULE&quot;))
+                    newrrule = newrrule[0].value() if len(newrrule) else None
+
+                    if newrrule is not None and oldrrule is not None:
+
+                        # Normalize the rrules by removing COUNT/UNTIL and then compare
+                        oldrrule = oldrrule.duplicate()
+                        newrrule = newrrule.duplicate()
+
+                        oldrrule.setUseUntil(False)
+                        oldrrule.setUntil(None)
+                        oldrrule.setUseCount(False)
+                        oldrrule.setCount(0)
+
+                        newrrule.setUseUntil(False)
+                        newrrule.setUntil(None)
+                        newrrule.setUseCount(False)
+                        newrrule.setCount(0)
+
+                        # If they are equal we have a simple change - no overall reschedule
+                        if newrrule == oldrrule:
+                            recurrence_reschedule = False
+
+        return (date_changed_rids, recurrence_reschedule,)
+
+
</ins><span class="cx">     def _componentDuplicateAndNormalize(self, comp):
</span><span class="cx">         comp = comp.duplicate()
</span><span class="cx">         comp.normalizePropertyValueLists(&quot;EXDATE&quot;)
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingimplicitpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/implicit.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/implicit.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/implicit.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -596,7 +596,7 @@
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitOrganizer(self, queued=False):
</span><span class="cx"> 
</span><del>-        if not queued:
</del><ins>+        if not queued or not config.Scheduling.Options.WorkQueues.Enabled:
</ins><span class="cx">             self.oldcalendar = None
</span><span class="cx">         self.changed_rids = None
</span><span class="cx">         self.cancelledAttendees = ()
</span><span class="lines">@@ -615,21 +615,21 @@
</span><span class="cx">             self.cancelledAttendees = [(attendee, None) for attendee in self.attendees]
</span><span class="cx"> 
</span><span class="cx">             # CANCEL always bumps sequence
</span><del>-            if not queued:
</del><ins>+            if not queued or not config.Scheduling.Options.WorkQueues.Enabled:
</ins><span class="cx">                 self.needs_sequence_change = True
</span><span class="cx"> 
</span><span class="cx">         # Check for a new resource or an update
</span><span class="cx">         elif self.action == &quot;modify&quot;:
</span><span class="cx"> 
</span><span class="cx">             # Read in existing data
</span><del>-            if not queued:
</del><ins>+            if not queued or not config.Scheduling.Options.WorkQueues.Enabled:
</ins><span class="cx">                 self.oldcalendar = (yield self.resource.componentForUser())
</span><span class="cx">             self.oldAttendeesByInstance = self.oldcalendar.getAttendeesByInstance(True, onlyScheduleAgentServer=True)
</span><span class="cx">             self.oldInstances = set(self.oldcalendar.getComponentInstances())
</span><span class="cx">             self.coerceAttendeesPartstatOnModify()
</span><span class="cx"> 
</span><span class="cx">             # Don't allow any SEQUENCE to decrease
</span><del>-            if self.oldcalendar and not queued:
</del><ins>+            if self.oldcalendar and (not queued or not config.Scheduling.Options.WorkQueues.Enabled):
</ins><span class="cx">                 self.calendar.sequenceInSync(self.oldcalendar)
</span><span class="cx"> 
</span><span class="cx">             # Significant change
</span><span class="lines">@@ -678,7 +678,7 @@
</span><span class="cx"> 
</span><span class="cx">                 # For now we always bump the sequence number on modifications because we cannot track DTSTAMP on
</span><span class="cx">                 # the Attendee side. But we check the old and the new and only bump if the client did not already do it.
</span><del>-                if not queued:
</del><ins>+                if not queued or not config.Scheduling.Options.WorkQueues.Enabled:
</ins><span class="cx">                     self.needs_sequence_change = self.calendar.needsiTIPSequenceChange(self.oldcalendar)
</span><span class="cx"> 
</span><span class="cx">         elif self.action == &quot;create&quot;:
</span><span class="lines">@@ -699,7 +699,7 @@
</span><span class="cx"> 
</span><span class="cx">         # If processing a queue item, actually execute the scheduling operations, else queue it.
</span><span class="cx">         # Note a split is always queued, so we do not need to re-queue
</span><del>-        if queued or self.split_details is not None:
</del><ins>+        if queued or not config.Scheduling.Options.WorkQueues.Enabled or self.split_details is not None:
</ins><span class="cx">             yield self.scheduleWithAttendees()
</span><span class="cx">         else:
</span><span class="cx">             yield self.queuedScheduleWithAttendees()
</span><span class="lines">@@ -1550,22 +1550,24 @@
</span><span class="cx">         if self.logItems is not None:
</span><span class="cx">             self.logItems[&quot;itip.reply&quot;] = &quot;reply&quot;
</span><span class="cx"> 
</span><del>-#        itipmsg = iTipGenerator.generateAttendeeReply(self.calendar, self.attendee, changedRids=changedRids)
-#
-#        # Send scheduling message
-#        return self.sendToOrganizer(&quot;REPLY&quot;, itipmsg)
</del><ins>+        if config.Scheduling.Options.WorkQueues.Enabled:
+            # Always make it look like scheduling succeeded when queuing
+            self.calendar.setParameterToValueForPropertyWithValue(
+                &quot;SCHEDULE-STATUS&quot;,
+                iTIPRequestStatus.MESSAGE_DELIVERED_CODE,
+                &quot;ORGANIZER&quot;,
+                self.organizer,
+            )
</ins><span class="cx"> 
</span><del>-        # Always make it look like scheduling succeeded when queuing
-        self.calendar.setParameterToValueForPropertyWithValue(
-            &quot;SCHEDULE-STATUS&quot;,
-            iTIPRequestStatus.MESSAGE_DELIVERED_CODE,
-            &quot;ORGANIZER&quot;,
-            self.organizer,
-        )
</del><ins>+            return ScheduleReplyWork.reply(self.txn, self.calendar_home, self.resource, changedRids, self.attendee)
</ins><span class="cx"> 
</span><del>-        return ScheduleReplyWork.reply(self.txn, self.calendar_home, self.resource, changedRids, self.attendee)
</del><ins>+        else:
+            itipmsg = iTipGenerator.generateAttendeeReply(self.calendar, self.attendee, changedRids=changedRids)
</ins><span class="cx"> 
</span><ins>+            # Send scheduling message
+            return self.sendToOrganizer(&quot;REPLY&quot;, itipmsg)
</ins><span class="cx"> 
</span><ins>+
</ins><span class="cx">     def scheduleCancelWithOrganizer(self):
</span><span class="cx"> 
</span><span class="cx">         # First make sure we are allowed to schedule
</span><span class="lines">@@ -1574,14 +1576,16 @@
</span><span class="cx">         if self.logItems is not None:
</span><span class="cx">             self.logItems[&quot;itip.reply&quot;] = &quot;cancel&quot;
</span><span class="cx"> 
</span><del>-#        itipmsg = iTipGenerator.generateAttendeeReply(self.calendar, self.attendee, force_decline=True)
-#
-#        # Send scheduling message
-#        return self.sendToOrganizer(&quot;CANCEL&quot;, itipmsg)
</del><ins>+        if config.Scheduling.Options.WorkQueues.Enabled:
+            return ScheduleReplyCancelWork.replyCancel(self.txn, self.calendar_home, self.calendar, self.attendee)
</ins><span class="cx"> 
</span><del>-        return ScheduleReplyCancelWork.replyCancel(self.txn, self.calendar_home, self.calendar, self.attendee)
</del><ins>+        else:
+            itipmsg = iTipGenerator.generateAttendeeReply(self.calendar, self.attendee, force_decline=True)
</ins><span class="cx"> 
</span><ins>+            # Send scheduling message
+            return self.sendToOrganizer(&quot;CANCEL&quot;, itipmsg)
</ins><span class="cx"> 
</span><ins>+
</ins><span class="cx">     @inlineCallbacks
</span><span class="cx">     def sendToOrganizer(self, action, itipmsg):
</span><span class="cx"> 
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingitippy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/itip.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/itip.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/itip.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -67,10 +67,11 @@
</span><span class="cx">             calendar.removeProperty(method)
</span><span class="cx"> 
</span><span class="cx">         if recipient:
</span><del>-            iTipProcessing.addTranspForNeedsAction(calendar.subcomponents(), recipient)
</del><span class="cx"> 
</span><span class="cx">             # Check for incoming DECLINED
</span><span class="cx">             if creating:
</span><ins>+                iTipProcessing.addTranspForNeedsAction(calendar.subcomponents(), recipient)
+
</ins><span class="cx">                 master = calendar.masterComponent()
</span><span class="cx">                 for component in tuple(calendar.subcomponents()):
</span><span class="cx">                     if component in ignoredComponents or component is master:
</span><span class="lines">@@ -111,14 +112,16 @@
</span><span class="cx"> 
</span><span class="cx">         # Merge Organizer data with Attendee's own changes (VALARMs, Comment only for now).
</span><span class="cx">         from txdav.caldav.datastore.scheduling.icaldiff import iCalDiff
</span><del>-        rids = iCalDiff(calendar, itip_message, False).whatIsDifferent()
</del><ins>+        differ = iCalDiff(calendar, itip_message, False)
+        rids = differ.whatIsDifferent()
+        needs_action_rids, reschedule = differ.attendeeNeedsAction(rids)
</ins><span class="cx"> 
</span><span class="cx">         # Different behavior depending on whether a master component is present or not
</span><span class="cx">         # Here we cache per-attendee data from the existing master that we need to use in any new
</span><span class="cx">         # overridden components that the organizer added
</span><span class="cx">         current_master = calendar.masterComponent()
</span><span class="cx">         if current_master:
</span><del>-            master_valarms = [comp for comp in current_master.subcomponents() if comp.name() == &quot;VALARM&quot;]
</del><ins>+            valarms = [comp for comp in current_master.subcomponents() if comp.name() == &quot;VALARM&quot;]
</ins><span class="cx">             private_comments = current_master.properties(&quot;X-CALENDARSERVER-PRIVATE-COMMENT&quot;)
</span><span class="cx">             transps = current_master.properties(&quot;TRANSP&quot;)
</span><span class="cx">             completeds = current_master.properties(&quot;COMPLETED&quot;)
</span><span class="lines">@@ -132,11 +135,12 @@
</span><span class="cx">                 if props:
</span><span class="cx">                     other_props[pname] = props
</span><span class="cx">         else:
</span><del>-            master_valarms = ()
</del><ins>+            valarms = ()
</ins><span class="cx">             private_comments = ()
</span><span class="cx">             transps = ()
</span><span class="cx">             completeds = ()
</span><span class="cx">             organizer_schedule_status = None
</span><ins>+            attendee = None
</ins><span class="cx">             attendee_dtstamp = None
</span><span class="cx">             other_props = {}
</span><span class="cx"> 
</span><span class="lines">@@ -145,31 +149,16 @@
</span><span class="cx">             # Get a new calendar object first
</span><span class="cx">             new_calendar = iTipProcessing.processNewRequest(itip_message, recipient)
</span><span class="cx"> 
</span><del>-            # Copy over master alarms, comments
</del><ins>+            # Copy over master alarms, comments etc
</ins><span class="cx">             master_component = new_calendar.masterComponent()
</span><del>-            for alarm in master_valarms:
-                master_component.addComponent(alarm)
-            for comment in private_comments:
-                master_component.addProperty(comment)
-            for transp in transps:
-                master_component.replaceProperty(transp)
-            for completed in completeds:
-                master_component.replaceProperty(completed)
-            if organizer_schedule_status:
-                organizer = master_component.getProperty(&quot;ORGANIZER&quot;)
-                if organizer:
-                    organizer.setParameter(&quot;SCHEDULE-STATUS&quot;, organizer_schedule_status)
-            if attendee_dtstamp:
-                attendee = master_component.getAttendeeProperty((recipient,))
-                if attendee:
-                    attendee.setParameter(&quot;X-CALENDARSERVER-DTSTAMP&quot;, attendee_dtstamp)
-            for props in other_props.values():
-                [master_component.replaceProperty(prop) for prop in props]
</del><ins>+            transfer_partstat = None not in needs_action_rids and not reschedule
+            seq_change = Component.compareComponentsForITIP(master_component, current_master, use_dtstamp=False) &lt;= 0
+            iTipProcessing._transferItems(master_component, transfer_partstat and seq_change, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient)
</ins><span class="cx"> 
</span><span class="cx">             # Now try to match recurrences in the new calendar
</span><span class="cx">             for component in tuple(new_calendar.subcomponents()):
</span><span class="cx">                 if component.name() != &quot;VTIMEZONE&quot; and component.getRecurrenceIDUTC() is not None:
</span><del>-                    iTipProcessing.transferItems(calendar, component, master_valarms, private_comments, transps, completeds, organizer_schedule_status, attendee_dtstamp, other_props, recipient)
</del><ins>+                    iTipProcessing.transferItems(calendar, component, needs_action_rids, reschedule, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient)
</ins><span class="cx"> 
</span><span class="cx">             # Now try to match recurrences from the old calendar
</span><span class="cx">             for component in calendar.subcomponents():
</span><span class="lines">@@ -181,10 +170,12 @@
</span><span class="cx">                         new_component = new_calendar.deriveInstance(rid, allowCancelled=allowCancelled and not hidden)
</span><span class="cx">                         if new_component is not None:
</span><span class="cx">                             new_calendar.addComponent(new_component)
</span><del>-                            iTipProcessing.transferItems(calendar, new_component, master_valarms, private_comments, transps, completeds, organizer_schedule_status, attendee_dtstamp, other_props, recipient)
</del><ins>+                            iTipProcessing.transferItems(calendar, new_component, needs_action_rids, reschedule, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient)
</ins><span class="cx">                             if hidden:
</span><span class="cx">                                 new_component.addProperty(Property(Component.HIDDEN_INSTANCE_PROPERTY, &quot;T&quot;))
</span><span class="cx"> 
</span><ins>+            iTipProcessing.addTranspForNeedsAction(new_calendar.subcomponents(), recipient)
+
</ins><span class="cx">             # Replace the entire object
</span><span class="cx">             return new_calendar, rids
</span><span class="cx"> 
</span><span class="lines">@@ -200,12 +191,12 @@
</span><span class="cx">                         calendar.addComponent(component)
</span><span class="cx">                 else:
</span><span class="cx">                     component = component.duplicate()
</span><del>-                    missingDeclined = iTipProcessing.transferItems(calendar, component, master_valarms, private_comments, transps, completeds, organizer_schedule_status, attendee_dtstamp, other_props, recipient, remove_matched=True)
</del><ins>+                    missingDeclined = iTipProcessing.transferItems(calendar, component, needs_action_rids, reschedule, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient, remove_matched=True)
</ins><span class="cx">                     if not missingDeclined:
</span><span class="cx">                         calendar.addComponent(component)
</span><del>-                        if recipient:
-                            iTipProcessing.addTranspForNeedsAction((component,), recipient)
</del><span class="cx"> 
</span><ins>+            iTipProcessing.addTranspForNeedsAction(calendar.subcomponents(), recipient)
+
</ins><span class="cx">             # Write back the modified object
</span><span class="cx">             return calendar, rids
</span><span class="cx"> 
</span><span class="lines">@@ -558,7 +549,7 @@
</span><span class="cx"> 
</span><span class="cx"> 
</span><span class="cx">     @staticmethod
</span><del>-    def transferItems(from_calendar, to_component, master_valarms, private_comments, transps, completeds, organizer_schedule_status, attendee_dtstamp, other_props, recipient, remove_matched=False):
</del><ins>+    def transferItems(from_calendar, to_component, needs_action_rids, reschedule, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient, remove_matched=False):
</ins><span class="cx">         &quot;&quot;&quot;
</span><span class="cx">         Transfer properties from a calendar to a component by first trying to match the component in the original calendar and
</span><span class="cx">         use the properties from that, or use the values provided as arguments (which have been derived from the original calendar's
</span><span class="lines">@@ -568,8 +559,8 @@
</span><span class="cx">         @type from_calendar: L{Component}
</span><span class="cx">         @param to_component: the new component to transfer items to
</span><span class="cx">         @type to_component: L{Component}
</span><del>-        @param master_valarms: a C{list} of VALARM components from the old master to use
-        @type master_valarms: C{list}
</del><ins>+        @param valarms: a C{list} of VALARM components from the old master to use
+        @type valarms: C{list}
</ins><span class="cx">         @param private_comments: a C{list} of private comment properties from the old master to use
</span><span class="cx">         @type private_comments: C{list}
</span><span class="cx">         @param transps: a C{list} of TRANSP properties from the old master to use
</span><span class="lines">@@ -592,44 +583,43 @@
</span><span class="cx"> 
</span><span class="cx">         rid = to_component.getRecurrenceIDUTC()
</span><span class="cx"> 
</span><ins>+        transfer_partstat = rid not in needs_action_rids and not reschedule
+
</ins><span class="cx">         # Is there a matching component
</span><span class="cx">         matched = from_calendar.overriddenComponent(rid)
</span><span class="cx">         if matched:
</span><del>-            # Copy over VALARMs from existing component
-            [to_component.addComponent(comp) for comp in matched.subcomponents() if comp.name() == &quot;VALARM&quot;]
-            [to_component.addProperty(prop) for prop in matched.properties(&quot;X-CALENDARSERVER-ATTENDEE-COMMENT&quot;)]
-            [to_component.replaceProperty(prop) for prop in matched.properties(&quot;TRANSP&quot;)]
-            [to_component.replaceProperty(prop) for prop in matched.properties(&quot;COMPLETED&quot;)]
-
</del><ins>+            valarms = [comp for comp in matched.subcomponents() if comp.name() == &quot;VALARM&quot;]
+            private_comments = matched.properties(&quot;X-CALENDARSERVER-PRIVATE-COMMENT&quot;)
+            transps = matched.properties(&quot;TRANSP&quot;)
+            completeds = matched.properties(&quot;COMPLETED&quot;)
</ins><span class="cx">             organizer = matched.getProperty(&quot;ORGANIZER&quot;)
</span><span class="cx">             organizer_schedule_status = organizer.parameterValue(&quot;SCHEDULE-STATUS&quot;, None) if organizer else None
</span><del>-            if organizer_schedule_status:
-                organizer = to_component.getProperty(&quot;ORGANIZER&quot;)
-                if organizer:
-                    organizer.setParameter(&quot;SCHEDULE-STATUS&quot;, organizer_schedule_status)
</del><ins>+            attendee = matched.getAttendeeProperty((recipient,))
+            attendee_dtstamp = attendee.parameterValue(&quot;X-CALENDARSERVER-DTSTAMP&quot;) if attendee else None
+            other_props = {}
+            for pname in config.Scheduling.CalDAV.PerAttendeeProperties:
+                props = tuple(matched.properties(pname))
+                if props:
+                    other_props[pname] = props
</ins><span class="cx"> 
</span><del>-            # Remove the old one
-            if remove_matched:
-                from_calendar.removeComponent(matched)
</del><ins>+            seq_change = Component.compareComponentsForITIP(to_component, matched, use_dtstamp=False) &lt;= 0
+            iTipProcessing._transferItems(to_component, transfer_partstat and seq_change, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient)
</ins><span class="cx"> 
</span><span class="cx">             # Check for incoming DECLINED
</span><del>-            attendee = to_component.getAttendeeProperty((recipient,))
-            if attendee and attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;) == &quot;DECLINED&quot;:
</del><ins>+            to_attendee = to_component.getAttendeeProperty((recipient,))
+            if to_attendee and to_attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;) == &quot;DECLINED&quot;:
</ins><span class="cx">                 # If existing item has HIDDEN property copy that over
</span><span class="cx">                 if matched.hasProperty(Component.HIDDEN_INSTANCE_PROPERTY):
</span><span class="cx">                     to_component.addProperty(Property(Component.HIDDEN_INSTANCE_PROPERTY, &quot;T&quot;))
</span><span class="cx"> 
</span><del>-            if attendee and attendee_dtstamp:
-                attendee.setParameter(&quot;X-CALENDARSERVER-DTSTAMP&quot;, attendee_dtstamp)
</del><ins>+            # Remove the old one
+            if remove_matched:
+                from_calendar.removeComponent(matched)
</ins><span class="cx"> 
</span><del>-            for pname in config.Scheduling.CalDAV.PerAttendeeProperties:
-                [to_component.replaceProperty(prop) for prop in matched.properties(pname)]
-
</del><span class="cx">             # Check to see if the new component is cancelled as that could mean we are copying in the wrong attendee state
</span><span class="cx">             if to_component.propertyValue(&quot;STATUS&quot;) == &quot;CANCELLED&quot;:
</span><del>-                from_attendee = matched.getAttendeeProperty((recipient,))
-                if attendee and from_attendee:
-                    attendee.setParameter(&quot;PARTSTAT&quot;, from_attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;))
</del><ins>+                if attendee and to_attendee:
+                    to_attendee.setParameter(&quot;PARTSTAT&quot;, attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;))
</ins><span class="cx"> 
</span><span class="cx">         else:
</span><span class="cx">             # Check for incoming DECLINED
</span><span class="lines">@@ -637,29 +627,89 @@
</span><span class="cx">             if attendee and attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;) == &quot;DECLINED&quot;:
</span><span class="cx">                 return True
</span><span class="cx"> 
</span><del>-            # It is a new override - copy any valarms on the existing master component
-            # into the new one.
-            [to_component.addComponent(alarm) for alarm in master_valarms]
-            [to_component.addProperty(comment) for comment in private_comments]
-            [to_component.replaceProperty(transp) for transp in transps]
-            [to_component.replaceProperty(completed) for completed in completeds]
</del><ins>+            master_component = from_calendar.masterComponent()
+            seq_change = (Component.compareComponentsForITIP(to_component, master_component, use_dtstamp=False) &lt;= 0) if master_component is not None else True
+            iTipProcessing._transferItems(to_component, transfer_partstat and seq_change, valarms, private_comments, transps, completeds, organizer_schedule_status, attendee, attendee_dtstamp, other_props, recipient)
</ins><span class="cx"> 
</span><del>-            if organizer_schedule_status:
-                organizer = to_component.getProperty(&quot;ORGANIZER&quot;)
-                if organizer:
-                    organizer.setParameter(&quot;SCHEDULE-STATUS&quot;, organizer_schedule_status)
-            if attendee_dtstamp:
-                attendee = to_component.getAttendeeProperty((recipient,))
-                if attendee:
-                    attendee.setParameter(&quot;X-CALENDARSERVER-DTSTAMP&quot;, attendee_dtstamp)
</del><ins>+        return False
</ins><span class="cx"> 
</span><del>-            for props in other_props.values():
-                [to_component.replaceProperty(prop) for prop in props]
</del><span class="cx"> 
</span><ins>+    @staticmethod
+    def _transferItems(to_component, transfer_partstat, valarms, private_comments, transps, completeds, organizer_schedule_status, old_attendee, attendee_dtstamp, other_props, recipient):
+        &quot;&quot;&quot;
+        Transfer properties the key per-attendee properties from one component to another. Note that the key properties are pulled out into separate items, because they
+        may have been derived from the master.
+
+        @param to_component: the new component to transfer items to
+        @type to_component: L{Component}
+        @param partstat_change: whether not to transfer the old PARTSTAT over
+        @type partstat_change: C{bool}
+        @param valarms: a C{list} of VALARM components from the old master to use
+        @type valarms: C{list}
+        @param private_comments: a C{list} of private comment properties from the old master to use
+        @type private_comments: C{list}
+        @param transps: a C{list} of TRANSP properties from the old master to use
+        @type transps: C{list}
+        @param completeds: a C{list} of COMPLETED properties from the old master to use
+        @type completeds: C{list}
+        @param organizer_schedule_status: a the SCHEDULE-STATUS value for the organizer from the old master to use
+        @type organizer_schedule_status: C{str}
+        @param attendee_dtstamp: an the ATTENDEE DTSTAMP parameter value from the old master to use
+        @type attendee_dtstamp: C{str}
+        @param other_props: other properties from the old master to use
+        @type other_props: C{list}
+        @param recipient: the calendar user address of the attendee whose data is being processed
+        @type recipient: C{str}
+
+        @return: C{True} if an EXDATE match occurred requiring the incoming component to be removed.
+        &quot;&quot;&quot;
+
+        # It is a new override - copy any valarms on the existing master component
+        # into the new one.
+        [to_component.addComponent(alarm) for alarm in valarms]
+        [to_component.addProperty(comment) for comment in private_comments]
+        [to_component.replaceProperty(transp) for transp in transps]
+        [to_component.replaceProperty(completed) for completed in completeds]
+
+        if organizer_schedule_status:
+            organizer = to_component.getProperty(&quot;ORGANIZER&quot;)
+            if organizer:
+                organizer.setParameter(&quot;SCHEDULE-STATUS&quot;, organizer_schedule_status)
+
+        # ATTENDEE property merge
+        attendee = to_component.getAttendeeProperty((recipient,))
+        if old_attendee and attendee and transfer_partstat:
+            iTipProcessing.mergePartStat(old_attendee, attendee)
+
+        if attendee_dtstamp and attendee:
+            attendee.setParameter(&quot;X-CALENDARSERVER-DTSTAMP&quot;, attendee_dtstamp)
+
+        for props in other_props.values():
+            [to_component.replaceProperty(prop) for prop in props]
+
</ins><span class="cx">         return False
</span><span class="cx"> 
</span><span class="cx"> 
</span><span class="cx">     @staticmethod
</span><ins>+    def mergePartStat(from_attendee, to_attendee):
+        &quot;&quot;&quot;
+        Make sure the existing attendee PARTSTAT is preserved and also get rid of any RSVP
+        if the new PARTSTAT is not NEEDS-ACTION.
+
+        @param from_attendee: attendee property to copy PARTSTAT from
+        @type from_attendee: L{twistedcaldav.ical.Property}
+        @param to_attendee: attendee property to copy PARTSTAT to
+        @type to_attendee: L{twistedcaldav.ical.Property}
+        &quot;&quot;&quot;
+
+        preserve = from_attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;)
+        if preserve != to_attendee.parameterValue(&quot;PARTSTAT&quot;, &quot;NEEDS-ACTION&quot;):
+            to_attendee.setParameter(&quot;PARTSTAT&quot;, preserve)
+        if preserve != &quot;NEEDS-ACTION&quot;:
+            to_attendee.removeParameter(&quot;RSVP&quot;)
+
+
+    @staticmethod
</ins><span class="cx">     def addTranspForNeedsAction(components, recipient):
</span><span class="cx">         &quot;&quot;&quot;
</span><span class="cx">         For each component where the ATTENDEE property of the recipient has PARTSTAT
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingprocessingpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/processing.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/processing.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/processing.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -77,12 +77,12 @@
</span><span class="cx">         Do implicit processing of a scheduling message, and possibly also auto-process it
</span><span class="cx">         if the recipient has auto-accept on.
</span><span class="cx"> 
</span><del>-        @param message:
-        @type message:
-        @param originator:
-        @type originator:
-        @param recipient:
-        @type recipient:
</del><ins>+        @param message: the iTIP message
+        @type message: L{twistedcaldav.ical.Component}
+        @param originator: calendar user sending the message
+        @type originator: C{str}
+        @param recipient: calendar user receiving the message
+        @type recipient: C{str}
</ins><span class="cx"> 
</span><span class="cx">         @return: a C{tuple} of (C{bool}, C{bool}) indicating whether the message was processed, and if it was whether
</span><span class="cx">             auto-processing has taken place.
</span><span class="lines">@@ -164,6 +164,9 @@
</span><span class="cx"> 
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitOrganizer(self):
</span><ins>+        &quot;&quot;&quot;
+        Process an iTIP message sent to the organizer.
+        &quot;&quot;&quot;
</ins><span class="cx"> 
</span><span class="cx">         # Locate the organizer's copy of the event.
</span><span class="cx">         yield self.getRecipientsCopy()
</span><span class="lines">@@ -184,8 +187,12 @@
</span><span class="cx"> 
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitOrganizerUpdate(self):
</span><ins>+        &quot;&quot;&quot;
+        An iTIP REPLY has been sent by an attendee to an organizer and the attendee state needs to be sync'd
+        to the organizer's copy of the event.
+        &quot;&quot;&quot;
</ins><span class="cx"> 
</span><del>-        # Check to see if this is a valid reply
</del><ins>+        # Check to see if this is a valid reply - this will also merge the changes to the organizer's copy
</ins><span class="cx">         result, processed = iTipProcessing.processReply(self.message, self.recipient_calendar)
</span><span class="cx">         if result:
</span><span class="cx"> 
</span><span class="lines">@@ -247,7 +254,7 @@
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def queueAttendeeUpdate(self, exclude_attendees):
</span><span class="cx">         &quot;&quot;&quot;
</span><del>-        Queue up an update to attendees and use a memcache lock to ensure we don't update too frequently.
</del><ins>+        Queue up a background update to attendees.
</ins><span class="cx"> 
</span><span class="cx">         @param exclude_attendees: list of attendees who should not be refreshed (e.g., the one that triggered the refresh)
</span><span class="cx">         @type exclude_attendees: C{list}
</span><span class="lines">@@ -307,12 +314,16 @@
</span><span class="cx"> 
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitAttendee(self):
</span><ins>+        &quot;&quot;&quot;
+        Process an iTIP message sent to an attendee.
+        &quot;&quot;&quot;
</ins><span class="cx"> 
</span><span class="cx">         # Locate the attendee's copy of the event if it exists.
</span><span class="cx">         yield self.getRecipientsCopy()
</span><span class="cx">         self.new_resource = self.recipient_calendar is None
</span><span class="cx"> 
</span><del>-        # Handle new items differently than existing ones.
</del><ins>+        # If we get a CANCEL and we don't have a matching resource already stored, simply
+        # ignore the CANCEL.
</ins><span class="cx">         if self.new_resource and self.method == &quot;CANCEL&quot;:
</span><span class="cx">             result = (True, True, False, None)
</span><span class="cx">         else:
</span><span class="lines">@@ -323,6 +334,10 @@
</span><span class="cx"> 
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitAttendeeUpdate(self):
</span><ins>+        &quot;&quot;&quot;
+        An iTIP message has been sent by to an attendee by the organizer. We need to update the attendee state
+        based on the nature of the iTIP message.
+        &quot;&quot;&quot;
</ins><span class="cx"> 
</span><span class="cx">         # Do security check: ORGANZIER in iTIP MUST match existing resource value
</span><span class="cx">         if self.recipient_calendar:
</span><span class="lines">@@ -388,6 +403,10 @@
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitAttendeeRequest(self):
</span><span class="cx">         &quot;&quot;&quot;
</span><ins>+        An iTIP REQUEST message has been sent to an attendee. If there is no existing resource, we will simply
+        create a new one. If there is an existing resource we need to reconcile the changes between it and the
+        iTIP message.
+
</ins><span class="cx">         @return: C{tuple} of (processed, auto-processed, store inbox item, changes)
</span><span class="cx">         &quot;&quot;&quot;
</span><span class="cx"> 
</span><span class="lines">@@ -508,11 +527,18 @@
</span><span class="cx"> 
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doImplicitAttendeeCancel(self):
</span><ins>+        &quot;&quot;&quot;
+        An iTIP CANCEL message has been sent to an attendee. If there is no existing resource, we will simply
+        ignore the message. If there is an existing resource we need to reconcile the changes between it and the
+        iTIP message.
</ins><span class="cx"> 
</span><ins>+        @return: C{tuple} of (processed, auto-processed, store inbox item, changes)
+        &quot;&quot;&quot;
+
</ins><span class="cx">         # If there is no existing copy, then ignore
</span><span class="cx">         if self.recipient_calendar is None:
</span><span class="cx">             log.debug(&quot;ImplicitProcessing - originator '%s' to recipient '%s' ignoring METHOD:CANCEL, UID: '%s' - attendee has no copy&quot; % (self.originator.cuaddr, self.recipient.cuaddr, self.uid))
</span><del>-            result = (True, True, None)
</del><ins>+            result = (True, True, True, None)
</ins><span class="cx">         else:
</span><span class="cx">             # Need to check for auto-respond attendees. These need to suppress the inbox message
</span><span class="cx">             # if the cancel is processed. However, if the principal is a user we always force the
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingtesttest_icaldiffpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_icaldiff.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_icaldiff.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_icaldiff.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -4292,6 +4292,576 @@
</span><span class="cx">             self.assertEqual(got_rids, rids, msg=&quot;%s expected R-IDs: '%s', got: '%s'&quot; % (description, rids, got_rids,))
</span><span class="cx"> 
</span><span class="cx"> 
</span><ins>+    def test_attendee_needs_action(self):
+
+        data = (
+            (
+                &quot;#1.1 Simple component, no change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#1.2 Simple component, one property change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test1
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#1.3 Simple component, date property change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T130000Z
+DTEND:20080601T140000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set((None,)),
+                False,
+            ),
+            (
+                &quot;#1.4 Simple component, duration property change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DURATION:PT1H
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DURATION:PT2H
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set((None,)),
+                False,
+            ),
+            (
+                &quot;#2.1 Recurring component, no change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#2.2 Recurring component, change property&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test1
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#2.3 Recurring component, change date&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T130000Z
+DTEND:20080601T140000Z
+SUMMARY:Test1
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set((None,)),
+                True,
+            ),
+            (
+                &quot;#2.4 Recurring component, change rule&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=WEEKLY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                True,
+            ),
+            (
+                &quot;#2.5 Recurring component, truncate rule&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY;COUNT=10
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#2.6 Recurring component, expand rule&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY;COUNT=5
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY;COUNT=10
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#3.1 Recurring component with override, no change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T120000Z
+DTEND:20080602T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T120000Z
+DTEND:20080602T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#3.2 Recurring component with override, property change&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T120000Z
+DTEND:20080602T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test1
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T120000Z
+DTEND:20080602T130000Z
+SUMMARY:Test2
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set(),
+                False,
+            ),
+            (
+                &quot;#3.3 Recurring component with override, date change in override&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T120000Z
+DTEND:20080602T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test1
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T130000Z
+DTEND:20080602T140000Z
+SUMMARY:Test2
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set((&quot;20080602T120000Z&quot;,)),
+                False,
+            ),
+            (
+                &quot;#3.4 Recurring component with override, date change in override and master&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T120000Z
+DTEND:20080601T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T120000Z
+DTSTART:20080602T120000Z
+DTEND:20080602T130000Z
+SUMMARY:Test
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890
+DTSTART:20080601T130000Z
+DTEND:20080601T140000Z
+SUMMARY:Test1
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+RRULE:FREQ=DAILY
+END:VEVENT
+BEGIN:VEVENT
+UID:12345-67890
+RECURRENCE-ID:20080602T130000Z
+DTSTART:20080602T130000Z
+DTEND:20080602T140000Z
+SUMMARY:Test2
+ORGANIZER;CN=&quot;User 01&quot;:mailto:user1@example.com
+ATTENDEE:mailto:user1@example.com
+ATTENDEE:mailto:user2@example.com
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                set((None,)),
+                True,
+            ),
+        )
+
+        for description, calendar1, calendar2, rids, rescheduled in data:
+            differ = iCalDiff(Component.fromString(calendar1), Component.fromString(calendar2), False)
+            diffs = differ.whatIsDifferent()
+            got_rids, got_rescheduled = differ.attendeeNeedsAction(diffs)
+            rids = set([DateTime.parseText(k) if k else None for k in rids])
+            self.assertEqual(got_rids, rids, msg=&quot;%s expected R-IDs: '%s', got: '%s'&quot; % (description, rids, got_rids,))
+            self.assertEqual(got_rescheduled, rescheduled, msg=&quot;%s expected rescheduled: '%s', got: '%s'&quot; % (description, rescheduled, got_rescheduled,))
+
+
</ins><span class="cx">     def test_organizer_smart_merge(self):
</span><span class="cx"> 
</span><span class="cx">         data1 = (
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingtesttest_implicitpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_implicit.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_implicit.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_implicit.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -1467,7 +1467,7 @@
</span><span class="cx"> 
</span><span class="cx">         # Need refreshes to occur immediately, not via reactor.callLater
</span><span class="cx">         self.patch(config.Scheduling.Options, &quot;AttendeeRefreshBatch&quot;, 5)
</span><del>-        self.patch(config.Scheduling.Options, &quot;AttendeeRefreshBatchDelaySeconds&quot;, 1)
</del><ins>+        self.patch(config.Scheduling.Options.WorkQueues, &quot;AttendeeRefreshBatchDelaySeconds&quot;, 1)
</ins><span class="cx"> 
</span><span class="cx">         yield self._createCalendarObject(data1, &quot;user01&quot;, &quot;test.ics&quot;)
</span><span class="cx"> 
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingtesttest_itippy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_itip.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_itip.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/test/test_itip.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -34,6 +34,224 @@
</span><span class="cx">     iCalendar support tests
</span><span class="cx">     &quot;&quot;&quot;
</span><span class="cx"> 
</span><ins>+    def test_processRequest_mergeAttendeePartstat(self):
+        &quot;&quot;&quot;
+        Test iTIPProcessing.processRequest properly preserves attendee PARTSTAT when there is no date change
+        &quot;&quot;&quot;
+
+        data = (
+            (
+                &quot;1.1 Simple Request - summary change only, partstats match&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+TRANSP:TRANSPARENT
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+METHOD:REQUEST
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test1
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test1
+TRANSP:TRANSPARENT
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+            ),
+            (
+                &quot;1.2 Simple Request - summary change only, partstat mismatch&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=ACCEPTED:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+METHOD:REQUEST
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test1
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=ACCEPTED:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test1
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+            ),
+            (
+                &quot;1.3 Simple Request - date change, partstats match&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+TRANSP:TRANSPARENT
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+METHOD:REQUEST
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+TRANSP:TRANSPARENT
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+            ),
+            (
+                &quot;1.4 Simple Request - date change, partstat mismatch&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071114T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=ACCEPTED:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+METHOD:REQUEST
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071115T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+                &quot;&quot;&quot;BEGIN:VCALENDAR
+VERSION:2.0
+PRODID:-//CALENDARSERVER.ORG//NONSGML Version 1//EN
+BEGIN:VEVENT
+UID:12345-67890-1
+DTSTART:20071115T000000Z
+DURATION:PT1H
+DTSTAMP:20071114T000000Z
+ATTENDEE:mailto:user01@example.com
+ATTENDEE;PARTSTAT=NEEDS-ACTION:mailto:user02@example.com
+ORGANIZER:mailto:user01@example.com
+SUMMARY:Test
+TRANSP:TRANSPARENT
+END:VEVENT
+END:VCALENDAR
+&quot;&quot;&quot;,
+            ),
+        )
+
+        for title, calendar_txt, itip_txt, changed_txt in data:
+            calendar = Component.fromString(calendar_txt)
+            itip = Component.fromString(itip_txt)
+            changed = Component.fromString(changed_txt)
+
+            result, _ignore = iTipProcessing.processRequest(itip, calendar, &quot;mailto:user02@example.com&quot;)
+            self.assertEqual(result, changed, msg=&quot;Calendar mismatch: %s&quot; % (title,))
+
+
</ins><span class="cx">     def test_processReply(self):
</span><span class="cx">         &quot;&quot;&quot;
</span><span class="cx">         Test iTIPProcessing.processReply
</span><span class="lines">@@ -1077,11 +1295,6 @@
</span><span class="cx">             calendar = Component.fromString(calendar_txt)
</span><span class="cx">             itipmsg = Component.fromString(itipmsg_txt)
</span><span class="cx">             reply_success, reply_processed = iTipProcessing.processReply(itipmsg, calendar)
</span><del>-#            if not description.startswith(&quot;#3.1&quot;):
-#                continue
-#            print(description)
-#            print(str(calendar))
-#            print(str(result))
</del><span class="cx">             self.assertEqual(
</span><span class="cx">                 str(calendar).replace(&quot;\r&quot;, &quot;&quot;).replace(&quot;\n &quot;, &quot;&quot;),
</span><span class="cx">                 str(result).replace(&quot;\n &quot;, &quot;&quot;),
</span></span></pre></div>
<a id="CalendarServerbranchesuserscdabooschedulingqueuerefreshtxdavcaldavdatastoreschedulingworkpy"></a>
<div class="modfile"><h4>Modified: CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/work.py (12458 => 12459)</h4>
<pre class="diff"><span>
<span class="info">--- CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/work.py        2014-01-28 00:07:43 UTC (rev 12458)
+++ CalendarServer/branches/users/cdaboo/scheduling-queue-refresh/txdav/caldav/datastore/scheduling/work.py        2014-01-28 01:25:30 UTC (rev 12459)
</span><span class="lines">@@ -20,7 +20,7 @@
</span><span class="cx"> from twext.enterprise.queue import WorkItem
</span><span class="cx"> from twext.python.log import Logger
</span><span class="cx"> 
</span><del>-from twisted.internet.defer import inlineCallbacks, returnValue
</del><ins>+from twisted.internet.defer import inlineCallbacks, returnValue, Deferred
</ins><span class="cx"> 
</span><span class="cx"> from twistedcaldav import caldavxml
</span><span class="cx"> from twistedcaldav.config import config
</span><span class="lines">@@ -102,6 +102,8 @@
</span><span class="cx">     their calendar object resource.
</span><span class="cx">     &quot;&quot;&quot;
</span><span class="cx"> 
</span><ins>+    _allDoneCallback = None
+
</ins><span class="cx">     @classmethod
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def schedule(cls, txn, uid, action, home, resource, calendar, organizer, smart_merge):
</span><span class="lines">@@ -116,7 +118,7 @@
</span><span class="cx">         Note that for (3), when work executes the resource will have been removed.
</span><span class="cx">         &quot;&quot;&quot;
</span><span class="cx">         # Always queue up new work - coalescing happens when work is executed
</span><del>-        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.QueuedRequestDelaySeconds)
</del><ins>+        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.WorkQueues.RequestDelaySeconds)
</ins><span class="cx">         proposal = (yield txn.enqueue(
</span><span class="cx">             cls,
</span><span class="cx">             notBefore=notBefore,
</span><span class="lines">@@ -142,6 +144,13 @@
</span><span class="cx">         returnValue(len(rows) &gt; 0)
</span><span class="cx"> 
</span><span class="cx"> 
</span><ins>+    @classmethod
+    def allDone(cls):
+        d = Deferred()
+        cls._allDoneCallback = d.callback
+        return d
+
+
</ins><span class="cx">     @inlineCallbacks
</span><span class="cx">     def doWork(self):
</span><span class="cx"> 
</span><span class="lines">@@ -161,6 +170,9 @@
</span><span class="cx">             scheduler = ImplicitScheduler()
</span><span class="cx">             yield scheduler.queuedOrganizerProcessing(self.transaction, scheduleActionFromSQL[self.scheduleAction], home, resource, self.icalendarUid, calendar, self.smartMerge)
</span><span class="cx"> 
</span><ins>+            if self._allDoneCallback:
+                self._allDoneCallback(None)
+
</ins><span class="cx">         except Exception, e:
</span><span class="cx">             log.debug(&quot;ScheduleOrganizerWork - exception ID: {id}, UID: '{uid}', {err}&quot;, id=self.workID, uid=self.icalendarUid, err=str(e))
</span><span class="cx">             raise
</span><span class="lines">@@ -210,7 +222,7 @@
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def reply(cls, txn, home, resource, changedRids, attendee):
</span><span class="cx">         # Always queue up new work - coalescing happens when work is executed
</span><del>-        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.QueuedReplyDelaySeconds)
</del><ins>+        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.WorkQueues.ReplyDelaySeconds)
</ins><span class="cx">         proposal = (yield txn.enqueue(
</span><span class="cx">             cls,
</span><span class="cx">             notBefore=notBefore,
</span><span class="lines">@@ -286,7 +298,7 @@
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def replyCancel(cls, txn, home, calendar, attendee):
</span><span class="cx">         # Always queue up new work - coalescing happens when work is executed
</span><del>-        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.QueuedReplyDelaySeconds)
</del><ins>+        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.WorkQueues.ReplyDelaySeconds)
</ins><span class="cx">         proposal = (yield txn.enqueue(
</span><span class="cx">             cls,
</span><span class="cx">             notBefore=notBefore,
</span><span class="lines">@@ -348,7 +360,11 @@
</span><span class="cx">     not unique wrt to attendees - this means that two simultaneous refreshes can happily insert the
</span><span class="cx">     same set of attendees without running into unique constraints and thus without having to use
</span><span class="cx">     savepoints to cope with that. This will mean duplicate attendees listed in the table, but we take
</span><del>-    care of that when executing the work item, as per the next point.
</del><ins>+    care of that when executing the work item, as per the next point. We also always schedule a new work
+    item for the refresh - even if others are present. The work items are coalesced when executed, with
+    the actual refresh only running at the time of the latest enqueued item. That ensures there is always
+    a pause between a change that causes a refresh and then next actual refresh batch being done, giving
+    some breathing space in case rapid changes are happening to the iCalendar data.
</ins><span class="cx"> 
</span><span class="cx">     2) When a work item is triggered we get the set of unique attendees needing a refresh from the
</span><span class="cx">     SCHEDULE_REFRESH_ATTENDEES table. We split out a batch of those to actually refresh - with the
</span><span class="lines">@@ -385,7 +401,7 @@
</span><span class="cx">             ).on(txn)
</span><span class="cx"> 
</span><span class="cx">         # Always queue up new work - coalescing happens when work is executed
</span><del>-        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.AttendeeRefreshBatchDelaySeconds)
</del><ins>+        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.WorkQueues.AttendeeRefreshBatchDelaySeconds)
</ins><span class="cx">         proposal = (yield txn.enqueue(
</span><span class="cx">             cls,
</span><span class="cx">             icalendarUid=organizer_resource.uid(),
</span><span class="lines">@@ -440,7 +456,7 @@
</span><span class="cx"> 
</span><span class="cx">         # Reschedule work item if pending attendees remain.
</span><span class="cx">         if len(pendingAttendees) != 0:
</span><del>-            notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.AttendeeRefreshBatchIntervalSeconds)
</del><ins>+            notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.WorkQueues.AttendeeRefreshBatchIntervalSeconds)
</ins><span class="cx">             yield self.transaction.enqueue(
</span><span class="cx">                 self.__class__,
</span><span class="cx">                 homeResourceID=self.homeResourceID,
</span><span class="lines">@@ -517,7 +533,7 @@
</span><span class="cx">     @inlineCallbacks
</span><span class="cx">     def autoReply(cls, txn, resource, partstat):
</span><span class="cx">         # Always queue up new work - coalescing happens when work is executed
</span><del>-        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.AutoReplyDelaySeconds)
</del><ins>+        notBefore = datetime.datetime.utcnow() + datetime.timedelta(seconds=config.Scheduling.Options.WorkQueues.AutoReplyDelaySeconds)
</ins><span class="cx">         proposal = (yield txn.enqueue(
</span><span class="cx">             cls,
</span><span class="cx">             icalendarUid=resource.uid(),
</span></span></pre>
</div>
</div>

</body>
</html>