<div>Hi,<br /></div><div><br /></div><div>I am trying to upgrade from calendar server version 4.2 to version 7.0 by starting version 7.0 of the server connected to a database originating from version 4.2. The upgrade process seems to start correctly, but after a while the process is aborted. Some properties in the resource_property table contains UTF-8 characters and that seems to abort the upgrade process and make the server shut down. The following lines are an example from the error log when the problem happens:<br /></div><div><br /></div><div>2015-12-14 13:14:03+0000 [-] [calendarserver.tap.util#error] Step failure<br /></div><div>        Traceback (most recent call last):<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 442, in errback<br /></div><div>            self._startRunCallbacks(fail)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 509, in _startRunCallbacks<br /></div><div>            self._runCallbacks()<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 596, in _runCallbacks<br /></div><div>            current.result = callback(current.result, *args, **kw)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1192, in gotResult<br /></div><div>            _inlineCallbacks(r, g, deferred)<br /></div><div>        --- &lt;exception caught here&gt; ---<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrade.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrade.py</a>", line 147, in databaseUpgra<br /></div><div>de<br /></div><div>            yield self.upgradeVersion(actual_version, required_version, dialect)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrade.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrade.py</a>", line 212, in upgradeVersio<br /></div><div>n<br /></div><div>            yield self.applyUpgrade(fp)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrade.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrade.py</a>", line 361, in applyUpgrade<br /></div><div>            yield doUpgrade(self.sqlStore)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py</a>", line 43, in doUpgrade<br /></div><div>            yield updateCalendarHomes(sqlStore, config.UpgradeHomePrefix)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py</a>", line 60, in updateCalendarHomes<br /></div><div>            yield doToEachHomeNotAtVersion(sqlStore, schema.CALENDAR_HOME, UPGRADE_TO_VERSION, updateCalendarHome, "Update Calendar Home", filterOwnerUID=prefix)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/util.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/util.py</a>", line 179, in doToEachHomeNotAtVersion<br /></div><div>            yield doIt(txn, homeResourceID)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1134, in _inlineCallbacks<br /></div><div>            result = result.throwExceptionIntoGenerator(g)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator<br /></div><div>            return g.throw(self.type, self.value, self.tb)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py</a>", line 71, in updateCalendarHome<br /></div><div>            yield moveCalendarTimezoneProperties(home)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/twisted/internet/defer.py", line 1136, in _inlineCallbacks<br /></div><div>            result = g.send(result)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py">-py2.7.egg/txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py</a>", line 90, in moveCalendarTimezoneProperties<br /></div><div>            prop = calendar.properties().get(PropertyName.fromElement(caldavxml.CalendarTimeZone))<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/UserDict.py", line 168, in get<br /></div><div>            return self[key]<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/base/propertystore/base.py">-py2.7.egg/txdav/base/propertystore/base.py</a>", line 227, in __getitem__<br /></div><div>            return self._getitem_uid(key, self._perUser)<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/base/propertystore/sql.py">-py2.7.egg/txdav/base/propertystore/sql.py</a>", line 275, in _getitem_uid<br /></div><div>            return WebDAVDocument.fromString(value).root_element<br /></div><div>          File "/usr/share/python/calendarserver/lib/python2.7/site-packages/CalendarServer-7.0.a1_unknown<a href="http://-py2.7.egg/txdav/xml/parser_base.py">-py2.7.egg/txdav/xml/parser_base.py</a>", line 43, in fromString<br /></div><div>            source = StringIO(source)<br /></div><div>        exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u2212' in position 348: ordinal not in range(128)<br /></div><div><br /></div><div><br /></div><div>The properties that I have found to be related to this problem are the following:<br /></div><div><br /></div><div>CALDAV:calendar-timezone<br /></div><div>CS:calendar-availability<br /></div><div><br /></div><div>They are both processed <span class="colour" style="color: rgb(0, 0, 0);"><span class="font" style="font-family: helvetica, arial, sans-serif;"><span class="size" style="font-size: 13px;">in txdav/common/datastore/upgrade/sql/upgrades/calendar_upgrade_from_4_to_5.py.</span></span></span></div><div><br /></div><div>The values of these properties seems to be valid xml data with the encoding attribute set to 'UTF-8', i.e.:<br /></div><div><br /></div><div>&lt;?xml version='1.0' encoding='UTF-8'?&gt;                                             <br /></div><div>&lt;calendar-timezone xmlns='urn:ietf:params:xml:ns:caldav'&gt;&lt;![CDATA[BEGIN:VCALENDAR<br /></div><div>VERSION:2.0<br /></div><div>PRODID:-//Apple Inc.//Mac OS X 10.11//EN<br /></div><div>CALSCALE:GREGORIAN<br /></div><div>...<br /></div><div>...</div><div><br /></div><div>Best regards,<br /></div><div>Kristoffer Møllerhøj.<br /></div>