<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 5, 2015, at 8:51 AM, Jacques Distler <<a href="mailto:distler@golem.ph.utexas.edu" class="">distler@golem.ph.utexas.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" class="">There is one known issue in 5.3 regarding how we handle updating calendar event data when the format changes (as it has recently). In 5.3, the approach is to leave the data in the old format until it is requested, and then update it on the fly as it is vended to the client. This fails when there are enough events being 'touched' by the request that the processing time for doing the format upgrades exceeds the client's request timeout threshold. This is fixed in 5.4-dev.<br class=""></blockquote><br class="">Is there (say) a commandline utility to upgrade the data to the new format?<br class=""></div></blockquote><div><br class=""></div><div>We don’t have a single command line tool to do this, no...</div><br class=""><blockquote type="cite" class=""><div class="">Or some other way to address these (rather persistent) timeouts?<br class=""></div></blockquote><div><br class=""></div><div>You can find out how many resources are in the old format by running this SQL:</div><div>select count(*) from calendar_object where dataversion = 0;</div><div><br class=""></div><div>If the timeouts are being caused by the data format upgrade taking too long (because it’s operating in batches that are too large), one way to work around it is to manually request all the calendar resources one by one, causing each one to be upgraded individually. I had to work around this problem on a live server without the benefit of any code fixes, and here’s what I ended up doing.</div><div><br class=""></div><div>There are two scripts: make-curl-command-file.sh and old_resources.sql; the contents of each are shown below, along with output from a run of make-curl-command-file.sh. To use this, you need to specify a principal in <span style="font-family: monospace; font-size: 11px; background-color: rgb(255, 255, 255);" class="">ReadPrincipals</span> in caldavd.plist - preferably an admin user - to authorize that user to access all the resources. ReadPrincipals is an array, with string values specifying principal URLs, like the ones you see on a user’s principal page (at /principals/users/foo). Use the credentials for this user with the scripts below by creating a netrc file for curl. The script gives an example if netrc is missing. You may also need to customize SOCKETPATH, or perhaps switch to tcp/ip - depending on how your postgres service is deployed.</div><div><br class=""></div><div><br class=""></div><div>(and note that although this script worked for me, it comes with no warranty)</div><div><br class=""><br class=""># begin make-curl-command-file.sh<br class="">#!/bin/bash<br class=""><br class=""># * execute some SQL that emits URLs of all calendar events with an old data<br class=""># format.<br class=""># * put the results into a new file, trimming the leading and trailing lines<br class=""># * emit the curl command required to process this file<br class=""># * run that command<br class=""><br class="">SOCKETPATH="/var/run/caldavd/PostgresSocket"<br class="">PGUSER="caldav"<br class="">SQLFILE="old_resources.sql"<br class="">CURLFILE="curl-commands.txt"<br class="">NETRC="netrc.txt"<br class=""><br class="">rm ${CURLFILE}<br class="">touch ${CURLFILE}<br class=""><br class="">echo "Getting total number of un-upgraded items: "<br class="">sudo psql -h ${SOCKETPATH} -U ${PGUSER} -c "select count(*) from calendar_object where dataversion = 0;"<br class="">echo ""<br class=""><br class="">echo "Gathering list of events that need to be upgraded in 1000 item chunks..."<br class="">sudo psql -q -h ${SOCKETPATH} -U ${PGUSER} -f ${SQLFILE} | grep "url =" > ${CURLFILE}<br class="">echo ""<br class=""><br class="">ITEMS=$(wc -l ${CURLFILE} | awk '{print $1}')<br class=""><br class="">if [ ${ITEMS} -eq 0 ]<br class=""> then echo "No more work to do!"<br class=""> exit 0<br class="">fi<br class=""><br class="">echo -n "Generated curl command file with this many items: ${ITEMS}"<br class="">echo ""<br class=""><br class="">if [ ! -f ${NETRC} ]<br class=""> then<br class=""> echo "Put the credentials of a ReadAdmin user into ${NETRC}, e.g. :"<br class=""> echo "machine <a href="http://solar.burningman.com" class="">example.com</a> login Administrator password 12345"<br class=""> echo ""<br class="">fi<br class=""><br class="">echo ""<br class="">echo "now running:"<br class="">echo "curl --netrc-file ${NETRC} -K ${CURLFILE} > /dev/null"<br class="">echo ""<br class="">curl --netrc-file ${NETRC} -K ${CURLFILE} > /dev/null<br class="">echo "Re-run this script until it has no more work to do.”</div><div># end make-curl-command-file.sh<br class=""><br class=""><br class=""><br class=""># begin old_resources.sql<br class="">SELECT 'url = "<a href="https://example.com:8443/calendars/__uids__/'" class="">https://example.com:8443/calendars/__uids__/'</a> <br class=""> || owner_uid <br class=""> || '/' <br class=""> || calendar_resource_name <br class=""> || '/' <br class=""> || resource_name<br class=""> || '"' as "curl command file contents"<br class="">FROM calendar_bind <br class=""> JOIN calendar_home <br class=""> ON calendar_bind.calendar_home_resource_id = calendar_home.resource_id <br class=""> JOIN calendar_object <br class=""> ON calendar_bind.calendar_resource_id = <br class=""> calendar_object.calendar_resource_id <br class="">WHERE calendar_bind.bind_mode = 0 <br class=""> AND calendar_object.dataversion = 0<br class=""> LIMIT 1000;</div><div># end old_resources.sql</div><div><br class=""></div><div><br class=""></div><div>server:bin administrator$ ./make-curl-command-file.sh <br class="">Total number of un-upgraded items: count <br class="">-------<br class="">48842<br class="">(1 row)<br class=""><br class=""><br class="">Gathering list of events that need to be upgraded in 1000 item chunks...<br class=""><br class="">Generated curl command file with this many items: 1000<br class=""><br class="">now running:<br class="">curl --netrc-file netrc.txt -K curl-commands.txt > /dev/null<br class=""><br class=""> % Total % Received % Xferd Average Speed Time Time Time Current<br class=""> Dload Upload Total Spent Left Speed<br class="">100 1514 100 1514 0 0 7656 0 --:--:-- --:--:-- --:--:-- 7685<br class="">100 803 100 803 0 0 5378 0 --:--:-- --:--:-- --:--:-- 784k<br class="">100 1140 100 1140 0 0 7580 0 --:--:-- --:--:-- --:--:-- 7580<br class="">100 707 100 707 0 0 4813 0 --:--:-- --:--:-- --:--:-- 4813<br class="">100 733 100 733 0 0 4982 0 --:--:-- --:--:-- --:--:-- 4982<br class="">100 1035 100 1035 0 0 6948 0 --:--:-- --:--:-- --:--:-- 6948<br class="">100 691 100 691 0 0 4690 0 --:--:-- --:--:-- --:--:-- 4690<br class="">100 1317 100 1317 0 0 8444 0 --:--:-- --:--:-- --:--:-- 1286k<br class="">100 748 100 748 0 0 5102 0 --:--:-- --:--:-- --:--:-- 5102<br class="">100 368 100 368 0 0 2507 0 --:--:-- --:--:-- --:--:-- 2507<br class="">100 725 100 725 0 0 4943 0 --:--:-- --:--:-- --:--:-- 4943<br class="">100 717 100 717 0 0 4773 0 --:--:-- --:--:-- --:--:-- 4773<br class="">...<br class=""><br class=""></div><div><br class=""></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class="">FWIW, I ran calendarserver_verify_data and everything checked out OK.<br class=""></div></blockquote><div><br class=""></div><div>Good deal.</div><div><br class=""></div><div><br class=""></div><div>-dre</div></div></body></html>