[CalendarServer-users] Connection aborted - took too long to close
Andre LaBranche
dre at apple.com
Thu Feb 5 10:19:51 PST 2015
> On Feb 5, 2015, at 8:51 AM, Jacques Distler <distler at golem.ph.utexas.edu> wrote:
>
>> There is one known issue in 5.3 regarding how we handle updating calendar event data when the format changes (as it has recently). In 5.3, the approach is to leave the data in the old format until it is requested, and then update it on the fly as it is vended to the client. This fails when there are enough events being 'touched' by the request that the processing time for doing the format upgrades exceeds the client's request timeout threshold. This is fixed in 5.4-dev.
>
> Is there (say) a commandline utility to upgrade the data to the new format?
We don’t have a single command line tool to do this, no...
> Or some other way to address these (rather persistent) timeouts?
You can find out how many resources are in the old format by running this SQL:
select count(*) from calendar_object where dataversion = 0;
If the timeouts are being caused by the data format upgrade taking too long (because it’s operating in batches that are too large), one way to work around it is to manually request all the calendar resources one by one, causing each one to be upgraded individually. I had to work around this problem on a live server without the benefit of any code fixes, and here’s what I ended up doing.
There are two scripts: make-curl-command-file.sh and old_resources.sql; the contents of each are shown below, along with output from a run of make-curl-command-file.sh. To use this, you need to specify a principal in ReadPrincipals in caldavd.plist - preferably an admin user - to authorize that user to access all the resources. ReadPrincipals is an array, with string values specifying principal URLs, like the ones you see on a user’s principal page (at /principals/users/foo). Use the credentials for this user with the scripts below by creating a netrc file for curl. The script gives an example if netrc is missing. You may also need to customize SOCKETPATH, or perhaps switch to tcp/ip - depending on how your postgres service is deployed.
(and note that although this script worked for me, it comes with no warranty)
# begin make-curl-command-file.sh
#!/bin/bash
# * execute some SQL that emits URLs of all calendar events with an old data
# format.
# * put the results into a new file, trimming the leading and trailing lines
# * emit the curl command required to process this file
# * run that command
SOCKETPATH="/var/run/caldavd/PostgresSocket"
PGUSER="caldav"
SQLFILE="old_resources.sql"
CURLFILE="curl-commands.txt"
NETRC="netrc.txt"
rm ${CURLFILE}
touch ${CURLFILE}
echo "Getting total number of un-upgraded items: "
sudo psql -h ${SOCKETPATH} -U ${PGUSER} -c "select count(*) from calendar_object where dataversion = 0;"
echo ""
echo "Gathering list of events that need to be upgraded in 1000 item chunks..."
sudo psql -q -h ${SOCKETPATH} -U ${PGUSER} -f ${SQLFILE} | grep "url =" > ${CURLFILE}
echo ""
ITEMS=$(wc -l ${CURLFILE} | awk '{print $1}')
if [ ${ITEMS} -eq 0 ]
then echo "No more work to do!"
exit 0
fi
echo -n "Generated curl command file with this many items: ${ITEMS}"
echo ""
if [ ! -f ${NETRC} ]
then
echo "Put the credentials of a ReadAdmin user into ${NETRC}, e.g. :"
echo "machine example.com <http://solar.burningman.com/> login Administrator password 12345"
echo ""
fi
echo ""
echo "now running:"
echo "curl --netrc-file ${NETRC} -K ${CURLFILE} > /dev/null"
echo ""
curl --netrc-file ${NETRC} -K ${CURLFILE} > /dev/null
echo "Re-run this script until it has no more work to do.”
# end make-curl-command-file.sh
# begin old_resources.sql
SELECT 'url = "https://example.com:8443/calendars/__uids__/' <https://example.com:8443/calendars/__uids__/'>
|| owner_uid
|| '/'
|| calendar_resource_name
|| '/'
|| resource_name
|| '"' as "curl command file contents"
FROM calendar_bind
JOIN calendar_home
ON calendar_bind.calendar_home_resource_id = calendar_home.resource_id
JOIN calendar_object
ON calendar_bind.calendar_resource_id =
calendar_object.calendar_resource_id
WHERE calendar_bind.bind_mode = 0
AND calendar_object.dataversion = 0
LIMIT 1000;
# end old_resources.sql
server:bin administrator$ ./make-curl-command-file.sh
Total number of un-upgraded items: count
-------
48842
(1 row)
Gathering list of events that need to be upgraded in 1000 item chunks...
Generated curl command file with this many items: 1000
now running:
curl --netrc-file netrc.txt -K curl-commands.txt > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1514 100 1514 0 0 7656 0 --:--:-- --:--:-- --:--:-- 7685
100 803 100 803 0 0 5378 0 --:--:-- --:--:-- --:--:-- 784k
100 1140 100 1140 0 0 7580 0 --:--:-- --:--:-- --:--:-- 7580
100 707 100 707 0 0 4813 0 --:--:-- --:--:-- --:--:-- 4813
100 733 100 733 0 0 4982 0 --:--:-- --:--:-- --:--:-- 4982
100 1035 100 1035 0 0 6948 0 --:--:-- --:--:-- --:--:-- 6948
100 691 100 691 0 0 4690 0 --:--:-- --:--:-- --:--:-- 4690
100 1317 100 1317 0 0 8444 0 --:--:-- --:--:-- --:--:-- 1286k
100 748 100 748 0 0 5102 0 --:--:-- --:--:-- --:--:-- 5102
100 368 100 368 0 0 2507 0 --:--:-- --:--:-- --:--:-- 2507
100 725 100 725 0 0 4943 0 --:--:-- --:--:-- --:--:-- 4943
100 717 100 717 0 0 4773 0 --:--:-- --:--:-- --:--:-- 4773
...
> FWIW, I ran calendarserver_verify_data and everything checked out OK.
Good deal.
-dre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.macosforge.org/pipermail/calendarserver-users/attachments/20150205/654dacb3/attachment.html>
More information about the calendarserver-users
mailing list