Unable to have a stable setup for CalendarServer 4.2
Hi, I’m Yoann. It’s my first message here so a little brief about me. I’m a OS X Server System Administrator and Trainer, actually working on a FreeBSD based setup for a simple services provider infrastructure. I plan to use the calenderserver project for CalDAV and CardDAV instead of the easy to use SOGo to run a server more close as possible to the RFCs. I successfully setup the 4.2 release with a LDAP and a PostgreSQL backend on a FreeBSD 9.1 64 bits processor. It was a pain because the 4.2 didn’t handle well the 64 bits socket descriptors but I’ve add some fixes for that. That was the same with de dependencies, a lot of things seems to be unstable on 64bits, so I install the requirements by my own and run it with a -nr to don’t use built-in deps. By the way, the behavior to install all deps in .. fold is really unusual and put a big mess… But what ever. It seems to look like a working config at the end. I can run the server, authenticate my self, if I enable the principals navigation I can see all my users, I can connect iCal and use it. Well, it’s work. But, after some times, the server seems to by buggy, I start to make time outs for request, or for opening sockets, or sometime it rework again… Really random behavior and not at lot of interesting things in logs… To have a idea, all my services, caldavd included are under nagios monitoring, I get something like 20 alerts per day about CalendarServer in a critical state… Anyone can give me a clue about what to look ? What can be the source of the problem ? How to troubleshooting it ? Best regards, Yoann.
Hi Yoann, Am 15.03.2013 um 18:49 schrieb Yoann Gini:
I’m Yoann. It’s my first message here so a little brief about me. I’m a OS X Server System Administrator and Trainer, actually working on a FreeBSD based setup for a simple services provider infrastructure.
I plan to use the calenderserver project for CalDAV and CardDAV instead of the easy to use SOGo to run a server more close as possible to the RFCs. You are not allone: I try to build a FreeBSD port for calendarserver, based on the 4.2 release.
I successfully setup the 4.2 release with a LDAP and a PostgreSQL backend on a FreeBSD 9.1 64 bits processor. I have (some how) working installations on 8.2 and 9.1, both 64 bit systems, using accounts.xml instead of LDAP.
It was a pain because the 4.2 didn’t handle well the 64 bits socket descriptors but I’ve add some fixes for that. I hit the same bug. It has been discussed here: http://www.mail-archive.com/calendarserver-dev@lists.macosforge.org/msg01611... That was the same with de dependencies, a lot of things seems to be unstable on 64bits, so I install the requirements by my own and run it with a -nr to don’t use built-in deps. My port does not use the run script at all, installing dependencies through the ports system.
By the way, the behavior to install all deps in .. fold is really unusual and put a big mess…
But what ever. It seems to look like a working config at the end.
I can run the server, authenticate my self, if I enable the principals navigation I can see all my users, I can connect iCal and use it. Well, it’s work. I just reached a similar point, where I start testing, using calendar and addressbook clients on OSX 10.6.8 and 10.8.3, but went soon in a problem, where the master process starts eating cpu cycles, as described here: http://www.mail-archive.com/calendarserver-dev@lists.macosforge.org/msg01618...
But, after some times, the server seems to by buggy, I start to make time outs for request, or for opening sockets, or sometime it rework again… Really random behavior and not at lot of interesting things in logs…
To have a idea, all my services, caldavd included are under nagios monitoring, I get something like 20 alerts per day about CalendarServer in a critical state…
Anyone can give me a clue about what to look ? What can be the source of the problem ? How to troubleshooting it ? I have no idea yet, but maybe you look through the above thread. Currently I'm trying to learn how to profile python processes...
Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
On Mar 15, 2013, at 11:37 AM, Axel Rau <Axel.Rau@Chaos1.DE> wrote:
Currently I'm trying to learn how to profile python processes...
Try runsnake. Most of the hard work is already done! All we need to do is create a directory to hold our stats, run the server with an additional cli argument, perform the operations we wish to profile, then stop the server. Then feed the pstats file to runsnake • mkdir foostats ; ./run -S foostats • Do stuff • Stop server (must be a graceful exit!) • Look in foostats for the goods. Using Run Snake Run: First, install it: easy_install runsnakerun Next, attempt to start it: runsnake You may be missing 'wx', if so, get it. Next, just feed one of your pstats files to Run Snake Run, then bear witness to the glory of many nested and brightly colored rounded rects, in addition to the list of functions encountered during profiling, sortable by a variety of criteria.
Am 15.03.2013 um 19:46 schrieb Andre LaBranche:
On Mar 15, 2013, at 11:37 AM, Axel Rau <Axel.Rau@Chaos1.DE> wrote:
Currently I'm trying to learn how to profile python processes...
Try runsnake.
Most of the hard work is already done! Thanks.
All we need to do is create a directory to hold our stats, run the server with an additional cli argument, perform the operations we wish to profile, then stop the server. Then feed the pstats file to runsnake
• mkdir foostats ; ./run -S foostats • Do stuff • Stop server (must be a graceful exit!) • Look in foostats for the goods. As my port does not install the run script, I used: /usr/local/bin/caldavd -f /usr/local/etc/caldavd/caldavd.plist -R kqueue -p /root/admin/profile and got these: -rw-r--r-- 1 caldavd wheel 628560 Mar 16 12:20 caldav-0.pstats -rw-r--r-- 1 caldavd wheel 83867 Mar 16 12:20 caldav-1.pstats -rw-r--r-- 1 root wheel 432429 Mar 16 12:20 master.pstats (-:
Using Run Snake Run:
First, install it: easy_install runsnakerun
Next, attempt to start it: runsnake
You may be missing 'wx', if so, get it.
wx is a GUI tool, which requires X11 on FreeBSD, which is not available on the server. )-: Are there any alternatives out there? Any recommendations?
Next, just feed one of your pstats files to Run Snake Run, then bear witness to the glory of many nested and brightly colored rounded rects, in addition to the list of functions encountered during profiling, sortable by a variety of criteria.
Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
On Mar 16, 2013, at 5:56 AM, Axel Rau <Axel.Rau@Chaos1.DE> wrote:
Am 15.03.2013 um 19:46 schrieb Andre LaBranche:
On Mar 15, 2013, at 11:37 AM, Axel Rau <Axel.Rau@Chaos1.DE> wrote:
Currently I'm trying to learn how to profile python processes...
Try runsnake.
Most of the hard work is already done! Thanks.
All we need to do is create a directory to hold our stats, run the server with an additional cli argument, perform the operations we wish to profile, then stop the server. Then feed the pstats file to runsnake
• mkdir foostats ; ./run -S foostats • Do stuff • Stop server (must be a graceful exit!) • Look in foostats for the goods. As my port does not install the run script, I used: /usr/local/bin/caldavd -f /usr/local/etc/caldavd/caldavd.plist -R kqueue -p /root/admin/profile and got these: -rw-r--r-- 1 caldavd wheel 628560 Mar 16 12:20 caldav-0.pstats -rw-r--r-- 1 caldavd wheel 83867 Mar 16 12:20 caldav-1.pstats -rw-r--r-- 1 root wheel 432429 Mar 16 12:20 master.pstats (-:
Using Run Snake Run:
First, install it: easy_install runsnakerun
Next, attempt to start it: runsnake
You may be missing 'wx', if so, get it.
wx is a GUI tool, which requires X11 on FreeBSD, which is not available on the server. )-: Are there any alternatives out there? Any recommendations?
You could copy the stats files to another host and analyze them there. -dre
Next, just feed one of your pstats files to Run Snake Run, then bear witness to the glory of many nested and brightly colored rounded rects, in addition to the list of functions encountered during profiling, sortable by a variety of criteria.
Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
Am 16.03.2013 um 20:12 schrieb Andre LaBranche:
You could copy the stats files to another host and analyze them there. Ahh, my Mac. That's great.
Thanks, Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
Hi Axel, Le 15 mars 2013 à 19:37, Axel Rau <Axel.Rau@Chaos1.DE> a écrit :
Am 15.03.2013 um 18:49 schrieb Yoann Gini:
It was a pain because the 4.2 didn’t handle well the 64 bits socket descriptors but I’ve add some fixes for that. I hit the same bug. It has been discussed here: http://www.mail-archive.com/calendarserver-dev@lists.macosforge.org/msg01611...
Yes, I’ve seen your post during my troubleshooting session. I haven’t use your code but that basically the same without logs. They already have too many of them in this tools, it’s such a mess to use. Even Italians spoke less than CalendarServers!!
That was the same with de dependencies, a lot of things seems to be unstable on 64bits, so I install the requirements by my own and run it with a -nr to don’t use built-in deps. My port does not use the run script at all, installing dependencies through the ports system.
That’s my final working config, the deps installed by the run script didn’t work. Can you share your rc.d config file with us?
I can run the server, authenticate my self, if I enable the principals navigation I can see all my users, I can connect iCal and use it. Well, it’s work. I just reached a similar point, where I start testing, using calendar and addressbook clients on OSX 10.6.8 and 10.8.3, but went soon in a problem, where the master process starts eating cpu cycles, as described here: http://www.mail-archive.com/calendarserver-dev@lists.macosforge.org/msg01618...
Well, same state here… PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 5454 0 3 20 0 270M 56592K usem 0 106:09 3.86% python2.7
Anyone can give me a clue about what to look ? What can be the source of the problem ? How to troubleshooting it ? I have no idea yet, but maybe you look through the above thread. Currently I'm trying to learn how to profile python processes...
That’s unbelievable… On OS X Server, we have CalendarServer and it’s work really well, it can handle a lot of loads and don’t fail like that… I will take some time to make a diff between the public 4.2 release and the built-in Calendar Server in Server for OS X Mountain Lion. I will share my result with you.
participants (4)
-
Andre LaBranche
-
Axel Rau
-
Axel Rau
-
Yoann Gini