Hi all. We are running a mac server 10.7.4 with caldav 3 in a commercial environment. We have the behavior that from time to time the spawned processes from caldavd (Python /usr/bin/twistd -f /etc/caldavd/caldavd.plist) will use memory up to 3GB. because of that our server is running out of memory and the processes will create very big swap files. and our server will getting slower and slower. How we can change the process count? we figured out we can do this via the caldavd.plist file. we want to change the xml tag MultiProcess/ProcessCount. is that right? The important question is how we can restrict the memory usage for every caldav process? What does "ProcessType" mean? There are 3 values: master,slave,combined Thanks a lot for your help. Marko
On Jun 7, 2012, at 2:15 AM, Marko Bauhardt <mb@datameer.com> wrote:
Hi all.
We are running a mac server 10.7.4 with caldav 3 in a commercial environment. We have the behavior that from time to time the spawned processes from caldavd (Python /usr/bin/twistd -f /etc/caldavd/caldavd.plist) will use memory up to 3GB. because of that our server is running out of memory and the processes will create very big swap files. and our server will getting slower and slower.
How we can change the process count? we figured out we can do this via the caldavd.plist file. we want to change the xml tag MultiProcess/ProcessCount. is that right?
Yes.
The important question is how we can restrict the memory usage for every caldav process?
The server has no built-in facility for doing this. You may be able to use operating system facilities (e.g. ulimit) to accomplish this, or alternatively you could use a small monitoring script to kill worker daemons that get too large (they will automatically be respawned by the master process). There are some known cases where memory use does seem to get out of hand... killing bloated daemons is not a very pretty workaround, but it does work. The downside is that any in-flight requests for the daemon getting killed are aborted. The database is very transactional, so this does not cause any data consistency problems in real-world use (hopefully you noticed the declarative nature of this statement ;)
What does "ProcessType" mean? There are 3 values: master,slave,combined
Not quite, and the names are a little confusing. First of all, the terms that are valid in the config file do not entirely overlap with the terms you see in a process listing. The two valid modes in caldavd.plist are "Combined" and "Single". Setting ProcessType to "Combined" tells the service to run in master / slave mode, and setting it to "Single" puts everything in one process - not recommended. See calendarserver/tap/caldav.py for how these settings are used. When you do a process listening, when running in Combined mode, the master will be shown as Combined, and the slaves as Slave. Sidecar processes don't have any ProcessType designation, and only do a very specific kind of work - sidecar processes have some value other than 'caldav' after the -n argument. HTH, -dre
Hi Andre.
The important question is how we can restrict the memory usage for every caldav process?
The server has no built-in facility for doing this. You may be able to use operating system facilities (e.g. ulimit) to accomplish this, or alternatively you could use a small monitoring script to kill worker daemons that get too large (they will automatically be respawned by the master process). There are some known cases where memory use does seem to get out of hand... killing bloated daemons is not a very pretty workaround, but it does work. The downside is that any in-flight requests for the daemon getting killed are aborted. The database is very transactional, so this does not cause any data consistency problems in real-world use (hopefully you noticed the declarative nature of this statement ;)
Many thanks for your comments. Today in the morning we had the same idea to have a script which kill the worker daemons that get too large :) at the moment we do this manually, so this mean we monitor the caldav worker and when some of these process is using too much memory we kill this process. You can imagine: our administrator has no funny nights ;) Anyway, thanks for your help, we will write a script or similar which will kill the daemons automatically when necessary. Marko
On Jun 7, 2012, at 11:37 AM, Marko Bauhardt wrote:
Hi Andre.
The important question is how we can restrict the memory usage for every caldav process?
The server has no built-in facility for doing this. You may be able to use operating system facilities (e.g. ulimit) to accomplish this, or alternatively you could use a small monitoring script to kill worker daemons that get too large (they will automatically be respawned by the master process). There are some known cases where memory use does seem to get out of hand... killing bloated daemons is not a very pretty workaround, but it does work. The downside is that any in-flight requests for the daemon getting killed are aborted. The database is very transactional, so this does not cause any data consistency problems in real-world use (hopefully you noticed the declarative nature of this statement ;)
Many thanks for your comments. Today in the morning we had the same idea to have a script which kill the worker daemons that get too large :) at the moment we do this manually, so this mean we monitor the caldav worker and when some of these process is using too much memory we kill this process. You can imagine: our administrator has no funny nights ;)
Anyway, thanks for your help, we will write a script or similar which will kill the daemons automatically when necessary.
Out of curiosity, what OS is your server, and how much memory is installed? In diagnosing this problem (you're not the first to see it), we've been able to determine that the OS X memory allocator behaves differently from Linux's, and the difference is amplified by our specific pattern of memory usage. It's quite an interesting state of affairs, probably worthy of a nice blog post some day... -dre
Hi Andre
Out of curiosity, what OS is your server, and how much memory is installed?
In diagnosing this problem (you're not the first to see it), we've been able to determine that the OS X memory allocator behaves differently from Linux's, and the difference is amplified by our specific pattern of memory usage. It's quite an interesting state of affairs, probably worthy of a nice blog post some day...
-dre
Thanks for some more comments. We are using a Mac OS X Server Lion 10.7.4 (11E53) running on a Mac Pro (2.8 GHz Quad-Core Intel Xeon with 8GB RAM). we are using the osx server incl caldav in a commercial environment and this issue is very very critical for us. Would be nice if we get any information/hint regarding this problem. So looks like, we have this problem since version 10.7.4. but we are not 100% sure. Let me know if you need more information, debug loggings or whatever you need to find the reason why we are running out of memory. Many Thanks Marko
participants (2)
-
Andre LaBranche
-
Marko Bauhardt