[CalendarServer-dev] memory usage

[Andre LaBranche]

On Jun 7, 2012, at 2:15 AM, Marko Bauhardt <mb at datameer.com> wrote:

> Hi all.
> 
> We are running a mac server 10.7.4 with caldav 3 in a commercial environment. We have the behavior that from time to time the spawned processes from caldavd (Python /usr/bin/twistd -f /etc/caldavd/caldavd.plist) will use memory up to 3GB. because of that our server is running out of memory and the processes will create very big swap files. and our server will getting slower and slower. 
> 
> How we can change the process count? we figured out we can do this via the caldavd.plist file. we want to change the xml tag MultiProcess/ProcessCount. is that right?

Yes.

> The important question is how we can restrict the memory usage for every caldav process?

The server has no built-in facility for doing this. You may be able to use operating system facilities (e.g. ulimit) to accomplish this, or alternatively you could use a small monitoring script to kill worker daemons that get too large (they will automatically be respawned by the master process). There are some known cases where memory use does seem to get out of hand... killing bloated daemons is not a very pretty workaround, but it does work. The downside is that any in-flight requests for the daemon getting killed are aborted. The database is very transactional, so this does not cause any data consistency problems in real-world use (hopefully you noticed the declarative nature of this statement ;)

> What does "ProcessType" mean? There are 3 values: master,slave,combined

Not quite, and the names are a little confusing. First of all, the terms that are valid in the config file do not entirely overlap with the terms you see in a process listing. The two valid modes in caldavd.plist are "Combined" and "Single". Setting ProcessType to "Combined" tells the service to run in master / slave mode, and setting it to "Single" puts everything in one process - not recommended. See calendarserver/tap/caldav.py for how these settings are used.

When you do a process listening, when running in Combined mode, the master will be shown as Combined, and the slaves as Slave. Sidecar processes don't have any ProcessType designation, and only do a very specific kind of work - sidecar processes have some value other than 'caldav' after the -n argument.

HTH,
-dre