DCS Production Deployment
Hi, Thank you very much for CalendarServer I have few questions: * I used CalendarServer-5.2.2 during development phase. * Now I want to use it for some real (users) use cases. * Right now, I do not have any scaling issues. * I run it on Ubuntu EC2 instance. My question is: * Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts? Any tip would be useful for PROD deployments. Even for 1000 users, we would like it not to go down at all. Best Regards, Gaurav Jain
On Feb 18, 2015, at 1:04 PM, Gaurav Jain <monkeyfdude@gmail.com> wrote:
My question is:
* Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts?
Hello, Calendar Server can be (and is, in rare instances) deployed in a high-availability (HA) configuration, which can also be used to scale. Check out the following document, which identifies the important parts of a multi-server configuration. http://trac.calendarserver.org/browser/CalendarServer/trunk/doc/Admin/MultiS... <http://trac.calendarserver.org/browser/CalendarServer/trunk/doc/Admin/MultiServerDeployment.rst> Removing all single points of failure would require implementing HA / redundancy for each of the items listed in that document, which is beyond the scope of our documentation, but should be a familiar process for anybody who has ever implemented HA. XMLDirectoryService is probably the most scalable directory service we support, due to its simplicity. If it meets your needs, I can’t think of a reason to switch away from it. -dre
Hi, One more thing: if you are ready to start supporting real users with real data, make sure you are familiar with our various command line tools for managing things over the long term, such as: calendarserver_purge_* tools calendarserver_upgrade - for upgrading data in-place when a new server version has been installed with newer schema backup and restore of the all the service data. This is not limited to only the stuff in Postgres, but also possibly service configs, file attachments, and directory services data. calendarserver_verify_data for detecting and fixing problems in the data itself, which can happen for a variety of reasons such as bugs or bad clients monitoring and measurement tools to help you evaluate performance and do scalability planning. Some of these are in ‘bin’, some are under ‘contrib’. Eventually I will write some actual documentation for these... -dre
On Feb 18, 2015, at 3:37 PM, Andre LaBranche <dre@apple.com> wrote:
On Feb 18, 2015, at 1:04 PM, Gaurav Jain <monkeyfdude@gmail.com <mailto:monkeyfdude@gmail.com>> wrote:
My question is:
* Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts?
Hello,
Calendar Server can be (and is, in rare instances) deployed in a high-availability (HA) configuration, which can also be used to scale. Check out the following document, which identifies the important parts of a multi-server configuration.
http://trac.calendarserver.org/browser/CalendarServer/trunk/doc/Admin/MultiS... <http://trac.calendarserver.org/browser/CalendarServer/trunk/doc/Admin/MultiServerDeployment.rst>
Removing all single points of failure would require implementing HA / redundancy for each of the items listed in that document, which is beyond the scope of our documentation, but should be a familiar process for anybody who has ever implemented HA.
XMLDirectoryService is probably the most scalable directory service we support, due to its simplicity. If it meets your needs, I can’t think of a reason to switch away from it.
-dre _______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
On Feb 18, 2015, at 3:44 PM, Andre LaBranche <dre@apple.com> wrote:
calendarserver_purge_* tools
Prior to 6.0, targeted purging of old data could only be accomplished using our command line tools. These days, we will automatically purge user data once the user record has been absent from directory services for 7 days. This behavior is configurable in caldavd.plist. -dre
I've been running DCS in production for the past ~year, at the moment my deployment supports ~800 users. It took *a lot* of reading through doc and code to feel even remotely comfortable with storing real data for real people and I still don't feel all that great about it. Please, allow me to ramble for a minute, take from it what you will... 1. IMO the idea of XMLDirectoryService is seriously dated and kinda crazy so I wrote my own postgresql backed DirectoryService with support for memcached. This SQL-backed DirectoryService is much easier to manage (because SQL) and allows me to have other services work off of the same account database. Speaking of memcached, DCS (as far as I can tell) caches very little and could stand to benefit from using it more often. 2. DCS uses the python twisted framework but at the same time it doesn't-- calls to DirectoryService (regardless of the implementation) are blocking 0.o 3. DCS has this weird multi-process, master-slave concept where multiple DCS instances run on the same box-- I just think this is weird and dealing with dead processes has been strange. 4. If a user uploads a contact containing a photo and you're using a database for storage (as you should), DCS will store that photo in your database-- yuck! Even with your 1000 users, each with maybe 100 contact photos, things get kinda bloated. So, I modified DCS to strip contact photos from contacts and store them in Amazon S3, then piece them back together on demand. 5. Someone could DOS my DCS instances via network resource exhaustion very easily and there is very little that I could do to stop it other than launch a crazy number of EC2 instances and burn through money. DCS is not efficient (blocking calls to DirectoryService, etc) and I feel very uncomfortable having any less than than two m3.medium instances running at all times (costly!). All that being said, DCS is still the best free and open source option out there but I wish I had written my own WebDAV server from scratch using 'dropwizard' instead. The world of high availability services changes so rapidly and many of the ideas in this project are dated-- like, even table joins really shouldn't exist anymore. -- rhodey orbits On 02/18/2015 01:04 PM, Gaurav Jain wrote:
Hi,
Thank you very much for CalendarServer
I have few questions:
* I used CalendarServer-5.2.2 during development phase. * Now I want to use it for some real (users) use cases. * Right now, I do not have any scaling issues. * I run it on Ubuntu EC2 instance.
My question is:
* Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts?
Any tip would be useful for PROD deployments. Even for 1000 users, we would like it not to go down at all.
Best Regards, Gaurav Jain
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
one more important note: If you're going to deploy this in production find yourself a python statsd library and add timers everywhere you can manage. -- rhodey orbits On 02/18/2015 04:18 PM, rhodey wrote:
I've been running DCS in production for the past ~year, at the moment my deployment supports ~800 users. It took *a lot* of reading through doc and code to feel even remotely comfortable with storing real data for real people and I still don't feel all that great about it.
Please, allow me to ramble for a minute, take from it what you will...
1. IMO the idea of XMLDirectoryService is seriously dated and kinda crazy so I wrote my own postgresql backed DirectoryService with support for memcached. This SQL-backed DirectoryService is much easier to manage (because SQL) and allows me to have other services work off of the same account database. Speaking of memcached, DCS (as far as I can tell) caches very little and could stand to benefit from using it more often.
2. DCS uses the python twisted framework but at the same time it doesn't-- calls to DirectoryService (regardless of the implementation) are blocking 0.o
3. DCS has this weird multi-process, master-slave concept where multiple DCS instances run on the same box-- I just think this is weird and dealing with dead processes has been strange.
4. If a user uploads a contact containing a photo and you're using a database for storage (as you should), DCS will store that photo in your database-- yuck! Even with your 1000 users, each with maybe 100 contact photos, things get kinda bloated. So, I modified DCS to strip contact photos from contacts and store them in Amazon S3, then piece them back together on demand.
5. Someone could DOS my DCS instances via network resource exhaustion very easily and there is very little that I could do to stop it other than launch a crazy number of EC2 instances and burn through money. DCS is not efficient (blocking calls to DirectoryService, etc) and I feel very uncomfortable having any less than than two m3.medium instances running at all times (costly!).
All that being said, DCS is still the best free and open source option out there but I wish I had written my own WebDAV server from scratch using 'dropwizard' instead. The world of high availability services changes so rapidly and many of the ideas in this project are dated-- like, even table joins really shouldn't exist anymore.
-- rhodey orbits
On 02/18/2015 01:04 PM, Gaurav Jain wrote:
Hi,
Thank you very much for CalendarServer
I have few questions:
* I used CalendarServer-5.2.2 during development phase. * Now I want to use it for some real (users) use cases. * Right now, I do not have any scaling issues. * I run it on Ubuntu EC2 instance.
My question is:
* Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts?
Any tip would be useful for PROD deployments. Even for 1000 users, we would like it not to go down at all.
Best Regards, Gaurav Jain
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
Hi Rhodey, Comments inline.
On Feb 18, 2015, at 4:18 PM, rhodey <rhodey@anhonesteffort.org> wrote:
I've been running DCS in production for the past ~year, at the moment my deployment supports ~800 users. It took *a lot* of reading through doc and code to feel even remotely comfortable with storing real data for real people and I still don't feel all that great about it.
We are slowly compiling actual documentation. I'll take this as feedback in support of accelerating this process.
...
2. DCS uses the python twisted framework but at the same time it doesn't-- calls to DirectoryService (regardless of the implementation) are blocking 0.o
I believe this is addressed in 6.0+ with the new twext.who stuff that is all async, iirc.
3. DCS has this weird multi-process, master-slave concept where multiple DCS instances run on the same box-- I just think this is weird and dealing with dead processes has been strange.
The Python GIL means we need a multi process architecture to saturate CPU on multi core systems. Technically these processes are all part of the same service instance because they all cooperate on doing the work.
4. If a user uploads a contact containing a photo and you're using a database for storage (as you should), DCS will store that photo in your database-- yuck! Even with your 1000 users, each with maybe 100 contact photos, things get kinda bloated. So, I modified DCS to strip contact photos from contacts and store them in Amazon S3, then piece them back together on demand.
Cool hack :)
5. Someone could DOS my DCS instances via network resource exhaustion very easily and there is very little that I could do to stop it other than launch a crazy number of EC2 instances and burn through money. DCS is not efficient (blocking calls to DirectoryService, etc) and I feel very uncomfortable having any less than than two m3.medium instances running at all times (costly!).
Understood, however this is true of any network service that limits its own concurrency (imo this is required of any sane service). Part of the challenge is the extremely large range of request cost for legitimate CalDAV requests. We are moving to a model where we do more work a synchronously in a database work queue, and we now have a notion of cost in that implementation, so we are becoming better equipped to address resource utilization problems.
All that being said, DCS is still the best free and open source option out there but I wish I had written my own WebDAV server from scratch using 'dropwizard' instead. The world of high availability services changes so rapidly and many of the ideas in this project are dated-- like, even table joins really shouldn't exist anymore.
We're not yet ready for NoSQL but maybe someday :) Thanks for the feedback. I'm interested to hear anything else you'd like to share about your experiences hosting DCS. -dre
-- rhodey orbits
On 02/18/2015 01:04 PM, Gaurav Jain wrote: Hi,
Thank you very much for CalendarServer
I have few questions:
* I used CalendarServer-5.2.2 during development phase. * Now I want to use it for some real (users) use cases. * Right now, I do not have any scaling issues. * I run it on Ubuntu EC2 instance.
My question is:
* Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts?
Any tip would be useful for PROD deployments. Even for 1000 users, we would like it not to go down at all.
Best Regards, Gaurav Jain
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
Thank you very much for all your support! On Wed, Feb 18, 2015 at 4:49 PM, Andre LaBranche <dre@apple.com> wrote:
Hi Rhodey,
Comments inline.
On Feb 18, 2015, at 4:18 PM, rhodey <rhodey@anhonesteffort.org> wrote:
I've been running DCS in production for the past ~year, at the moment my deployment supports ~800 users. It took *a lot* of reading through doc and code to feel even remotely comfortable with storing real data for real people and I still don't feel all that great about it.
We are slowly compiling actual documentation. I'll take this as feedback in support of accelerating this process.
...
2. DCS uses the python twisted framework but at the same time it doesn't-- calls to DirectoryService (regardless of the implementation) are blocking 0.o
I believe this is addressed in 6.0+ with the new twext.who stuff that is all async, iirc.
3. DCS has this weird multi-process, master-slave concept where multiple DCS instances run on the same box-- I just think this is weird and dealing with dead processes has been strange.
The Python GIL means we need a multi process architecture to saturate CPU on multi core systems. Technically these processes are all part of the same service instance because they all cooperate on doing the work.
4. If a user uploads a contact containing a photo and you're using a database for storage (as you should), DCS will store that photo in your database-- yuck! Even with your 1000 users, each with maybe 100 contact photos, things get kinda bloated. So, I modified DCS to strip contact photos from contacts and store them in Amazon S3, then piece them back together on demand.
Cool hack :)
5. Someone could DOS my DCS instances via network resource exhaustion very easily and there is very little that I could do to stop it other than launch a crazy number of EC2 instances and burn through money. DCS is not efficient (blocking calls to DirectoryService, etc) and I feel very uncomfortable having any less than than two m3.medium instances running at all times (costly!).
Understood, however this is true of any network service that limits its own concurrency (imo this is required of any sane service). Part of the challenge is the extremely large range of request cost for legitimate CalDAV requests. We are moving to a model where we do more work a synchronously in a database work queue, and we now have a notion of cost in that implementation, so we are becoming better equipped to address resource utilization problems.
All that being said, DCS is still the best free and open source option out there but I wish I had written my own WebDAV server from scratch using 'dropwizard' instead. The world of high availability services changes so rapidly and many of the ideas in this project are dated-- like, even table joins really shouldn't exist anymore.
We're not yet ready for NoSQL but maybe someday :)
Thanks for the feedback. I'm interested to hear anything else you'd like to share about your experiences hosting DCS.
-dre
-- rhodey orbits
On 02/18/2015 01:04 PM, Gaurav Jain wrote: Hi,
Thank you very much for CalendarServer
I have few questions:
* I used CalendarServer-5.2.2 during development phase. * Now I want to use it for some real (users) use cases. * Right now, I do not have any scaling issues. * I run it on Ubuntu EC2 instance.
My question is:
* Has CalendarSever-5.2.2 been used in PROD deployments? * Since I don't have scaling issues, is it ok to use XMLDirectoryService for like 1000 accounts?
Any tip would be useful for PROD deployments. Even for 1000 users, we
would
like it not to go down at all.
Best Regards, Gaurav Jain
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
calendarserver-users mailing list calendarserver-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/calendarserver-users
participants (3)
-
Andre LaBranche
-
Gaurav Jain
-
rhodey