Good morning! First off, I hope there's not an issue with asking Apple server-specific questions on this list. If that's an issue, I apologize, but this is somewhat documentation-related :) What is the "preferred" method for migration of calendar data? I have two use cases that I'm studying: - Migration of users in an enterprise from local calendars or another server to a Snow or Lion calendar server - Migration of users from Snow to Lion without using the upgrade installation (a migration installation instead) In both cases, we've tried to use iCal to either back up a copy of the local client store to copy into the new server store. That generally works, but both calendar servers (Snow and Lion) will try to mail out new invitations. I've mitigated that before by disabling iMIP temporarily in serveradmin, which generally seems to work, but then there are some errors when the calendar server tries to resolve users on the invites: HTTP/1.1: 500 Internal Server Error to operation CalDAVWriteEntityQueueableOperation ...after which you can revert to server or go offline. Revert to server's not a very good choice because it will revert to the non-existence of the item. All in all, what is a good way to approach this migration? If you can provide some clues/pointers, I'm sure we can figure out the rest. This will be a very good migration document in the end, I'm sure. Unless I'm missing something, there's not a lot of information about this in the Lion server documentation. --Jason --- Jason Miller jmiller@red-abstract.com
I'd just like to chime in here and second the request for better tools and/or documentation. I have a similar question (two existing users, data going back more than two years), and Conrad Wasmer asked essentially the same question a couple weeks ago in an email with the subject "migrating to Lion Server." I found a response in the archives about nuking from orbit, copying old data into the directory where the server will do a one-time upgrade, and then performing the initialization all over again, but that doesn't work for my scenario either (the existing users had auto-generated UUIDs in an XML config; the new server has Open Directory and thus new UUIds for both those existing users and a number of new ones). Is there a tool or command that will allow one to parse a directory copied from the old server and insert those items into the database directly, ideally with UUID mapping available? Alternatively, is there a command that will insert a single item into the database with notifications suppressed? If that exists, I can write a wrapper that will parse the directories and do individual inserts for each of them. Thanks! - e On Sep 20, 2011, at 10:19 AM, Jason Miller wrote:
Good morning!
First off, I hope there's not an issue with asking Apple server-specific questions on this list. If that's an issue, I apologize, but this is somewhat documentation-related :)
What is the "preferred" method for migration of calendar data?
I have two use cases that I'm studying:
- Migration of users in an enterprise from local calendars or another server to a Snow or Lion calendar server
- Migration of users from Snow to Lion without using the upgrade installation (a migration installation instead)
In both cases, we've tried to use iCal to either back up a copy of the local client store to copy into the new server store. That generally works, but both calendar servers (Snow and Lion) will try to mail out new invitations. I've mitigated that before by disabling iMIP temporarily in serveradmin, which generally seems to work, but then there are some errors when the calendar server tries to resolve users on the invites:
HTTP/1.1: 500 Internal Server Error to operation CalDAVWriteEntityQueueableOperation
...after which you can revert to server or go offline. Revert to server's not a very good choice because it will revert to the non-existence of the item.
All in all, what is a good way to approach this migration? If you can provide some clues/pointers, I'm sure we can figure out the rest. This will be a very good migration document in the end, I'm sure. Unless I'm missing something, there's not a lot of information about this in the Lion server documentation.
--Jason
--- Jason Miller jmiller@red-abstract.com
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
While I did call out the need for documentation, I'm willing to help provide that documentation, which is why I was asking ;) --Jason --- Jason Miller jmiller@red-abstract.com On Sep 20, 2011, at 12:53 PM, Ed Poe wrote:
I'd just like to chime in here and second the request for better tools and/or documentation. I have a similar question (two existing users, data going back more than two years), and Conrad Wasmer asked essentially the same question a couple weeks ago in an email with the subject "migrating to Lion Server."
I found a response in the archives about nuking from orbit, copying old data into the directory where the server will do a one-time upgrade, and then performing the initialization all over again, but that doesn't work for my scenario either (the existing users had auto-generated UUIDs in an XML config; the new server has Open Directory and thus new UUIds for both those existing users and a number of new ones). Is there a tool or command that will allow one to parse a directory copied from the old server and insert those items into the database directly, ideally with UUID mapping available? Alternatively, is there a command that will insert a single item into the database with notifications suppressed? If that exists, I can write a wrapper that will parse the directories and do individual inserts for each of them.
Thanks!
- e
On Sep 20, 2011, at 10:19 AM, Jason Miller wrote:
Good morning!
First off, I hope there's not an issue with asking Apple server-specific questions on this list. If that's an issue, I apologize, but this is somewhat documentation-related :)
What is the "preferred" method for migration of calendar data?
I have two use cases that I'm studying:
- Migration of users in an enterprise from local calendars or another server to a Snow or Lion calendar server
- Migration of users from Snow to Lion without using the upgrade installation (a migration installation instead)
In both cases, we've tried to use iCal to either back up a copy of the local client store to copy into the new server store. That generally works, but both calendar servers (Snow and Lion) will try to mail out new invitations. I've mitigated that before by disabling iMIP temporarily in serveradmin, which generally seems to work, but then there are some errors when the calendar server tries to resolve users on the invites:
HTTP/1.1: 500 Internal Server Error to operation CalDAVWriteEntityQueueableOperation
...after which you can revert to server or go offline. Revert to server's not a very good choice because it will revert to the non-existence of the item.
All in all, what is a good way to approach this migration? If you can provide some clues/pointers, I'm sure we can figure out the rest. This will be a very good migration document in the end, I'm sure. Unless I'm missing something, there's not a lot of information about this in the Lion server documentation.
--Jason
--- Jason Miller jmiller@red-abstract.com
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
Hi Ed, UUIDs are essential to calendaring data, and they really can't change without breaking stuff. For this reason, it's critical that all migrations of calendar data preserve the UUIDs, and the relevant user accounts do the same. I would recommend preserving the UUIDs instead of trying to 'fix' the data. I realize this is totally inconvenient, but it's pretty much the only way to get the job done without doing highly unsupported and potentially impossible things to your calendar data. I don't believe there's a way to upload calendar data (via CalDAV, which is the only way you should try to do this) while suppressing notifications - other than by turning notifications off in the server config. The bottom line is that this is a complex data set with many inter-related moving parts. Mac OS X Server mostly hides the complexity (assuming you don't try to get crazy with it), but in this, the open source world, it's not so hidden. -dre On Sep 20, 2011, at 10:53 AM, Ed Poe wrote:
I'd just like to chime in here and second the request for better tools and/or documentation. I have a similar question (two existing users, data going back more than two years), and Conrad Wasmer asked essentially the same question a couple weeks ago in an email with the subject "migrating to Lion Server."
I found a response in the archives about nuking from orbit, copying old data into the directory where the server will do a one-time upgrade, and then performing the initialization all over again, but that doesn't work for my scenario either (the existing users had auto-generated UUIDs in an XML config; the new server has Open Directory and thus new UUIds for both those existing users and a number of new ones). Is there a tool or command that will allow one to parse a directory copied from the old server and insert those items into the database directly, ideally with UUID mapping available? Alternatively, is there a command that will insert a single item into the database with notifications suppressed? If that exists, I can write a wrapper that will parse the directories and do individual inserts for each of them.
Thanks!
- e
On Sep 20, 2011, at 10:19 AM, Jason Miller wrote:
Good morning!
First off, I hope there's not an issue with asking Apple server-specific questions on this list. If that's an issue, I apologize, but this is somewhat documentation-related :)
What is the "preferred" method for migration of calendar data?
I have two use cases that I'm studying:
- Migration of users in an enterprise from local calendars or another server to a Snow or Lion calendar server
- Migration of users from Snow to Lion without using the upgrade installation (a migration installation instead)
In both cases, we've tried to use iCal to either back up a copy of the local client store to copy into the new server store. That generally works, but both calendar servers (Snow and Lion) will try to mail out new invitations. I've mitigated that before by disabling iMIP temporarily in serveradmin, which generally seems to work, but then there are some errors when the calendar server tries to resolve users on the invites:
HTTP/1.1: 500 Internal Server Error to operation CalDAVWriteEntityQueueableOperation
...after which you can revert to server or go offline. Revert to server's not a very good choice because it will revert to the non-existence of the item.
All in all, what is a good way to approach this migration? If you can provide some clues/pointers, I'm sure we can figure out the rest. This will be a very good migration document in the end, I'm sure. Unless I'm missing something, there's not a lot of information about this in the Lion server documentation.
--Jason
--- Jason Miller jmiller@red-abstract.com
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
But if you disable iMIP on the server and take the client's Apple Mail offline, that seems to mitigate the email storm that results from importing events. It doesn't, however, solve the HTTP 500 errors that result. I think your answer provides clues, though, Dre. Sounds like there's a need for someone to crack open some scripting. --Jason --- Jason Miller jmiller@red-abstract.com On Sep 20, 2011, at 1:26 PM, Andre LaBranche wrote:
Hi Ed,
UUIDs are essential to calendaring data, and they really can't change without breaking stuff. For this reason, it's critical that all migrations of calendar data preserve the UUIDs, and the relevant user accounts do the same. I would recommend preserving the UUIDs instead of trying to 'fix' the data. I realize this is totally inconvenient, but it's pretty much the only way to get the job done without doing highly unsupported and potentially impossible things to your calendar data.
I don't believe there's a way to upload calendar data (via CalDAV, which is the only way you should try to do this) while suppressing notifications - other than by turning notifications off in the server config.
The bottom line is that this is a complex data set with many inter-related moving parts. Mac OS X Server mostly hides the complexity (assuming you don't try to get crazy with it), but in this, the open source world, it's not so hidden.
-dre
On Sep 20, 2011, at 10:53 AM, Ed Poe wrote:
I'd just like to chime in here and second the request for better tools and/or documentation. I have a similar question (two existing users, data going back more than two years), and Conrad Wasmer asked essentially the same question a couple weeks ago in an email with the subject "migrating to Lion Server."
I found a response in the archives about nuking from orbit, copying old data into the directory where the server will do a one-time upgrade, and then performing the initialization all over again, but that doesn't work for my scenario either (the existing users had auto-generated UUIDs in an XML config; the new server has Open Directory and thus new UUIds for both those existing users and a number of new ones). Is there a tool or command that will allow one to parse a directory copied from the old server and insert those items into the database directly, ideally with UUID mapping available? Alternatively, is there a command that will insert a single item into the database with notifications suppressed? If that exists, I can write a wrapper that will parse the directories and do individual inserts for each of them.
Thanks!
- e
On Sep 20, 2011, at 10:19 AM, Jason Miller wrote:
Good morning!
First off, I hope there's not an issue with asking Apple server-specific questions on this list. If that's an issue, I apologize, but this is somewhat documentation-related :)
What is the "preferred" method for migration of calendar data?
I have two use cases that I'm studying:
- Migration of users in an enterprise from local calendars or another server to a Snow or Lion calendar server
- Migration of users from Snow to Lion without using the upgrade installation (a migration installation instead)
In both cases, we've tried to use iCal to either back up a copy of the local client store to copy into the new server store. That generally works, but both calendar servers (Snow and Lion) will try to mail out new invitations. I've mitigated that before by disabling iMIP temporarily in serveradmin, which generally seems to work, but then there are some errors when the calendar server tries to resolve users on the invites:
HTTP/1.1: 500 Internal Server Error to operation CalDAVWriteEntityQueueableOperation
...after which you can revert to server or go offline. Revert to server's not a very good choice because it will revert to the non-existence of the item.
All in all, what is a good way to approach this migration? If you can provide some clues/pointers, I'm sure we can figure out the rest. This will be a very good migration document in the end, I'm sure. Unless I'm missing something, there's not a lot of information about this in the Lion server documentation.
--Jason
--- Jason Miller jmiller@red-abstract.com
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
Hi Jason, --On September 20, 2011 1:51:39 PM -0500 Jason Miller <jmiller@red-abstract.com> wrote:
But if you disable iMIP on the server and take the client's Apple Mail offline, that seems to mitigate the email storm that results from importing events. It doesn't, however, solve the HTTP 500 errors that result.
I think your answer provides clues, though, Dre. Sounds like there's a need for someone to crack open some scripting.
What shows up in /var/log/caldavd/error.log when the 500 error occurs? -- Cyrus Daboo
Cyrus... We're doing some traceback here as I have two datasets to look at. Still looking at one of them, but I believe these events correspond to the use case of migrating data from a Snow iCal client to a CalDAV Lion server: 2011-09-04 22:51:56-0500 [-] [caldav-0] [-] [twistedcaldav.directory.principal#error] No principal found for UID: 01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:51:56-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.directory.principal#error] No principal found for UID: 01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:51:56-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.scheduling.scheduler#error] Unknown calendar user address: urn:uuid:01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:51:56-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.scheduling.scheduler.ScheduleResponseQueue#error] Error during PUT for urn:uuid:01A3FBCF-A61C-4EE8-A0FE-44C30A660191: Unknown recipient 2011-09-04 22:52:11-0500 [-] [caldav-0] [-] [twistedcaldav.directory.principal#error] No principal found for UID: 01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:52:11-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.directory.principal#error] No principal found for UID: 01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:52:11-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.scheduling.scheduler#error] Unknown calendar user address: urn:uuid:01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:52:11-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.scheduling.scheduler.ScheduleResponseQueue#error] Error during PUT for urn:uuid:01A3FBCF-A61C-4EE8-A0FE-44C30A660191: Unknown recipient 2011-09-04 22:52:29-0500 [-] [caldav-0] [-] [twistedcaldav.directory.principal#error] No principal found for UID: 01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:52:29-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.scheduling.scheduler#error] Unknown calendar user address: urn:uuid:01A3FBCF-A61C-4EE8-A0FE-44C30A660191 2011-09-04 22:52:29-0500 [-] [caldav-0] [PooledMemCacheProtocol,client] [twistedcaldav.scheduling.scheduler.ScheduleResponseQueue#error] Error during POST for urn:uuid:01A3FBCF-A61C-4EE8-A0FE-44C30A660191: Unknown recipient 2011-09-05 00:48:51-0500 [-] [caldav-0] [NotificationClientLineProtocol,client] [twistedcaldav.notify.NotificationClientFactory#error] Connect to notification server lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly. 2011-09-05 00:48:51-0500 [-] [caldav-1] [NotificationClientLineProtocol,client] [twistedcaldav.notify.NotificationClientFactory#error] Connect to notification server lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly. This would seem to corroborate with what Dre proposed as an issue: the UUID should be preserved. What's confusing about this is that the users had been imported to OD prior to pulling in any calendar data, so I wouldn't expect the UUID to be different. It should also not force me to use a fatal option of "revert to server" when that will cause it to delete the imported event. It would be nice if it would say, "Okie, I'll take this event, but I don't know who those invitees are." The alternative seems to be an inability to import the data at all with an iCal client...? --Jason --- Jason Miller jmiller@red-abstract.com On Sep 20, 2011, at 2:58 PM, Cyrus Daboo wrote:
Hi Jason,
--On September 20, 2011 1:51:39 PM -0500 Jason Miller <jmiller@red-abstract.com> wrote:
But if you disable iMIP on the server and take the client's Apple Mail offline, that seems to mitigate the email storm that results from importing events. It doesn't, however, solve the HTTP 500 errors that result.
I think your answer provides clues, though, Dre. Sounds like there's a need for someone to crack open some scripting.
What shows up in /var/log/caldavd/error.log when the 500 error occurs?
-- Cyrus Daboo
On Sep 20, 2011, at 2:07 PM, Jason Miller wrote:
It should also not force me to use a fatal option of "revert to server" when that will cause it to delete the imported event. It would be nice if it would say, "Okie, I'll take this event, but I don't know who those invitees are."
The alternative seems to be an inability to import the data at all with an iCal client...?
The server is doing you a big favor by not accepting invalid data :) It's critical that the starting conditions be correct, or this just won't work. -dre
On Sep 20, 2011, at 4:11 PM, Andre LaBranche wrote:
On Sep 20, 2011, at 2:07 PM, Jason Miller wrote:
It should also not force me to use a fatal option of "revert to server" when that will cause it to delete the imported event. It would be nice if it would say, "Okie, I'll take this event, but I don't know who those invitees are."
The alternative seems to be an inability to import the data at all with an iCal client...?
The server is doing you a big favor by not accepting invalid data :) It's critical that the starting conditions be correct, or this just won't work.
I don't know that I would classify an unknown attendee as invalid data. If you invite someone to a meeting that isn't a user local to the server, that would be the same criteria, no? The server accepts that data. From the user's standpoint, there's nothing invalid enough about an unknown attendee to force me to take the event out of my calendar store. It also severely hinders your ability to migrate data for users into your shiny new calendar server as there's a pretty good chance they've invited someone external to the calendar server to a meeting at some point. ...or is there a different piece of data that it's choking on that I'm missing? Is the calendar server using the UUID from the Open Directory? Since I imported those users with workgroup manager I would think they are the same, but maybe not. --Jason P.S. Where my original message in this thread said "invitee" I meant to say "attendee." --- Jason Miller jmiller@red-abstract.com
On Sep 20, 2011, at 7:19 AM, Jason Miller wrote:
Good morning!
First off, I hope there's not an issue with asking Apple server-specific questions on this list. If that's an issue, I apologize, but this is somewhat documentation-related :)
What is the "preferred" method for migration of calendar data?
I have two use cases that I'm studying:
- Migration of users in an enterprise from local calendars or another server to a Snow or Lion calendar server
For local calendars, you can create CalDAV accounts and simply upload the users events into their respective CalDAV accounts. To automate this, something like curl would be fine, just PUTing each one. Since they are local calendars, these events should be all non-scheduled (i.e. no attendees), so there's not much to worry about in the way of relationships to other data or users.
- Migration of users from Snow to Lion without using the upgrade installation (a migration installation instead)
I don't think I can easily explain this, as it's quite complicated. If you have a Lion server, look at: /System/Library/ServerSetup/MigrationExtras/59_calendarmigrator.py In the past, it has been possible (generically, not necessarily with calendar stuff) to manually invoke these migration scripts to get the job done, provided you have set the stage properly. Perhaps that can work here as well... HTH, -dre
participants (4)
-
Andre LaBranche
-
Cyrus Daboo
-
Ed Poe
-
Jason Miller