[CalendarServer-users] CalendarServer 3.1 on Ubuntu 11.10 Server

Jim J jahosefet at gmail.com
Fri Dec 30 14:08:55 PST 2011


Sorry if I'm missing something, but I'm not seeing how this fixes the hard
coded '_postgres' username in calendarserver_bootstrap_database script. I
still can't bootstrap the database using the script because it doesn't pull
database login information from the caldavd.plist file, it's hardcoded.
What am I missing?

Thanks,

Jim

On Wed, Dec 28, 2011 at 3:35 PM, Andre LaBranche <dre at apple.com> wrote:

>
> On Dec 28, 2011, at 11:29 AM, Jim J wrote:
>
> Thanks Andre!
>
> The idea works, not so much the implementation though. I'll post what I've
> found, perhaps I missed something otherwise it might be something to look
> into or at least help others.
>
> I was able to set up the database schema but I couldn't do it using the
> bootstrap Python script. The script imports
> calendarserver.tools.bootstrapdatabase and runs main(). The
> bootstrapdatabase.py file makes some improper (and un-overrideable)
> assumptions. First, CONNECTNAME is assumed to be "_postgres" but on Ubuntu
> it's just "postgres" (no leading underscore). Maybe this isn't a bad
> assumption on Mac OS X Server? Second it assumes the existence of database
> user "caldav" and database "caldav". Unfortunately I'd already committed to
> 'caldavd' (trailing 'd') being the username and group for the daemon so
> that didn't work either.
>
>
> See the 'database connectivity' section in this document:
>
> http://trac.calendarserver.org/wiki/MultiServerDeployment
>
> Basically you just need to set the postgres DSN as appropriate for your
> setup.
>
> Cheers,
> -dre
>
>
> I went into the Python file and changed those. Then it assumed that the
> SQL script that it needed to execute was located at
> /usr/share/caldavd/lib/python/.../current.sql. However, when I installed
> calendarserver using ./run -i that's not where it put the SQL script! It
> put it in /usr/local/lib/python2.7/dist-packages/.../current.sql. I ended
> up quitting at this point and logged into PostgresSQL as the caldavd user
> and manually executed the SQL script.
>
> Finally I was able to get calendarserver fully running *without SSL
> encryption*. When I went to enable SSL, I seem to have tripped over some
> crazy bug in the twisted framework. I didn't dig too deep but in Twisted
> 11.0.0 there's a file internet/tcp.py. In the Connection class there's an
> "if SSL:" and then a bunch of functions. Inside _startTLS there's a line:
>
> self.fileno = self.socket.fileno
>
> This line asserts. I don't really know why. Something about
> OpenSSH.SSL.Connection doesn't have attribute 'fileno'. Anyway, I gave up
> at this point, locked the unencrypted calendarserver to localhost and
> wrapped it in stunnel4. That seems to work. That line doesn't seem to exist
> in Twisted 10.x or even in 11.1.0 so I'm guessing it was a problem.
>
> Thanks for the help.
>
> Jim
>
> On Tue, Dec 27, 2011 at 7:10 PM, Andre LaBranche <dre at apple.com> wrote:
>
>> Forgot to cc the list... See quoted text for answers :)
>>
>> -dre
>>
>> Sent from my iPhone
>>
>> On Dec 27, 2011, at 6:34 PM, Andre LaBranche <dre at apple.com> wrote:
>>
>> We only provision schema if we spawn Postgres. To use an externally
>> managed DB, provision schema with calendarserver_bootstrap_database.
>>
>> Cheers,
>> -dre
>>
>> Sent from my iPhone
>>
>> On Dec 27, 2011, at 4:33 PM, Jim J <jahosefet at gmail.com> wrote:
>>
>> Andre,
>>
>> Thanks! That's a very useful link actually. I've been trying to figure
>> out what all the options are. For some reason it didn't catch my attention
>> during my 'grep's. That fixes all the above problems, I can even use -u and
>> -g now.
>>
>> Next problem:
>> I created the caldavd user in PostgreSQL and created the caldavd database
>> (I'm using -u caldavd -g caldavd again). I noticed this connects to the
>> database using the caldavd user. Now I get this error:
>>
>> 2011-12-27 16:17:47-0500 [-]
>> [txdav.common.datastore.util.UpgradeDatabaseSchemaService#warn] Beginning
>> database schema check.
>> 2011-12-27 16:17:47-0500 [-]
>> [txdav.common.datastore.util.UpgradeDatabaseSchemaService#warn] Required
>> schema version: 3.
>> 2011-12-27 16:17:47-0500 [-] Unhandled error in Deferred:
>> 2011-12-27 16:17:47-0500 [-] Unhandled Error
>>         Traceback (most recent call last):
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 388, in
>> errback
>>             self._startRunCallbacks(fail)
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 455, in
>> _startRunCallbacks
>>             self._runCallbacks()
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 542, in
>> _runCallbacks
>>             current.result = callback(current.result, *args, **kw)
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1076, in
>> gotResult
>>             _inlineCallbacks(r, g, deferred)
>>         --- <exception caught here> ---
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1018, in
>> _inlineCallbacks
>>             result = result.throwExceptionIntoGenerator(g)
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 350, in
>> throwExceptionIntoGenerator
>>             return g.throw(self.type, self.value, self.tb)
>>           File
>> "/usr/local/lib/python2.7/dist-packages/txdav/common/datastore/util.py",
>> line 271, in doUpgrade
>>             actual_version = yield sqlTxn.schemaVersion()
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1018, in
>> _inlineCallbacks
>>             result = result.throwExceptionIntoGenerator(g)
>>           File
>> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 350, in
>> throwExceptionIntoGenerator
>>             return g.throw(self.type, self.value, self.tb)
>>           File
>> "/usr/local/lib/python2.7/dist-packages/txdav/common/datastore/sql.py",
>> line 233, in schemaVersion
>>             result = yield self._schemaVersion.on(self)
>>           File
>> "/usr/local/lib/python2.7/dist-packages/twext/internet/threadutils.py",
>> line 64, in _oneWorkUnit
>>             result = instruction()
>>           File
>> "/usr/local/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line
>> 155, in <lambda>
>>             lambda : self._reallyExecSQL(*args, **kw)
>>           File
>> "/usr/local/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line
>> 136, in _reallyExecSQL
>>             result = self._reallyExecSQL(sql, args, raiseOnZeroRowCount)
>>           File
>> "/usr/local/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line
>> 130, in _reallyExecSQL
>>             self._cursor.execute(sql, args)
>>           File
>> "/usr/local/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py",
>> line 76, in execute
>>             self.realCursor.execute(sql, args)
>>           File "/usr/local/lib/python2.7/dist-packages/pgdb.py", line
>> 259, in execute
>>             self.executemany(operation, (params,))
>>           File "/usr/local/lib/python2.7/dist-packages/pgdb.py", line
>> 289, in executemany
>>             raise DatabaseError("error '%s' in '%s'" % (msg, sql))
>>         pg.DatabaseError: error 'ERROR:  relation "calendarserver" does
>> not exist
>>         LINE 1: select VALUE from CALENDARSERVER where NAME = 'VERSION'
>>                                   ^
>>         ' in 'select VALUE from CALENDARSERVER where NAME = 'VERSION''
>>
>>
>> I would have expected calendarserver to set up its own tables/schema (as
>> it appears to be attempting to do). Can I force it to rebuild the database?
>>
>> Thanks,
>>
>> Jim
>>
>> On Tue, Dec 27, 2011 at 1:40 PM, Andre LaBranche <dre at apple.com> wrote:
>>
>>> Hi,
>>>
>>> Try setting UseDatabase to Postgres, as described here:
>>> http://trac.calendarserver.org/browser/CalendarServer/tags/release/CalendarServer-3.1/twistedcaldav/stdconfig.py
>>>
>>> You will also need to specify DSN which tells calendar server how to
>>> connect to postgres.
>>>
>>> You can also set ServerEnabled to false for memcached to keep calendar
>>> server from spawning memcached.
>>>
>>> I can provide examples once I get back home to a real machine...
>>>
>>> Cheers,
>>> -dre
>>>
>>> Sent from my iPhone
>>>
>>> On Dec 27, 2011, at 1:21 PM, Jim J <jahosefet at gmail.com> wrote:
>>>
>>> All,
>>>
>>> I'm quite new to this calendarserver thing. I'm trying to set up 3.1
>>> (from tags/releases in Subversion) on a 64-bit Ubuntu 11.10 server
>>> installation. It's been a disaster. I was able to run calendarserver from
>>> the SVN directory using ./run and then I "installed" it using ./run -i.
>>> That's where things went bad. Some issues I've worked around, but I'd like
>>> to get good solutions for. Currently I'm stuck trying to figure out what's
>>> going on with PostgreSQL. Perhaps you guys can clue me in on how to
>>> properly fix the following issues.
>>>
>>> This is how I'm currently executing caldavd. Once it works I'll put it
>>> into init.d and run it as a proper server. The () are because I *was*
>>> running with those flags (see part 1) and have given up on them.
>>>
>>> sudo caldavd -X -L -f <path to caldavd.plist file> (-u caldavd -g
>>> caldavd)
>>>
>>> 1. As I understand it, it is good practice to run Linux services that
>>> are exposed to the internet as a restricted user. Given that and existence
>>> of the -u and -g flags for caldavd, I figured those flags would be the
>>> right way to do things. That doesn't work at all.
>>>
>>> The first problem is that caldavd wants to write to
>>> /var/run/ccs_postgresql_xxxxx which it can't do because /var belongs to
>>> root and it attempts to do this after the setuid() call to become the
>>> caldavd user. So, I went into caldavd.plist and changed the "DataRoot" to a
>>> different directory and gave it to caldavd:caldavd.
>>>
>>> Then I tried again. This time it failed because it appears that
>>> calendarserver attempts to start postgresql whether or not the postgresql
>>> service is already running on the local box. To do this, it tries to
>>> setuid() again,  this time to ROOT! If the entire point of setting a user
>>> for a service is to restrict it's privileges, why would one of those privs
>>> be to allow setuid to root? That makes no sense.
>>>
>>> So I gave up, now I'm trying to run it as root to see if the 3.1 release
>>> even functions. How do I make this server run as somebody other than root?
>>>
>>> 2. Running as root (without the -u and -g flags shown above) gets the
>>> application started and then it fails attempting to start memcached. I get
>>> the error "cannot listen on tcp port". This is because memcachd is a system
>>> service and is *already running*. Again, why is calendarserver attempting
>>> to start a service that's already running?
>>>
>>> So I stopped memcachd and tried again.
>>>
>>> 3. Calendarserver now successfully starts and connects to memcached but
>>> it cannot connect to postgresql. This I haven't yet worked around. I get
>>> the following error:
>>> 2011-12-27 12:43:11-0500 [-] [txdav.base.datastore.subpostgres#error]
>>> [Failure instance: Traceback: <class 'pg.InternalError'>: could not connect
>>> to server: No such file or directory
>>>                 Is the server running locally and accepting
>>>                 connections on Unix domain socket
>>> "/tmp/ccs_postgres_f5e263275ce7de576bd42ca08db4fc56/.s.PGSQL.5432"?
>>>
>>>
>>> /usr/lib/python2.7/dist-packages/twisted/internet/_baseprocess.py:60:maybeCallProcessEnded
>>>
>>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:92:processEnded
>>>
>>> /usr/lib/python2.7/dist-packages/twisted/internet/defer.py:361:callback
>>>
>>> /usr/lib/python2.7/dist-packages/twisted/internet/defer.py:455:_startRunCallbacks
>>>         --- <exception caught here> ---
>>>
>>> /usr/lib/python2.7/dist-packages/twisted/internet/defer.py:542:_runCallbacks
>>>
>>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:367:gotReady
>>>
>>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:269:ready
>>>
>>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:261:produceConnection
>>>
>>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py:196:connect
>>>         /usr/local/lib/python2.7/dist-packages/pgdb.py:482:connect
>>>         ]
>>>
>>> Here I'm stuck. Quite clearly calendarserver is expecting to launch the
>>> postgresql service (which, again, doesn't make sense to me), but then it
>>> can't connect when it does. Well, at least I'm assuming calendarserver
>>> successfully launches postgresql because I don't get the setuid() error I
>>> got in part 1 when it tries to setuid() to root in order to launch
>>> postgresql. I can't figure out how to fix this. Any thoughts?
>>>
>>> Thanks for the help,
>>>
>>> Jim
>>>
>>> _______________________________________________
>>> calendarserver-users mailing list
>>> calendarserver-users at lists.macosforge.org
>>> http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
>>>
>>>
>>
>> _______________________________________________
>> calendarserver-users mailing list
>> calendarserver-users at lists.macosforge.org
>> http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/calendarserver-users/attachments/20111230/8607e6eb/attachment-0001.html>


More information about the calendarserver-users mailing list