[CalendarServer-changes] [7770] CalendarServer/trunk
source_changes at macosforge.org
source_changes at macosforge.org
Mon Jul 11 15:08:29 PDT 2011
Revision: 7770
http://trac.macosforge.org/projects/calendarserver/changeset/7770
Author: glyph at apple.com
Date: 2011-07-11 15:08:29 -0700 (Mon, 11 Jul 2011)
Log Message:
-----------
When xattrs are not enabled on the filesystem containing a file-based store to
be migrated into a database, attempt to read WebDAV properties out of
AppleDouble-encoded files instead.
Modified Paths:
--------------
CalendarServer/trunk/txdav/base/propertystore/xattr.py
CalendarServer/trunk/txdav/caldav/datastore/file.py
CalendarServer/trunk/txdav/common/datastore/file.py
CalendarServer/trunk/txdav/common/datastore/util.py
Added Paths:
-----------
CalendarServer/trunk/txdav/base/propertystore/appledouble_xattr.py
CalendarServer/trunk/txdav/base/propertystore/test/test_appledouble.py
Removed Paths:
-------------
CalendarServer/trunk/contrib/tools/appledouble_xattr.py
Property Changed:
----------------
CalendarServer/trunk/
Property changes on: CalendarServer/trunk
___________________________________________________________________
Modified: svn:mergeinfo
- /CalendarServer/branches/config-separation:4379-4443
/CalendarServer/branches/egg-info-351:4589-4625
/CalendarServer/branches/generic-sqlstore:6167-6191
/CalendarServer/branches/new-store:5594-5934
/CalendarServer/branches/new-store-no-caldavfile:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692:5693-5702
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464:4465-4957
/CalendarServer/branches/users/cdaboo/pods:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar:7085-7206
/CalendarServer/branches/users/cdaboo/pycard:7227-7237
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187:5188-5440
/CalendarServer/branches/users/cdaboo/timezones:7443-7699
/CalendarServer/branches/users/glyph/conn-limit:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge:4971-5080
/CalendarServer/branches/users/glyph/dalify:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect:6824-6876
/CalendarServer/branches/users/glyph/deploybuild:7563-7572
/CalendarServer/branches/users/glyph/disable-quota:7718-7727
/CalendarServer/branches/users/glyph/dont-start-postgres:6592-6614
/CalendarServer/branches/users/glyph/linux-tests:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7:6369-6445
/CalendarServer/branches/users/glyph/new-export:7444-7485
/CalendarServer/branches/users/glyph/oracle:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls:7340-7351
/CalendarServer/branches/users/glyph/quota:7604-7637
/CalendarServer/branches/users/glyph/sendfdport:5388-5424
/CalendarServer/branches/users/glyph/sharedpool:6490-6550
/CalendarServer/branches/users/glyph/sql-store:5929-6073
/CalendarServer/branches/users/glyph/subtransactions:7248-7258
/CalendarServer/branches/users/glyph/uidexport:7673-7676
/CalendarServer/branches/users/glyph/use-system-twisted:5084-5149
/CalendarServer/branches/users/sagen/inboxitems:7380-7381
/CalendarServer/branches/users/sagen/locations-resources:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066:4068-4075
/CalendarServer/branches/users/sagen/resources-2:5084-5093
/CalendarServer/branches/users/wsanchez/transations:5515-5593
+ /CalendarServer/branches/config-separation:4379-4443
/CalendarServer/branches/egg-info-351:4589-4625
/CalendarServer/branches/generic-sqlstore:6167-6191
/CalendarServer/branches/new-store:5594-5934
/CalendarServer/branches/new-store-no-caldavfile:5911-5935
/CalendarServer/branches/new-store-no-caldavfile-2:5936-5981
/CalendarServer/branches/users/cdaboo/batchupload-6699:6700-7198
/CalendarServer/branches/users/cdaboo/cached-subscription-calendars-5692:5693-5702
/CalendarServer/branches/users/cdaboo/directory-cache-on-demand-3627:3628-3644
/CalendarServer/branches/users/cdaboo/more-sharing-5591:5592-5601
/CalendarServer/branches/users/cdaboo/partition-4464:4465-4957
/CalendarServer/branches/users/cdaboo/pods:7297-7377
/CalendarServer/branches/users/cdaboo/pycalendar:7085-7206
/CalendarServer/branches/users/cdaboo/pycard:7227-7237
/CalendarServer/branches/users/cdaboo/relative-config-paths-5070:5071-5105
/CalendarServer/branches/users/cdaboo/shared-calendars-5187:5188-5440
/CalendarServer/branches/users/cdaboo/timezones:7443-7699
/CalendarServer/branches/users/glyph/conn-limit:6574-6577
/CalendarServer/branches/users/glyph/contacts-server-merge:4971-5080
/CalendarServer/branches/users/glyph/dalify:6932-7023
/CalendarServer/branches/users/glyph/db-reconnect:6824-6876
/CalendarServer/branches/users/glyph/deploybuild:7563-7572
/CalendarServer/branches/users/glyph/disable-quota:7718-7727
/CalendarServer/branches/users/glyph/dont-start-postgres:6592-6614
/CalendarServer/branches/users/glyph/linux-tests:6893-6900
/CalendarServer/branches/users/glyph/misc-portability-fixes:7365-7374
/CalendarServer/branches/users/glyph/more-deferreds-6:6322-6368
/CalendarServer/branches/users/glyph/more-deferreds-7:6369-6445
/CalendarServer/branches/users/glyph/new-export:7444-7485
/CalendarServer/branches/users/glyph/oracle:7106-7155
/CalendarServer/branches/users/glyph/oracle-nulls:7340-7351
/CalendarServer/branches/users/glyph/quota:7604-7637
/CalendarServer/branches/users/glyph/sendfdport:5388-5424
/CalendarServer/branches/users/glyph/sharedpool:6490-6550
/CalendarServer/branches/users/glyph/sql-store:5929-6073
/CalendarServer/branches/users/glyph/subtransactions:7248-7258
/CalendarServer/branches/users/glyph/uidexport:7673-7676
/CalendarServer/branches/users/glyph/use-system-twisted:5084-5149
/CalendarServer/branches/users/glyph/xattrs-from-files:7757-7769
/CalendarServer/branches/users/sagen/inboxitems:7380-7381
/CalendarServer/branches/users/sagen/locations-resources:5032-5051
/CalendarServer/branches/users/sagen/locations-resources-2:5052-5061
/CalendarServer/branches/users/sagen/purge_old_events:6735-6746
/CalendarServer/branches/users/sagen/resource-delegates-4038:4040-4067
/CalendarServer/branches/users/sagen/resource-delegates-4066:4068-4075
/CalendarServer/branches/users/sagen/resources-2:5084-5093
/CalendarServer/branches/users/wsanchez/transations:5515-5593
Deleted: CalendarServer/trunk/contrib/tools/appledouble_xattr.py
===================================================================
--- CalendarServer/trunk/contrib/tools/appledouble_xattr.py 2011-07-11 22:03:37 UTC (rev 7769)
+++ CalendarServer/trunk/contrib/tools/appledouble_xattr.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -1,166 +0,0 @@
-#!/usr/bin/env python
-##
-# Copyright (c) 2010 Apple Inc. All rights reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-##
-
-from zlib import decompress
-import struct
-import sys
-import zlib
-
-# A lot of this is copied from python/plat-mac/applesingle.py,
-# with data structure information taken from
-# http://www.opensource.apple.com/source/Libc/Libc-391/darwin/copyfile.c
-
-# File header format: magic, version, unused, number of entries
-AS_HEADER_FORMAT=">LL16sh"
-AS_HEADER_LENGTH=26
-
-# The flag words for AppleDouble
-AS_MAGIC=0x00051607
-AS_VERSION=0x00020000
-
-# Entry header format: id, offset, length
-AS_ENTRY_FORMAT=">lll"
-AS_ENTRY_LENGTH=12
-
-# The id values
-AS_DATAFORK=1
-AS_RESOURCEFORK=2
-AS_REALNAME=3
-AS_COMMENT=4
-AS_ICONBW=5
-AS_ICONCOLOR=6
-AS_DATESINFO=8
-AS_FINDERINFO=9
-AS_MACFILEINFO=10
-AS_PRODOSFILEINFO=11
-AS_MSDOSFILEINFO=12
-AS_SHORTNAME=13
-AS_AFPFILEINFO=14
-AS_DIECTORYID=15
-
-FINDER_INFO_LENGTH = 32
-XATTR_OFFSET = FINDER_INFO_LENGTH + 2
-
-XATTR_HDR_MAGIC = 0x41545452 # ATTR
-XATTR_HEADER = ">llllllllhh"
-XATTR_HEADER_LENGTH = 36
-XATTR_ENTRY = ">llhb"
-XATTR_ENTRY_LENGTH = 11
-
-class AppleDouble(object):
-
- def __init__(self, fileobj, verbose=False, dezlib=False):
-
- self.xattrs = {}
-
- # Get the top-level header
- header = fileobj.read(AS_HEADER_LENGTH)
- try:
- magic, version, _ignore, nentry = struct.unpack(AS_HEADER_FORMAT, header)
- except ValueError, arg:
- raise ValueError("Unpack header error: %s" % (arg,))
- if verbose:
- print 'Magic: 0x%8.8x' % (magic,)
- print 'Version: 0x%8.8x' % (version,)
- print 'Entries: %d' % (nentry,)
- if magic != AS_MAGIC:
- raise ValueError("Unknown AppleDouble magic number 0x%8.8x" % (magic,))
- if version != AS_VERSION:
- raise ValueError("Unknown AppleDouble version number 0x%8.8x" % (version,))
- if nentry <= 0:
- raise ValueError("AppleDouble file contains no forks")
-
- # Get each entry
- headers = [fileobj.read(AS_ENTRY_LENGTH) for _ignore in xrange(nentry)]
- for hdr in headers:
- try:
- restype, offset, length = struct.unpack(AS_ENTRY_FORMAT, hdr)
- except ValueError, arg:
- raise ValueError("Unpack entry error: %s" % (arg,))
- if verbose:
- print "\n-- Fork %d, offset %d, length %d" % (restype, offset, length)
-
- # Look for the FINDERINFO entry with extra bits
- if restype == AS_FINDERINFO and length > FINDER_INFO_LENGTH:
-
- # Get the xattr header
- fileobj.seek(offset+XATTR_OFFSET)
- data = fileobj.read(length-XATTR_OFFSET)
- if len(data) != length-XATTR_OFFSET:
- raise ValueError("Short read: expected %d bytes got %d" % (length-XATTR_OFFSET, len(data)))
- magic, _ignore_tag, total_size, data_start, data_length, \
- _ignore_reserved1, _ignore_reserved2, _ignore_reserved3, \
- flags, num_attrs = struct.unpack(XATTR_HEADER, data[:XATTR_HEADER_LENGTH])
- if magic != XATTR_HDR_MAGIC:
- raise ValueError("No xattrs found")
- if verbose:
- print "\n Xattr Header"
- print ' Magic: 0x%08X' % (magic,)
- print ' Total Size: %d' % (total_size,)
- print ' Data Start: 0x%02X' % (data_start,)
- print ' Data Length: %d' % (data_length,)
- print ' Flags: 0x%02X' % (flags,)
- print ' Number: %d' % (num_attrs,)
-
- # Get each xattr entry
- data = data[XATTR_HEADER_LENGTH:]
- for _ignore in xrange(num_attrs):
- xattr_offset, xattr_length, xattr_flags, xattr_name_len = struct.unpack(XATTR_ENTRY, data[:XATTR_ENTRY_LENGTH])
- xattr_name = data[XATTR_ENTRY_LENGTH:XATTR_ENTRY_LENGTH+xattr_name_len]
- fileobj.seek(xattr_offset)
- xattr_value = fileobj.read(xattr_length)
-
- if dezlib:
- try:
- xattr_value = decompress(xattr_value)
- except zlib.error:
- pass
-
- if verbose:
- print "\n Xattr Entry"
- print ' Offset: 0x%02X' % (xattr_offset,)
- print ' Length: %d' % (xattr_length,)
- print ' Flags: 0x%02X' % (xattr_flags,)
- print ' Name: %s' % (xattr_name,)
- print ' Value: %s' % (xattr_value,)
- self.xattrs[xattr_name] = xattr_value
-
- # Skip over entry taking padding into account
- advance = (XATTR_ENTRY_LENGTH + xattr_name_len + 3) & ~3
- data = data[advance:]
-
-def _test():
- if len(sys.argv) < 2:
- print 'Usage: appledouble_xattr.py [-v] [-z] appledoublefile'
- sys.exit(1)
- if '-v' in sys.argv[1:]:
- verbose = True
- sys.argv.remove('-v')
- else:
- verbose = False
- if '-z' in sys.argv[1:]:
- dezlib = True
- sys.argv.remove('-z')
- else:
- dezlib = False
-
- adfile = AppleDouble(open(sys.argv[1]), verbose=verbose, dezlib=dezlib)
- for k, v in adfile.xattrs.items():
- print "%s: %s" % (k, v)
-
-if __name__ == '__main__':
- _test()
Copied: CalendarServer/trunk/txdav/base/propertystore/appledouble_xattr.py (from rev 7769, CalendarServer/branches/users/glyph/xattrs-from-files/txdav/base/propertystore/appledouble_xattr.py)
===================================================================
--- CalendarServer/trunk/txdav/base/propertystore/appledouble_xattr.py (rev 0)
+++ CalendarServer/trunk/txdav/base/propertystore/appledouble_xattr.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -0,0 +1,188 @@
+# -*- test-case-name: txdav.base.propertystore.test.test_appledouble -*- ##
+##
+# Copyright (c) 2011 Apple Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+##
+
+
+import struct
+
+from txdav.base.propertystore.xattr import PropertyStore as XattrPropStore
+
+# A lot of this is copied from python/plat-mac/applesingle.py,
+# with data structure information taken from
+# http://www.opensource.apple.com/source/Libc/Libc-391/darwin/copyfile.c
+
+# File header format: magic, version, unused, number of entries
+AS_HEADER_FORMAT=">LL16sh"
+AS_HEADER_LENGTH=26
+
+# The flag words for AppleDouble
+AS_MAGIC=0x00051607
+AS_VERSION=0x00020000
+
+# Entry header format: id, offset, length
+AS_ENTRY_FORMAT=">lll"
+AS_ENTRY_LENGTH=12
+
+# The id values
+AS_DATAFORK=1
+AS_RESOURCEFORK=2
+AS_REALNAME=3
+AS_COMMENT=4
+AS_ICONBW=5
+AS_ICONCOLOR=6
+AS_DATESINFO=8
+AS_FINDERINFO=9
+AS_MACFILEINFO=10
+AS_PRODOSFILEINFO=11
+AS_MSDOSFILEINFO=12
+AS_SHORTNAME=13
+AS_AFPFILEINFO=14
+AS_DIECTORYID=15
+
+FINDER_INFO_LENGTH = 32
+XATTR_OFFSET = FINDER_INFO_LENGTH + 2
+
+XATTR_HDR_MAGIC = 0x41545452 # ATTR
+XATTR_HEADER = ">llllllllhh"
+XATTR_HEADER_LENGTH = struct.calcsize(XATTR_HEADER)
+XATTR_ENTRY = ">llhb"
+XATTR_ENTRY_LENGTH = struct.calcsize(XATTR_ENTRY)
+
+
+def attrsFromFile(fileobj, debugFile=None):
+ """
+ Parse the extended attributes from a file.
+ """
+
+ attrs = {}
+
+ # Get the top-level header
+ header = fileobj.read(AS_HEADER_LENGTH)
+ try:
+ magic, version, _ignore, nentry = struct.unpack(
+ AS_HEADER_FORMAT, header
+ )
+ except ValueError, arg:
+ raise ValueError("Unpack header error: %s" % (arg,))
+ if debugFile is not None:
+ debugFile.write('Magic: 0x%8.8x\n' % (magic,))
+ debugFile.write('Version: 0x%8.8x\n' % (version,))
+ debugFile.write('Entries: %d\n' % (nentry,))
+ if magic != AS_MAGIC:
+ raise ValueError(
+ "Unknown AppleDouble magic number 0x%8.8x" % (magic,)
+ )
+ if version != AS_VERSION:
+ raise ValueError(
+ "Unknown AppleDouble version number 0x%8.8x" % (version,)
+ )
+ if nentry <= 0:
+ raise ValueError("AppleDouble file contains no forks")
+
+ # Get each entry
+ headers = [fileobj.read(AS_ENTRY_LENGTH) for _ignore in xrange(nentry)]
+ for hdr in headers:
+ try:
+ restype, offset, length = struct.unpack(AS_ENTRY_FORMAT, hdr)
+ except ValueError, arg:
+ raise ValueError("Unpack entry error: %s" % (arg,))
+ if debugFile is not None:
+ debugFile.write("\n-- Fork %d, offset %d, length %d\n" %
+ (restype, offset, length))
+
+ # Look for the FINDERINFO entry with extra bits
+ if restype == AS_FINDERINFO and length > FINDER_INFO_LENGTH:
+
+ # Get the xattr header
+ fileobj.seek(offset + XATTR_OFFSET)
+ data = fileobj.read(length - XATTR_OFFSET)
+ if len(data) != length-XATTR_OFFSET:
+ raise ValueError("Short read: expected %d bytes got %d" %
+ (length-XATTR_OFFSET, len(data)))
+ magic, _ignore_tag, total_size, data_start, data_length, \
+ _ignore_reserved1, _ignore_reserved2, _ignore_reserved3, \
+ flags, num_attrs = struct.unpack(XATTR_HEADER,
+ data[:XATTR_HEADER_LENGTH])
+ if magic != XATTR_HDR_MAGIC:
+ raise ValueError("No xattrs found")
+
+ if debugFile is not None:
+ debugFile.write("\n Xattr Header\n")
+ debugFile.write(' Magic: 0x%08X\n' % (magic,))
+ debugFile.write(' Total Size: %d\n' % (total_size,))
+ debugFile.write(' Data Start: 0x%02X\n' % (data_start,))
+ debugFile.write(' Data Length: %d\n' % (data_length,))
+ debugFile.write(' Flags: 0x%02X\n' % (flags,))
+ debugFile.write(' Number: %d\n' % (num_attrs,))
+
+ # Get each xattr entry
+ data = data[XATTR_HEADER_LENGTH:]
+ for _ignore in xrange(num_attrs):
+ [xattr_offset, xattr_length,
+ xattr_flags, xattr_name_len] = struct.unpack(
+ XATTR_ENTRY, data[:XATTR_ENTRY_LENGTH]
+ )
+ xattr_name = data[
+ XATTR_ENTRY_LENGTH:
+ XATTR_ENTRY_LENGTH + xattr_name_len
+ -1 # strip NULL terminator
+ ]
+ fileobj.seek(xattr_offset)
+ xattr_value = fileobj.read(xattr_length)
+
+ if debugFile is not None:
+ debugFile.write("\n Xattr Entry\n")
+ debugFile.write(' Offset: 0x%02X\n' %
+ (xattr_offset,))
+ debugFile.write(' Length: %d\n' %
+ (xattr_length,))
+ debugFile.write(' Flags: 0x%02X\n' %
+ (xattr_flags,))
+ debugFile.write(' Name: %s\n' % (xattr_name,))
+ debugFile.write(' Value: %s\n' %
+ (xattr_value,))
+ attrs[xattr_name] = xattr_value
+
+ # Skip over entry taking padding into account
+ advance = (XATTR_ENTRY_LENGTH + xattr_name_len + 3) & ~3
+ data = data[advance:]
+ return attrs
+
+
+
+class PropertyStore(XattrPropStore):
+ """
+ A property store that will read extended attributes from AppleDouble data in
+ "._"-prefixed files alongside their data.
+
+ Note that this is read-only but will not attempt to prevent write operations
+ (since some operations attempt to transparently upgrade things by writing
+ back properties, and we don't want to disrupt that operation). Its only
+ real purpose is to facilitate importing data with the file->database
+ migration code.
+ """
+
+ # In this case, this prefix will _always_ be the MacOS prefix. If you're
+ # using this on some other OS, then you're importing some data from MacOS;
+ # no Linux or BSD 'tar' will create AppleDouble files in the first place.
+
+ deadPropertyXattrPrefix = "WebDAV:"
+
+ @property
+ def attrs(self):
+ return attrsFromFile(self.path.sibling("._"+self.path.basename()).open())
+
+
Copied: CalendarServer/trunk/txdav/base/propertystore/test/test_appledouble.py (from rev 7769, CalendarServer/branches/users/glyph/xattrs-from-files/txdav/base/propertystore/test/test_appledouble.py)
===================================================================
--- CalendarServer/trunk/txdav/base/propertystore/test/test_appledouble.py (rev 0)
+++ CalendarServer/trunk/txdav/base/propertystore/test/test_appledouble.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -0,0 +1,121 @@
+##
+# Copyright (c) 2011 Apple Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+##
+
+"""
+Tests for txdav.base.propertystore.appledouble_xattr
+"""
+
+from twisted.trial.unittest import TestCase
+
+from tarfile import TarFile
+from cStringIO import StringIO
+
+from txdav.base.propertystore.appledouble_xattr import (
+ attrsFromFile, PropertyStore
+)
+
+from txdav.base.propertystore.base import PropertyName
+from twisted.python.filepath import FilePath
+from twext.web2.dav.element.rfc2518 import GETContentType, HRef, Depth
+
+# This tar file contains a single file, 'f', with 2 xattrs; 'alpha' with
+# contents 'beta', and 'gamma' with contents 'delta'.
+
+simpleTarWithXattrs = """
+H4sICDvqGk4AA2YudGFyAO2WvU7DMBSFrYofEYmBhdlPkPraTpwMGRBLkUAg2gGmyrShjUhLVFIo
+G4/CwMLIy/A82GlCS6RSEAo/wp90Fec4sa4tnZzY9SM5aYSyG47qdvscVQEhxOUcI5GBEeEMGMVa
+zwBBMIBLXMd1CDBMgDucIzyppJsS46tUjlQrvfg26b/znEySOGz35KL56Vbw6/WPAAKPo25AfZ96
+VBCwKMGdNBqEATAgzHMFF1qT85rr+xbluLnb2Ns/tbvhdUAZ9zw151gUCj0aXgaCCY87FniFOIyj
+4UUA1k/v2zDFrsz1Mwr/36nx1vPTZtn/1IGS/xlxXYRJxX1l/HP/o9XtdVRD6EB28GETn+AcraEN
+VVTVjSp9//ixJXdareN8qN+4z1eap5brKwityTjpF8f6oDtSWk8OBpl2FqYqm+J04bkbvsQs/av7
+CizPf1rOfxAm/7+F5flPP5P/xKJsPv8BOOeMU/MH8EupOvs1y/Jf++Wt/6lDuMl/g8FgqJIXLvCv
+wAASAAA=
+""".decode('base64')
+
+
+class DecoderTests(TestCase):
+ """
+ Tests for decoding extended attributes from AppleDouble format.
+ """
+
+ def test_attrsFromFile(self):
+ """
+ Extracting a simple AppleDouble file representing some extended
+ attributes should result in a dictionary of those attributes.
+ """
+ tarfile = TarFile.gzopen('sample.tgz',
+ fileobj=StringIO(simpleTarWithXattrs))
+ self.assertEqual(attrsFromFile(tarfile.extractfile("./._f")),
+ {"alpha": "beta", "gamma": "delta"})
+
+
+# The following tar file was also created with a single file, but it was
+# assigned a few bogus properties first, with a python program something like
+# this:
+
+# ps = PropertyStore("bob", lambda : FilePath("tmp/f"))
+# ps[PropertyName.fromElement(HRef)] = HRef("http://sample.example.com/")
+# ps[PropertyName.fromElement(Depth)] = Depth("1")
+# ps[PropertyName.fromElement(GETContentType)] = GETContentType("text/example")
+# ps.flush()
+
+
+samplePropertyTar = """
+H4sICJ/5Gk4AA2YudGFyAO3W3W4SQRQA4EFjiMTGqNErTUavoAjMz/5hJLHRRLZYwVKsRpNmhZWi
+lVLcVhpj0gvvfAVfQBOiFSvxAUyKAhW1mnijiXe+gb3RXSilkCjlAn/i+ZKd3T0zTGYze/bg9UW0
+XFDXEnrW5524ivqBECIJAkZyHUZE4JQzbMXrqEwwpRKRRJmL5gBCBVFiCOf6spoOszcNLWsuJTk1
+n5n8xTgtk5nSJ5Laz/obj4I3zv8IKuPZVCLA/H6mMJlQByM4bqRu6AHKKeF+IimyFdM2x6jAHUzA
+0ZNB9cxFb0KfCzAuKIrZJzoYbcZT6emAzBXiFx1UaQbTU6n09QB1/OnnBg3evmV9SzP/F8zrPe8X
+BzrzX+ZyR/5zSkWESZ/XVfef5z/accCOtiE0osVxOIov4HVWDO00D4aQLW+ezXvbp61NOTQ2Ntq4
+qv/is3l87xiyfT0eRGjfuH7l1ND5Y7et5k5CzxiNbbA5zeYyQgfb+pO6EZ9OG3raMOYzen3cXbO5
+hNDetnGTWb3+WufuFxYfLVfOOUOewWp5pRCT3/HIeLASqoZL5VAxtntg5u1SpFj88kRVPc5SBD2s
+uJ/Fbs3sKowW+dOa/d5HG/p6ZP9wD/OEPWp1xa1W3K4HR4MxdO3lxoRuNe+shfPBsmdZXaqdjqXs
+iB8/XNr63CXn8EgUDbaW+Mrpcj1+88F31ppyNZRfeF369nxNWsOrc+jEi0OlrnvVqv79+wp0r/+s
+s/4zCvX/t+he/wWhh/pPHIxvrv+UCoJobjD8A/hL9bv2W7rVfytf2vOfiYxB/QcAAAAAAAAAAAAA
+AAAAAACgVz8Aa/aaaQAoAAA=
+""".decode('base64')
+
+
+class PropertyStoreTests(TestCase):
+ """
+ Tests for decoding WebDAV properties.
+ """
+
+ def test_propertiesFromTarball(self):
+ """
+ Extracting a tarball with included AppleDouble WebDAV property
+ information should allow properties to be retrieved using
+ L{PropertyStore}.
+ """
+ tf = TarFile.gzopen('sample.tgz', fileobj=StringIO(samplePropertyTar))
+ tmpdir = self.mktemp()
+
+ # Note that 'tarfile' doesn't know anything about xattrs, so while OS
+ # X's 'tar' will restore these as actual xattrs, the 'tarfile' module
+ # will drop "._" parallel files into the directory structure on all
+ # platforms.
+ tf.extractall(tmpdir)
+
+ props = PropertyStore("bob", lambda : FilePath(tmpdir).child('f'))
+
+ self.assertEqual(props[PropertyName.fromElement(HRef)],
+ HRef("http://sample.example.com/"))
+ self.assertEqual(props[PropertyName.fromElement(Depth)],
+ Depth("1"))
+ self.assertEqual(props[PropertyName.fromElement(GETContentType)],
+ GETContentType("text/example"))
+
+
Modified: CalendarServer/trunk/txdav/base/propertystore/xattr.py
===================================================================
--- CalendarServer/trunk/txdav/base/propertystore/xattr.py 2011-07-11 22:03:37 UTC (rev 7769)
+++ CalendarServer/trunk/txdav/base/propertystore/xattr.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -1,4 +1,4 @@
-# -*- test-case-name: txdav.base.propertystore.test.test_xattr,txdav.caldav.datastore,txdav.carddav.datastore -*-
+# -*- test-case-name: txdav.base.propertystore.test.test_xattr -*-
##
# Copyright (c) 2010 Apple Inc. All rights reserved.
#
@@ -34,22 +34,20 @@
from twext.web2.dav.davxml import WebDAVDocument
-from txdav.base.propertystore.base import AbstractPropertyStore, PropertyName, validKey
+from txdav.base.propertystore.base import AbstractPropertyStore, PropertyName,\
+ validKey
from txdav.idav import PropertyStoreError
from twisted.python.reflect import namedAny
-
#
-# RFC 2518 Section 12.13.1 says that removal of non-existing property
-# is not an error. python-xattr on Linux fails with ENODATA in this
-# case. On OS X, the xattr library fails with ENOATTR, which CPython
-# does not expose. Its value is 93.
+# RFC 2518 Section 12.13.1 says that removal of non-existing property is not an
+# error. python-xattr on Linux fails with ENODATA in this case. On OS X, the
+# xattr library fails with ENOATTR, which some versions of CPython do not
+# expose. Its value is 93.
#
+
if sys.platform is "darwin":
- if hasattr(errno, "ENOATTR"):
- _ERRNO_NO_ATTR = errno.ENOATTR
- else:
- _ERRNO_NO_ATTR = 93
+ _ERRNO_NO_ATTR = getattr(errno, "ENOATTR", 93)
else:
_ERRNO_NO_ATTR = errno.ENODATA
@@ -68,9 +66,9 @@
"twext.web2.dav.xattrprops.xattrPropertyStore.deadPropertyXattrPrefix"
)
- # There is a 127 character limit for xattr keys so we need to compress/expand
- # overly long namespaces to help stay under that limit now that GUIDs are also
- # encoded in the keys.
+ # There is a 127 character limit for xattr keys so we need to
+ # compress/expand overly long namespaces to help stay under that limit now
+ # that GUIDs are also encoded in the keys.
_namespaceCompress = {
"urn:ietf:params:xml:ns:caldav" :"CALDAV:",
"urn:ietf:params:xml:ns:carddav" :"CARDDAV:",
@@ -79,13 +77,17 @@
"http://twistedmatrix.com/xml_namespace/dav/" :"TD:",
"http://twistedmatrix.com/xml_namespace/dav/private/" :"TDP:",
}
- _namespaceExpand = dict([ (v, k) for k, v in _namespaceCompress.iteritems() ])
+ _namespaceExpand = dict(
+ [ (v, k) for k, v in _namespaceCompress.iteritems() ]
+ )
+
def __init__(self, defaultuser, pathFactory):
"""
Initialize a L{PropertyStore}.
- @param pathFactory: a 0-arg callable that returns the L{CachingFilePath} to set extended attributes on.
+ @param pathFactory: a 0-arg callable that returns the L{CachingFilePath}
+ to set extended attributes on.
"""
super(PropertyStore, self).__init__(defaultuser)
@@ -109,7 +111,11 @@
def _encodeKey(self, effective, compressNamespace=True):
qname, uid = effective
- namespace = self._namespaceCompress.get(qname.namespace, qname.namespace) if compressNamespace else qname.namespace
+ if compressNamespace:
+ namespace = self._namespaceCompress.get(qname.namespace,
+ qname.namespace)
+ else:
+ namespace = qname.namespace
result = urllib.quote("{%s}%s" % (namespace, qname.name), safe="{}:")
if uid and uid != self._defaultUser:
result = uid + result
@@ -160,17 +166,20 @@
# Check for uncompressed namespace
if effectiveKey[0].namespace in self._namespaceCompress:
try:
- data = self.attrs[self._encodeKey(effectiveKey, compressNamespace=False)]
+ data = self.attrs[self._encodeKey(effectiveKey,
+ compressNamespace=False)]
except IOError, e:
raise KeyError(key)
try:
# Write it back using the compressed format
self.attrs[self._encodeKey(effectiveKey)] = data
- del self.attrs[self._encodeKey(effectiveKey, compressNamespace=False)]
+ del self.attrs[self._encodeKey(effectiveKey,
+ compressNamespace=False)]
except IOError, e:
- msg = "Unable to upgrade property to compressed namespace: %s" % (
- key.toString()
+ msg = (
+ "Unable to upgrade property "
+ "to compressed namespace: %s" % (key.toString())
)
self.log_error(msg)
raise PropertyStoreError(msg)
@@ -258,7 +267,8 @@
yield effectivekey[0]
def _removeResource(self):
- # xattrs are removed when the underlying file is deleted so just clear out cached changes
+ # xattrs are removed when the underlying file is deleted so just clear
+ # out cached changes
self.removed.clear()
self.modified.clear()
Modified: CalendarServer/trunk/txdav/caldav/datastore/file.py
===================================================================
--- CalendarServer/trunk/txdav/caldav/datastore/file.py 2011-07-11 22:03:37 UTC (rev 7769)
+++ CalendarServer/trunk/txdav/caldav/datastore/file.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -36,8 +36,6 @@
from twisted.python.failure import Failure
-from txdav.base.propertystore.xattr import PropertyStore
-
from twext.python.vcomponent import VComponent
from twext.web2.dav import davxml
from twext.web2.dav.element.rfc2518 import ResourceType, GETContentType
@@ -666,8 +664,10 @@
def properties(self):
- uid = self._calendarObject._parentCollection._home.uid()
- return PropertyStore(uid, lambda: self._path)
+ home = self._calendarObject._parentCollection._home
+ uid = home.uid()
+ propStoreClass = home._dataStore._propertyStoreClass
+ return propStoreClass(uid, lambda: self._path)
def store(self, contentType):
Modified: CalendarServer/trunk/txdav/common/datastore/file.py
===================================================================
--- CalendarServer/trunk/txdav/common/datastore/file.py 2011-07-11 22:03:37 UTC (rev 7769)
+++ CalendarServer/trunk/txdav/common/datastore/file.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -50,7 +50,7 @@
from txdav.base.propertystore.base import PropertyName
from txdav.base.propertystore.none import PropertyStore as NonePropertyStore
-from txdav.base.propertystore.xattr import PropertyStore
+from txdav.base.propertystore.xattr import PropertyStore as XattrPropertyStore
from errno import EEXIST, ENOENT
from zope.interface import implements, directlyProvides
@@ -85,11 +85,18 @@
for storing attachments, or C{None} if quota should not be enforced.
@type quota: C{int} or C{NoneType}
+
+ @ivar _propertyStoreClass: The class (or callable object / factory) that
+ produces an L{IPropertyStore} provider for a path. This has the
+ signature of the L{XattrPropertyStore} type: take 2 arguments
+ C{(default-user-uid, path-factory)}, return an L{IPropertyStore}
+ provider.
"""
implements(ICalendarStore)
def __init__(self, path, notifierFactory, enableCalendars=True,
- enableAddressBooks=True, quota=(2 ** 20)):
+ enableAddressBooks=True, quota=(2 ** 20),
+ propertyStoreClass=XattrPropertyStore):
"""
Create a store.
@@ -102,6 +109,7 @@
self.enableAddressBooks = enableAddressBooks
self._notifierFactory = notifierFactory
self._transactionClass = CommonStoreTransaction
+ self._propertyStoreClass = propertyStoreClass
self.quota = quota
@@ -488,7 +496,9 @@
# FIXME: needs tests for actual functionality
# FIXME: needs to be cached
# FIXME: transaction tests
- props = PropertyStore(self.uid(), lambda : self._path)
+ props = self._dataStore._propertyStoreClass(
+ self.uid(), lambda : self._path
+ )
self._transaction.addOperation(props.flush, "flush home properties")
return props
@@ -840,7 +850,8 @@
def properties(self):
# FIXME: needs direct tests - only covered by store tests
# FIXME: transactions
- props = PropertyStore(self._home.uid(), lambda: self._path)
+ propStoreClass = self._home._dataStore._propertyStoreClass
+ props = propStoreClass(self._home.uid(), lambda: self._path)
self.initPropertyStore(props)
self._transaction.addOperation(props.flush,
@@ -933,8 +944,13 @@
@cached
def properties(self):
- uid = self._parentCollection._home.uid()
- props = PropertyStore(uid, lambda : self._path) if self._parentCollection.objectResourcesHaveProperties() else NonePropertyStore(uid)
+ home = self._parentCollection._home
+ uid = home.uid()
+ if self._parentCollection.objectResourcesHaveProperties():
+ propStoreClass = home._dataStore._propertyStoreClass
+ props = propStoreClass(uid, lambda : self._path)
+ else:
+ props = NonePropertyStore(uid)
self.initPropertyStore(props)
self._transaction.addOperation(props.flush, "object properties flush")
return props
Modified: CalendarServer/trunk/txdav/common/datastore/util.py
===================================================================
--- CalendarServer/trunk/txdav/common/datastore/util.py 2011-07-11 22:03:37 UTC (rev 7769)
+++ CalendarServer/trunk/txdav/common/datastore/util.py 2011-07-11 22:08:29 UTC (rev 7770)
@@ -15,16 +15,29 @@
# limitations under the License.
##
+"""
+Utilities, mostly related to upgrading, common to calendar and addresbook
+data stores.
+"""
+
+import os
+import re
+import errno
+import xattr
+
from twext.python.log import LoggingMixIn
from twisted.application.service import Service
from twisted.internet import reactor
from twisted.internet.defer import inlineCallbacks
from twisted.python.modules import getModule
+from twisted.python.runtime import platform
+
from txdav.caldav.datastore.util import migrateHome as migrateCalendarHome
from txdav.carddav.datastore.util import migrateHome as migrateAddressbookHome
from txdav.common.datastore.file import CommonDataStore as FileStore, TOPPATHS
-import os
-import re
+from txdav.base.propertystore.xattr import PropertyStore as XattrPropertyStore
+from txdav.base.propertystore.appledouble_xattr import (
+ PropertyStore as AppleDoubleStore)
class UpgradeToDatabaseService(Service, LoggingMixIn, object):
@@ -38,7 +51,8 @@
Create an L{UpgradeToDatabaseService} if there are still file-based
calendar or addressbook homes remaining in the given path.
- @param path: a path pointing at the document root.
+ @param path: a path pointing at the document root, where the file-based
+ data-store is located.
@type path: L{CachingFilePath}
@param service: the service to wrap. This service should be started
@@ -59,8 +73,39 @@
# not hard coded.
for homeType in TOPPATHS:
if path.child(homeType).exists():
+ if platform.isMacOSX():
+ appropriateStoreClass = XattrPropertyStore
+ else:
+ attrs = xattr.xattr(path.path)
+ try:
+ attrs.get('user.should-not-be-set')
+ except IOError, ioe:
+ if ioe.errno == errno.ENODATA:
+ # xattrs are supported and enabled on the filesystem
+ # where the calendar data lives. this takes some
+ # doing (you have to edit fstab), so this means
+ # we're trying to migrate some 2.x data from a
+ # previous linux installation.
+ appropriateStoreClass = XattrPropertyStore
+ elif ioe.errno == errno.EOPNOTSUPP:
+ # The operation wasn't supported. This is what will
+ # usually happen on a naively configured filesystem,
+ # so this means we're most likely trying to migrate
+ # some data from an untarred archive created on an
+ # OS X installation using xattrs.
+ appropriateStoreClass = AppleDoubleStore
+ else:
+ # No need to check for ENOENT and the like; we just
+ # checked above to make sure the parent exists.
+ # Other errors are not anticipated here, so fail
+ # fast.
+ raise
+
+ appropriateStoreClass = AppleDoubleStore
+
self = cls(
- FileStore(path, None, True, True),
+ FileStore(path, None, True, True,
+ propertyStoreClass=appropriateStoreClass),
store, service, uid=uid, gid=gid,
)
return self
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/calendarserver-changes/attachments/20110711/8ceb1228/attachment-0001.html>
More information about the calendarserver-changes
mailing list