[libdispatch-dev] libdispatch on VS 2008 and VS 2010 64bit

[Marius Zwicker]

Thanks for your reminder that there were still some warnings waiting to
be fixed ;)

Unfortunately I disagree with you in some aspects. I cannot see any real
benefit from using (u)intptr_t within libdispatch as in my eyes we do
not need a datatype scaling with the architecture but should prefer
another one promising a fixed range on all architectures. This should
increase portability while ensuring feature parity again. As such I
clame size_t the victim as there is no standard concerning its size -
only its minimum size. Bad enough it is used within the public api of
both kqueues and libdispatch. But it can be replaced by (u)int32_t in
most other cases within the implementation easily. Pointers is another
story of course, but those are already fixed within the xdispatch
windows port.

So why stay with a 32 bit datatype while compiling a 64bit binary? For
me the main reason to use 64bit is bigger memory and (some) calculations
running more efficiently. But libdispatch is using neither of them -
only applications built on top of libdispatch. As you wrote the
limitations of a 32bit variable will have no real impact on semaphore
counts and priorities. I claim they also work very well for the allowed
number of queues and the suspend/reference count. As such we can reduce
future efforts by using a fixed size datatype.

On Windows the InterlockedIncrement functions and their relatives
(except the pointer operations of course) will work on 32bit LONG
variables on all architectures - for working on LONGLONG variables on
64bit architectures we would have to use InterlockedIncrement64 and
friends. So all is safe when using the solution explained above.

So for me the remaining problem is the kqueue interface needing the
dispatch_source api to use unsigned long for compatibility and I welcome
any suggestions solving this. Still I consider it difficult to do public
API changes to support a single operating system (Windows) when
resulting in changes on various other operating systems. On Windows it
is sadly very difficult to provide the same sources as on unix platforms
(speaking e.g. of DISPATCH_SOURCE_TYPE_PROC,
DISPATCH_SOURCE_TYPE_SIGNAL, DISPATCH_SOURCE_TYPE_VNODE, ...) as well.
So for me the only practical solution seems to provide a slightly
different dispatch_source (and possibly kqueue) api on Windows,
something I am not really happy about.

 -- Marius

Am 13.06.2011 04:28, schrieb DrPizza:
> I may be wrong, but I believe libxdispatch needs further modifications 
> before it'll build reasonable 64-bit binaries, at least in VC++. There are 
> some bugs that definitely need fixing, and some warts that would be nice to 
> fix.
>
> Mac OS X and Linux are both, I believe, LP64 platforms. Win32 (er... Win64?) 
> isn't; if you want a 64-bit integer, you'll have to use the hideous "long 
> long" (though I would have to double check that even works; I know it works 
> for compiling C++, but VC++ has no C99 support, so I'm not sure what C mode 
> does) or __int64.
>
> To attain reasonable compatibility between the platforms, I believe we 
> should therefore change any location that currently uses long to use intptr_t, 
> and any location using unsigned long to use uintptr_t. There are possibly a 
> few exceptions; we can probably get away with using longs for semaphore counts, 
> because four billion is a lot, and long is still fine for queue priorities, as 
> there are only three/six.
>
> IMHO we should change anything that might reasonably be used for pointer-size 
> values. This has some API-level consequences; I think it would be good if we 
> changed:
> unsigned long dispatch_source_get_data(dispatch_source_t source);
> void dispatch_source_merge_data(dispatch_source_t source, unsigned long value);
>
> This change in turn has consequences in a few places within the codebase, 
> where longs and related types need to be changed to intptr_ts.
>
> It would be desirable in many regards to change the mask similarly, in:
> unsigned long dispatch_source_get_mask(dispatch_source_t source);
> dispatch_source_t dispatch_source_create(dispatch_source_type_t type,
>                                          uintptr_t handle,
>                                          unsigned long mask,
>                                          dispatch_queue_t queue);
>
> However, this is a problem, because struct kevent uses an int32 for the mask 
> (libdispatch stores the mask in the fflags member, which has type u_int). So, 
> for that matter, does kevent64_s. Two thoughts occur. One, switch to 
> kevent64_s and store the mask in ext[0]. Two, stop using long for the mask 
> value, so that the interface becomes honest. We can't use it as a long, so 
> stop pretending that we can.
>
> There are also repercussions for atomic operations. VC++'s atomic operations 
> are sadly not as convenient as gcc's overloaded ones. If this were a C++ 
> project then that would of course not be a problem, but it isn't, so it is. 
> Generally speaking, VC++ has two sets of atomic operations; 32-bit ones, that 
> are defined in terms of LONG, and 64-bit ones, that use LONGLONG or similar. 
> These are plain old C functions--no overloads--so have different names.
>
> The functions (well, macros) for which this is most relevant are:
> dispatch_atomic_add(p, v)
> dispatch_atomic_sub(p, v)
>
> In 32-bit builds, they use InterlockedExchangeAdd--which works on longs--in 
> 64-bit builds, InterlockedExchangeAdd--which works on __int64s. To be safe, 
> 64-bit builds must, therefore, use __int64 operands. Otherwise, 32-bit overflow 
> will cause the subsequent 32-bit words to be corrupted. There may also be 
> alignment issues.
>
> The operands are, however, uint32_ts (dispatch_queue_s::dq_running), unsigned 
> longs (dispatch_source_s::ds_pending_data), or unsigned ints (
> dispatch_object_s::do_suspend_cnt).
>
> dispatch_source_s::ds_pending_data is also used with a dispatch_atomic_xchg 
> call, so if it is changed to 64-bit (to make the dispatch_atomic_add safe) then 
> dispatch_atomic_xchg would likewise have to be altered to use 64-bit integers.
>
> dispatch_source_s::ds_pending_data is the only one likely to ever cause a 
> problem, of course, as it's the only one that might reasonable wrap around 
> with 32-bit values.
>
> There are also a few pointer trunctions (probably harmless, e.g. the casts to 
> long in _dispatch_continuation_pop).
>
> Ultimately I think there are two options. The slightly easier one is to ensure 
> that VC++ builds only use 32-bit atomic operations and include a note somewhere 
> that while 64-bit Linux or Mac OS X builds can use 64-bit values in some 
> places, this won't be possible in Windows.
>
> My preference would be to go an alternative route: replace longs with intptr_ts 
> and ensure that atomic operations are working on the right data types. If it's 
> desirable to keep suspend/reference counts as 32-bit integers, then there'll 
> need to be both 64- and 32-bit atomic arithmetic; I think it's easier to just 
> make them use pointer-sized integers too. That's what my fork does.
>
>> -----Original Message-----
>> From: libdispatch-dev-bounces at lists.macosforge.org [mailto:libdispatch-dev-
>> bounces at lists.macosforge.org] On Behalf Of Marius Zwicker
>> Sent: 26 May 2011 21:02
>> To: libdispatch-dev at lists.macosforge.org
>> Subject: [libdispatch-dev] libdispatch on VS 2008 and VS 2010 64bit
>>
>> Hi everyone,
>>
>> Just wanted to inform you that as of today the trunk of the xdispatch project
>> is supporting building with Visual Studio 2008 and Visual Studio
>> 2010 64bit as well. Although VS 2008 is of limited use as we cannot provide
>> blocks/closures/lambdas on this platform.
>>
>> With this milestone I consider libdispatch (and xdispatch/QtDispatch of
>> course) production ready on Linux 32bit/64bit, Windows 32bit/64bit (using
>> VS2008 or VS2010) and on Mac OS 64bit.
>>
>> As always I welcome any comments.
>>
>> Regards,
>> Marius
>> _______________________________________________
>> libdispatch-dev mailing list
>> libdispatch-dev at lists.macosforge.org
>> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev