From mark at heily.com  Mon Jun  7 19:02:53 2010
From: mark at heily.com (Mark Heily)
Date: Mon, 07 Jun 2010 22:02:53 -0400
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
Message-ID: <4C0DA4CD.2060001@heily.com>

I have been able to compile libdispatch on Linux with a blocks-enabled clang 
and ran into some issues. The biggest problem is <unistd.h> in the GNU libc 
declares a function which takes an argument named "__block" which causes an 
error when compiling with Clang. This has been reported as a bug to GNU and 
rejected by drepper@ as a WONTFIX bug.

I'll submit some patches soon to address these issues, but see below for a 
list of what I encountered and the "fix" used to get it to work.

Regards,

  - Mark



Build environment:

   clang version 1.1 (branches/release_27)
   Target: x86_64-pc-linux-gnu

Build Commands:

   cd ~/src/libdispatch	
   sh ./autogen.sh && CC=clang CFLAGS=-fPIC ./configure && make clean && make

Problems:

[1] __block used within unistd.h declaration

In file included from ./internal.h:35:
In file included from ../dispatch/dispatch.h:36:
/usr/include/unistd.h:1128:35: error: __block attribute not allowed, only 
allowed on local
       variables
extern void encrypt (char *__block, int __edflag) __THROW __nonnull ((1));
                                   ^

	fix: edit unistd.h and remove the __block identifier

	see: http://sources.redhat.com/bugzilla/show_bug.cgi?id=11157


[2] two private_extern variables cause compile failures.

/usr/bin/ld: .libs/libdispatch_la-queue_kevent.o: relocation R_X86_64_PC32 
against symbol `_dispatch_safe_fork' can not be used when making a shared 
object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value

	fix: remove __private_extern__ from variable declarations


[3] clang wants arc4random defined

	fix: add u_int32_t arc4random(); to source

[4] clang doesn't like __unused

dispatch_after.c:32:21: error: expected ')'
void done(void *arg __unused) {
                     ^
dispatch_after.c:32:10: note: to match this '('
void done(void *arg __unused) {
          ^

	fix: delete this attribute

[5] fabs() undefined symbol

dispatch_drift.o: In function `__main_block_invoke_':
dispatch_drift.c:(.text+0x529): undefined reference to `fabs'

	fix: add -lm to testing/Makefile.am


[6] fgetln() not declared

summarize.c:69:8: warning: implicit declaration of function 'fgetln' is 
invalid in C99
       [-Wimplicit-function-declaration]
                 ln = fgetln(stdin, &len);

	fix: add declaration: char *fgetln(FILE *fp, size_t *lenp);

[7] Dispatch Source Read test failed

==================================================
[TEST] Dispatch Source Read
[PID] 8515
==================================================

	Actual: 0x603200
	Expected: 0x603200
[PASS] dispatch_get_main_queue
	Actual: 0x12dd010
	Expected: 0x12dd010
[PASS] DISPATCH_SOURCE_TYPE_READ
bytes available: 1
bytes read: 512000
^^^^^ hung here, process 8515 was defunct and I killed it manually
/bin/bash: line 5:  8515 Terminated              ${dir}$tst
FAIL: dispatch_read

	fix: none as of yet


From leimy2k at gmail.com  Mon Jun  7 19:16:25 2010
From: leimy2k at gmail.com (David Leimbach)
Date: Mon, 7 Jun 2010 19:16:25 -0700
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <4C0DA4CD.2060001@heily.com>
References: <4C0DA4CD.2060001@heily.com>
Message-ID: <AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>

 Oh well time to use a different libc?  I hear FreeBSD has a nice one!  :-)

On Monday, June 7, 2010, Mark Heily <mark at heily.com> wrote:
> I have been able to compile libdispatch on Linux with a blocks-enabled clang and ran into some issues. The biggest problem is <unistd.h> in the GNU libc declares a function which takes an argument named "__block" which causes an error when compiling with Clang. This has been reported as a bug to GNU and rejected by drepper@ as a WONTFIX bug.
>
> I'll submit some patches soon to address these issues, but see below for a list of what I encountered and the "fix" used to get it to work.
>
> Regards,
>
> ?- Mark
>
>
>
> Build environment:
>
>  ?clang version 1.1 (branches/release_27)
>  ?Target: x86_64-pc-linux-gnu
>
> Build Commands:
>
>  ?cd ~/src/libdispatch
>  ?sh ./autogen.sh && CC=clang CFLAGS=-fPIC ./configure && make clean && make
>
> Problems:
>
> [1] __block used within unistd.h declaration
>
> In file included from ./internal.h:35:
> In file included from ../dispatch/dispatch.h:36:
> /usr/include/unistd.h:1128:35: error: __block attribute not allowed, only allowed on local
>  ? ? ?variables
> extern void encrypt (char *__block, int __edflag) __THROW __nonnull ((1));
>  ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?^
>
>  ? ? ? ?fix: edit unistd.h and remove the __block identifier
>
>  ? ? ? ?see: http://sources.redhat.com/bugzilla/show_bug.cgi?id=11157
>
>
> [2] two private_extern variables cause compile failures.
>
> /usr/bin/ld: .libs/libdispatch_la-queue_kevent.o: relocation R_X86_64_PC32 against symbol `_dispatch_safe_fork' can not be used when making a shared object; recompile with -fPIC
> /usr/bin/ld: final link failed: Bad value
>
>  ? ? ? ?fix: remove __private_extern__ from variable declarations
>
>
> [3] clang wants arc4random defined
>
>  ? ? ? ?fix: add u_int32_t arc4random(); to source
>
> [4] clang doesn't like __unused
>
> dispatch_after.c:32:21: error: expected ')'
> void done(void *arg __unused) {
>  ? ? ? ? ? ? ? ? ? ?^
> dispatch_after.c:32:10: note: to match this '('
> void done(void *arg __unused) {
>  ? ? ? ? ^
>
>  ? ? ? ?fix: delete this attribute
>
> [5] fabs() undefined symbol
>
> dispatch_drift.o: In function `__main_block_invoke_':
> dispatch_drift.c:(.text+0x529): undefined reference to `fabs'
>
>  ? ? ? ?fix: add -lm to testing/Makefile.am
>
>
> [6] fgetln() not declared
>
> summarize.c:69:8: warning: implicit declaration of function 'fgetln' is invalid in C99
>  ? ? ?[-Wimplicit-function-declaration]
>  ? ? ? ? ? ? ? ?ln = fgetln(stdin, &len);
>
>  ? ? ? ?fix: add declaration: char *fgetln(FILE *fp, size_t *lenp);
>
> [7] Dispatch Source Read test failed
>
> ==================================================
> [TEST] Dispatch Source Read
> [PID] 8515
> ==================================================
>
>  ? ? ? ?Actual: 0x603200
>  ? ? ? ?Expected: 0x603200
> [PASS] dispatch_get_main_queue
>  ? ? ? ?Actual: 0x12dd010
>  ? ? ? ?Expected: 0x12dd010
> [PASS] DISPATCH_SOURCE_TYPE_READ
> bytes available: 1
> bytes read: 512000
> ^^^^^ hung here, process 8515 was defunct and I killed it manually
> /bin/bash: line 5: ?8515 Terminated ? ? ? ? ? ? ?${dir}$tst
> FAIL: dispatch_read
>
>  ? ? ? ?fix: none as of yet
>
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev
>

From mark at heily.com  Mon Jun  7 19:19:12 2010
From: mark at heily.com (Mark Heily)
Date: Mon, 07 Jun 2010 22:19:12 -0400
Subject: [libdispatch-dev] [PATCH] Do not build the unit tests by default
Message-ID: <4C0DA8A0.5010703@heily.com>

This patch will prevent the unit tests from being built when "make" is run 
from the top-level project directory. This is helpful when porting 
libdispatch to non-BSD platforms.

   - Mark

Index: Makefile.am
===================================================================
--- Makefile.am	(revision 188)
+++ Makefile.am	(working copy)
@@ -6,5 +6,6 @@
  SUBDIRS=		\
  	dispatch	\
  	man		\
-	src		\
-	testing
+	src		
+
+DIST_SUBDIRS=testing

From bonzini at gnu.org  Tue Jun  8 01:02:56 2010
From: bonzini at gnu.org (Paolo Bonzini)
Date: Tue, 08 Jun 2010 10:02:56 +0200
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <4C0DA4CD.2060001@heily.com>
References: <4C0DA4CD.2060001@heily.com>
Message-ID: <4C0DF930.5030603@gnu.org>

On 06/08/2010 04:02 AM, Mark Heily wrote:
> fix: edit unistd.h and remove the __block identifier

Or #define __block __glibc_block/#undef __block around the inclusion?

Something like the attached untested patch.

Paolo
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fff.patch
URL: <http://lists.macosforge.org/pipermail/libdispatch-dev/attachments/20100608/bb041786/attachment-0001.ksh>

From bonzini at gnu.org  Tue Jun  8 01:06:20 2010
From: bonzini at gnu.org (Paolo Bonzini)
Date: Tue, 08 Jun 2010 10:06:20 +0200
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
References: <4C0DA4CD.2060001@heily.com>
	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
Message-ID: <4C0DF9FC.3070107@gnu.org>

On 06/08/2010 04:16 AM, David Leimbach wrote:
> Oh well time to use a different libc?  I hear FreeBSD has a nice one!
> :-)

No, time to fix clang.  See my answer to

http://sources.redhat.com/bugzilla/show_bug.cgi?id=11157

Paolo

From david-m at orbotech.com  Tue Jun  8 05:53:12 2010
From: david-m at orbotech.com (David Minor)
Date: Tue, 8 Jun 2010 15:53:12 +0300
Subject: [libdispatch-dev] OpenCL
Message-ID: <FCB44A2146B78C479695CF9CCA7EEA870699E7DD@excg-isl01>

Is anybody working on the idea of getting OpenCL kernels to run under
libdispatch?  A kernel is semantically similar to a block.

I think it would be a big win to be able to do this.

 

David Minor 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/libdispatch-dev/attachments/20100608/62ce5ead/attachment.html>

From devlists at shadowlab.org  Tue Jun  8 06:00:25 2010
From: devlists at shadowlab.org (Jean-Daniel Dupas)
Date: Tue, 8 Jun 2010 15:00:25 +0200
Subject: [libdispatch-dev] OpenCL
In-Reply-To: <FCB44A2146B78C479695CF9CCA7EEA870699E7DD@excg-isl01>
References: <FCB44A2146B78C479695CF9CCA7EEA870699E7DD@excg-isl01>
Message-ID: <5EB79B06-BD2F-4BB5-8367-6368F90B5271@shadowlab.org>

AFAIK, that's already how it works on OS X:

From OpenCL Programming Guide:

?Apple?s implementation of OpenCL is built on top of Grand Central Dispatch (GCD).?

http://developer.apple.com/mac/library/documentation/Performance/Conceptual/OpenCL_MacProgGuide/OpenCLontheMacPlatform/OpenCLontheMacPlatform.html

Le 8 juin 2010 ? 14:53, David Minor a ?crit :

> Is anybody working on the idea of getting OpenCL kernels to run under libdispatch?  A kernel is semantically similar to a block.
> I think it would be a big win to be able to do this.
>  
> David Minor
>  
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev

-- Jean-Daniel





From leimy2k at gmail.com  Tue Jun  8 08:54:29 2010
From: leimy2k at gmail.com (David Leimbach)
Date: Tue, 8 Jun 2010 08:54:29 -0700
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <4C0DF9FC.3070107@gnu.org>
References: <4C0DA4CD.2060001@heily.com>
	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
	<4C0DF9FC.3070107@gnu.org>
Message-ID: <AANLkTikHXZ5qE_rSLhg2UMicNKrsJ8--qSzCcYPUS052@mail.gmail.com>

On Tue, Jun 8, 2010 at 1:06 AM, Paolo Bonzini <bonzini at gnu.org> wrote:

> On 06/08/2010 04:16 AM, David Leimbach wrote:
>
>> Oh well time to use a different libc?  I hear FreeBSD has a nice one!
>> :-)
>>
>
> No, time to fix clang.  See my answer to
>
>
> http://sources.redhat.com/bugzilla/show_bug.cgi?id=11157
>
> Paolo
>

It seems pretty cut and dry to me.

1. C99 defines that double underscored tokens are reserved identifiers in
the implementation of C99.
2. C99 is defined by both the compiler and the standard C libraries defined
in C99.
3. unistd.h is not part of C99.

Unistd.h is therefore clashing with a C99's totally valid usage of a double
underscored and reserved name.  Yes, unistd.h is shipping with a libc
implementation, but nowhere does it say in C99 that that's an excuse for
utilizing the C99 implementation's reserved namespace.

One could question whether it is a good idea or not to tell the users of
your C99 implementation to have to use extensions to the language via the
use of double underscored names, but it really does say in C99 that these
are there for ANY use by the implementation.  This means that if you see
double underscore'd identifiers in a user's code, that this code is at worst
"not portable to other compiler environments".

Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/libdispatch-dev/attachments/20100608/094c3042/attachment.html>

From bonzini at gnu.org  Tue Jun  8 09:12:30 2010
From: bonzini at gnu.org (Paolo Bonzini)
Date: Tue, 08 Jun 2010 18:12:30 +0200
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <AANLkTikHXZ5qE_rSLhg2UMicNKrsJ8--qSzCcYPUS052@mail.gmail.com>
References: <4C0DA4CD.2060001@heily.com>	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>	<4C0DF9FC.3070107@gnu.org>
	<AANLkTikHXZ5qE_rSLhg2UMicNKrsJ8--qSzCcYPUS052@mail.gmail.com>
Message-ID: <4C0E6BEE.3030401@gnu.org>

> It seems pretty cut and dry to me.
>
> 1. C99 defines that double underscored tokens are reserved identifiers
> in the implementation of C99.
> 2. C99 is defined by both the compiler and the standard C libraries
> defined in C99.
> 3. unistd.h is not part of C99.

You know perfectly that the same thing could have happened with __block 
was used in stdlib.h (libdispatch is using _GNU_SOURCE, so it is 
allowing explicitly to declare non-C99 things in stdlib.h).  So this is 
at best a strawman.

BTW, from your favorite libc's _ctype.h:

static __inline int
__istype(__ct_rune_t c, unsigned long _f)
{
	return (!!__maskrune(_c, _f));
}

__istype looks like a nice candidate for a new compiler keyword...

> Unistd.h is therefore clashing with a C99's totally valid usage of a
> double underscored and reserved name.  Yes, unistd.h is shipping with a
> libc implementation, but nowhere does it say in C99 that that's an
> excuse for utilizing the C99 implementation's reserved namespace.

The alternative is not using double underscores and breaking on anyone 
using "#define block blah".  What kind of conflict do you think is more 
common?  User macros or extended keywords?

> One could question whether it is a good idea or not to tell the users of
> your C99 implementation to have to use extensions to the language via
> the use of double underscored names, but it really does say in C99 that
> these are there for ANY use by the implementation.

True, but everything IMO is against clang in this circumstance:

1) GCC has followed the standard of using __keyword__, instead leaving 
__keyword to libc and libstdc++.  __complex__, __asm__, __attribute__, 
you name it.  No reason why clang should not have done the same for 
__block.  It wouldn't have clashed with glibc.

2) past experience has suggested a workaround, namely "fixincludes". 
Currently clang is being sloppy in this respect; it can afford that 
because it only supports a few targets (basically Linux and Mac OS X). 
It's a young project, so I don't have anything against that.  But sooner 
or later you will have to deal with the consequences of not controlling 
the whole C99/POSIX environment.

> This means that if
> you see double underscore'd identifiers in a user's code, that this code
> is at worst "not portable to other compiler environments".

libc is not user code.

Paolo

From leimy2k at gmail.com  Tue Jun  8 10:27:31 2010
From: leimy2k at gmail.com (David Leimbach)
Date: Tue, 8 Jun 2010 10:27:31 -0700
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <4C0E6BEE.3030401@gnu.org>
References: <4C0DA4CD.2060001@heily.com>
	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
	<4C0DF9FC.3070107@gnu.org>
	<AANLkTikHXZ5qE_rSLhg2UMicNKrsJ8--qSzCcYPUS052@mail.gmail.com>
	<4C0E6BEE.3030401@gnu.org>
Message-ID: <AANLkTintdsbffDieQaq2KDC-jU4r5UieVkfgvw38nkYr@mail.gmail.com>

On Tue, Jun 8, 2010 at 9:12 AM, Paolo Bonzini <bonzini at gnu.org> wrote:

> It seems pretty cut and dry to me.
>>
>> 1. C99 defines that double underscored tokens are reserved identifiers
>> in the implementation of C99.
>> 2. C99 is defined by both the compiler and the standard C libraries
>> defined in C99.
>> 3. unistd.h is not part of C99.
>>
>
> You know perfectly that the same thing could have happened with __block was
> used in stdlib.h (libdispatch is using _GNU_SOURCE, so it is allowing
> explicitly to declare non-C99 things in stdlib.h).  So this is at best a
> strawman.
>
>
This is not a strawman because it is not an argument.  I'm telling you that
by definition, C99 says double underscore names are for the implementation.
 It's also not arguable that unistd.h is not part of C99's standard library
by definition, and therefore had no business using double underscored names
to begin with.

If we were talking about stdlib.h, it would actually be a different story.
 I'm not the one who drew these seemingly arbitrary lines, but they exist
all the same.

It may very well be that the folks sitting on the committee sometimes are
out of touch with reality, and don't realize that C99 compilers don't always
ship with exactly their own libc, and that libc libraries often are scoped
to handle far more than just the C99 defined standard library, but the point
of the standard is try to keep things portable, and to lay down some rules
for users and implementors to collaborate and work together.

Someone's not following the rules, and it's not the clang or llvm teams.


> BTW, from your favorite libc's _ctype.h:
>
> static __inline int
> __istype(__ct_rune_t c, unsigned long _f)
> {
>        return (!!__maskrune(_c, _f));
> }
>
> __istype looks like a nice candidate for a new compiler keyword...
>
>
Unfortunately for your counter-example, ctype.h is part of C99, and
therefore a use of double underscored names is valid within that context as
far as I can tell.



>  Unistd.h is therefore clashing with a C99's totally valid usage of a
>> double underscored and reserved name.  Yes, unistd.h is shipping with a
>> libc implementation, but nowhere does it say in C99 that that's an
>> excuse for utilizing the C99 implementation's reserved namespace.
>>
>
> The alternative is not using double underscores and breaking on anyone
> using "#define block blah".  What kind of conflict do you think is more
> common?  User macros or extended keywords?
>
>
It's irrelvant what's more common if you're looking at this from the
perspective of someone trying to comply with C99.

If you're looking at this situation from the perspective of what changes
will affect the most users, then glibc still loses most likely because I bet
the number of people with Clang -fblock enabled code outnumbers the people
maintaining glibc and using the __block definition (which is illegal by C99)
from unistd.h.

I didn't draw the lines, so if they don't make sense, do not blame me, but
they're there to help people collaborate and write portable code.  Claiming
that Apple and the LLVM team is being a poor collaborator for using __block
is hard to maintain when that entire namespace was allotted to them by the
C99 standards committee.


>
>  One could question whether it is a good idea or not to tell the users of
>> your C99 implementation to have to use extensions to the language via
>> the use of double underscored names, but it really does say in C99 that
>> these are there for ANY use by the implementation.
>>
>
> True, but everything IMO is against clang in this circumstance:
>
> 1) GCC has followed the standard of using __keyword__, instead leaving
> __keyword to libc and libstdc++.  __complex__, __asm__, __attribute__, you
> name it.  No reason why clang should not have done the same for __block.  It
> wouldn't have clashed with glibc.
>
> I disagree with your claim that "everything is against clang", but you make
a valid point all the same.

GCC is a de-facto standard due to its ubiquity and longevity, and there's
some value in tracking this, as Intel has realized with their compiler
toolchain.  And you're correct that Clang could have done this for __block,
but the primary target platform was *not* glibc for this stuff.

Maybe it could easily be __block__ going forward, but don't you think that's
going to change a lot of people's libdispatch related code just to make
things compatible with unistd.h from glibc?  Isn't that worse than fixing
glibc's header to comply with C99?

As you say below, it's a young implementation, and possibly easier to change
that unistd.h.  All I know about the lack of desire to make the change in
unistd.h for glibc is that Ulrich Drepper doesn't want to do it, and
believes incorrectly that the Clang and LLVM develoeprs "stole" something
that he thought was his, when it really wasn't.


2) past experience has suggested a workaround, namely "fixincludes".
> Currently clang is being sloppy in this respect; it can afford that because
> it only supports a few targets (basically Linux and Mac OS X). It's a young
> project, so I don't have anything against that.  But sooner or later you
> will have to deal with the consequences of not controlling the whole
> C99/POSIX environment.


fixincludes was to deal with non-ANSI headers right?  How is that going to
help with the __block keyword?  I can't find the Block implementation text
that used to be at clang.llvm.org anymore as the site has moved stuff around
or deleted the file, but my impression is it's a keyword, not a macro.


>
>
>  This means that if
>> you see double underscore'd identifiers in a user's code, that this code
>> is at worst "not portable to other compiler environments".
>>
>
> libc is not user code.
>

>From one perspective you're correct, however unistd.h is a user of the
standard C library code, and should not have used double underscore
identifiers per the C99 standard.  They're ALL reserved.

Dave


>
> Paolo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/libdispatch-dev/attachments/20100608/cf925eca/attachment.html>

From jkh at apple.com  Tue Jun  8 11:40:40 2010
From: jkh at apple.com (Jordan K. Hubbard)
Date: Tue, 8 Jun 2010 11:40:40 -0700
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <4C0DF9FC.3070107@gnu.org>
References: <4C0DA4CD.2060001@heily.com>
	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
	<4C0DF9FC.3070107@gnu.org>
Message-ID: <0207DB73-BD88-45D5-9508-DCA76393C502@apple.com>

Or we could just come to the conclusion that the time is simply not yet right to add Linux to the supported OS list.

From what I can read between the lines on the discussion thread in that bug, there doesn't seem to be much interest in clang, or even in merging block support into whatever branch of gcc linux follows, and the ultimate goal of any porting work we do (or incorporate) is to see the technology adopted more along the lines of what FreeBSD is on schedule to do with their 8.1 release.  If there's already this much push-back on a simple header change then that tends to suggest that there will be even more insurmountable obstacles just around the corner as we move on to the stuff that's actually *hard*, like kernel support. :-)

My vote would be to focus on platforms who's user communities display an active interest in the GCD API.  We're not here to do a "hard sell" on it since that's just a good way to alienate folks who might someday, under different circumstances, be interested in what it has to offer.

Regards,

- Jordan

On Jun 8, 2010, at 1:06 AM, Paolo Bonzini wrote:

> On 06/08/2010 04:16 AM, David Leimbach wrote:
>> Oh well time to use a different libc?  I hear FreeBSD has a nice one!
>> :-)
> 
> No, time to fix clang.  See my answer to
> 
> http://sources.redhat.com/bugzilla/show_bug.cgi?id=11157
> 
> Paolo
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev


From bonzini at gnu.org  Tue Jun  8 11:49:09 2010
From: bonzini at gnu.org (Paolo Bonzini)
Date: Tue, 08 Jun 2010 20:49:09 +0200
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <AANLkTintdsbffDieQaq2KDC-jU4r5UieVkfgvw38nkYr@mail.gmail.com>
References: <4C0DA4CD.2060001@heily.com>	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>	<4C0DF9FC.3070107@gnu.org>	<AANLkTikHXZ5qE_rSLhg2UMicNKrsJ8--qSzCcYPUS052@mail.gmail.com>	<4C0E6BEE.3030401@gnu.org>
	<AANLkTintdsbffDieQaq2KDC-jU4r5UieVkfgvw38nkYr@mail.gmail.com>
Message-ID: <4C0E90A5.6000109@gnu.org>

On 06/08/2010 07:27 PM, David Leimbach wrote:
> This is not a strawman because it is not an argument.  I'm telling you
> that by definition, C99 says double underscore names are for the
> implementation.  It's also not arguable that unistd.h is not part of
> C99's standard library by definition, and therefore had no business
> using double underscored names to begin with.

Ok, it is not C99.  It's POSIX.  Come on.  You're "drawing the line" in 
the most unreasonable place.  Anyway.

>     BTW, from your favorite libc's _ctype.h:
>
>     static __inline int
>     __istype(__ct_rune_t c, unsigned long _f)
>     {
>             return (!!__maskrune(_c, _f));
>     }
>
>     __istype looks like a nice candidate for a new compiler keyword...
>
> Unfortunately for your counter-example, ctype.h is part of C99, and
> therefore a use of double underscored names is valid within that context
> as far as I can tell.

No, it is _ctype.h with the underscore.  Nowhere in that file I read 
that it is internal use only.  Anyway, 4 seconds of grep and here is 
this from FreeBSD pthread.h:

#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \
{ \
   struct _pthread_cleanup_info __cleanup_info__; \
   __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg, \
      &__cleanup_info__); \
   {

#define pthread_cleanup_pop(execute)
   } \
   __pthread_cleanup_pop_imp(execute); \
}

Tada, __cleanup_info__.  BTW pthread_cleanup_{push,pop} just _cannot_ be 
implemented without macros of this kind, so pthreads cannot be 
implemented without either following your interpretation of the C99 
standard, or breaking the user's namespace.

> Claiming that Apple and the LLVM team is being a poor collaborator for
> using __block is hard to maintain when that entire namespace was
> allotted to them by the C99 standards committee.

I never said that.  I said that the choice of __block without trailing 
underscores was poor.  Nobody's perfect, GCC did the same for __thread, 
but guess what, there is a fixincludes for that:

/*
  * __thread is now a keyword.
  */
fix = {
     hackname  = thread_keyword;
     files     = "pthread.h";
     files     = "bits/sigthread.h";
     select    = "([* ])__thread([,)])";
     c_fix     = format;
     c_fix_arg = "%1__thr%2";
};

 > As you say below, it's a young implementation, and possibly easier to
 > change that unistd.h.

No, it's set in stone now.  I'm not suggesting changing it.  I'm just 
suggesting that for the first time clang is showing the need for 
fixincludes.

> Maybe it could easily be __block__ going forward, but don't you think
> that's going to change a lot of people's libdispatch related code just
> to make things compatible with unistd.h from glibc?

I never proposed that.

> Isn't that worse
> than fixing glibc's header to comply with C99?

No need to say that again.  Repeating the same argument sounds like the 
preferred strategy of people writing to glibc bugzilla, just to see 
Drepper getting upset.  But we're not on glibc bugzilla, and I'm not 
Drepper.

>     2) past experience has suggested a workaround, namely "fixincludes".
>     Currently clang is being sloppy in this respect; it can afford that
>     because it only supports a few targets (basically Linux and Mac OS
>     X). It's a young project, so I don't have anything against that.
>       But sooner or later you will have to deal with the consequences of
>     not controlling the whole C99/POSIX environment.
>
> fixincludes was to deal with non-ANSI headers right?  How is that going
> to help with the __block keyword?

Nowadays fixincludes nowadays deals with all sort of problems.  It is 
even fixing old glibc to cope with removed extensions, or 
backwards-incompatible changes in the compiler (in addition to the above 
example with __thread, I can count two cases: C99 inline semantics, and 
removal of lvalue casts).  It can run arbitrary substitutions and sed 
scripts.  Here is another example:

/*
  *  Incorrect sprintf declaration in X11/Xmu.h
  */
fix = {
     hackname = x11_sprintf;
     files    = X11/Xmu.h;
     files    = X11/Xmu/Xmu.h;
     select   = "^extern char \\*\tsprintf\\(\\);$";

     c_fix     = format;
     c_fix_arg = "#ifndef __STDC__\n%0\n#endif /* !defined __STDC__ */";

     test_text = "extern char *\tsprintf();";
};

Paolo

From bonzini at gnu.org  Tue Jun  8 11:52:48 2010
From: bonzini at gnu.org (Paolo Bonzini)
Date: Tue, 08 Jun 2010 20:52:48 +0200
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <0207DB73-BD88-45D5-9508-DCA76393C502@apple.com>
References: <4C0DA4CD.2060001@heily.com>
	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
	<4C0DF9FC.3070107@gnu.org>
	<0207DB73-BD88-45D5-9508-DCA76393C502@apple.com>
Message-ID: <4C0E9180.4040705@gnu.org>

On 06/08/2010 08:40 PM, Jordan K. Hubbard wrote:
> Or we could just come to the conclusion that the time is simply not
> yet right to add Linux to the supported OS list.

I agree, but it's nice to have it build for people who want to experiment.

Have you seen the attached patch that fixes libdispatch?

Paolo

From leimy2k at gmail.com  Tue Jun  8 11:54:17 2010
From: leimy2k at gmail.com (David Leimbach)
Date: Tue, 8 Jun 2010 11:54:17 -0700
Subject: [libdispatch-dev] linux + libdispatch + clang + blocks
In-Reply-To: <4C0E90A5.6000109@gnu.org>
References: <4C0DA4CD.2060001@heily.com>
	<AANLkTimwhK6AZwLFynyJ421DMV4GaFLOpKB37KHgRzpI@mail.gmail.com>
	<4C0DF9FC.3070107@gnu.org>
	<AANLkTikHXZ5qE_rSLhg2UMicNKrsJ8--qSzCcYPUS052@mail.gmail.com>
	<4C0E6BEE.3030401@gnu.org>
	<AANLkTintdsbffDieQaq2KDC-jU4r5UieVkfgvw38nkYr@mail.gmail.com>
	<4C0E90A5.6000109@gnu.org>
Message-ID: <AANLkTikFGnH3OpH6H5apjLFJO2F9iHqhcDsr74ItYNOQ@mail.gmail.com>

Well I guess we don't agree at all :-).  Have a nice day.

On Tue, Jun 8, 2010 at 11:49 AM, Paolo Bonzini <bonzini at gnu.org> wrote:

> On 06/08/2010 07:27 PM, David Leimbach wrote:
>
>> This is not a strawman because it is not an argument.  I'm telling you
>> that by definition, C99 says double underscore names are for the
>> implementation.  It's also not arguable that unistd.h is not part of
>> C99's standard library by definition, and therefore had no business
>> using double underscored names to begin with.
>>
>
> Ok, it is not C99.  It's POSIX.  Come on.  You're "drawing the line" in the
> most unreasonable place.  Anyway.
>
>
>     BTW, from your favorite libc's _ctype.h:
>>
>>    static __inline int
>>    __istype(__ct_rune_t c, unsigned long _f)
>>    {
>>            return (!!__maskrune(_c, _f));
>>    }
>>
>>    __istype looks like a nice candidate for a new compiler keyword...
>>
>> Unfortunately for your counter-example, ctype.h is part of C99, and
>> therefore a use of double underscored names is valid within that context
>> as far as I can tell.
>>
>
> No, it is _ctype.h with the underscore.  Nowhere in that file I read that
> it is internal use only.  Anyway, 4 seconds of grep and here is this from
> FreeBSD pthread.h:
>
> #define pthread_cleanup_push(cleanup_routine, cleanup_arg) \
> { \
>  struct _pthread_cleanup_info __cleanup_info__; \
>  __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg, \
>     &__cleanup_info__); \
>  {
>
> #define pthread_cleanup_pop(execute)
>  } \
>  __pthread_cleanup_pop_imp(execute); \
> }
>
> Tada, __cleanup_info__.  BTW pthread_cleanup_{push,pop} just _cannot_ be
> implemented without macros of this kind, so pthreads cannot be implemented
> without either following your interpretation of the C99 standard, or
> breaking the user's namespace.
>
>
>  Claiming that Apple and the LLVM team is being a poor collaborator for
>> using __block is hard to maintain when that entire namespace was
>> allotted to them by the C99 standards committee.
>>
>
> I never said that.  I said that the choice of __block without trailing
> underscores was poor.  Nobody's perfect, GCC did the same for __thread, but
> guess what, there is a fixincludes for that:
>
> /*
>  * __thread is now a keyword.
>  */
> fix = {
>    hackname  = thread_keyword;
>    files     = "pthread.h";
>    files     = "bits/sigthread.h";
>    select    = "([* ])__thread([,)])";
>    c_fix     = format;
>    c_fix_arg = "%1__thr%2";
>
> };
>
> > As you say below, it's a young implementation, and possibly easier to
> > change that unistd.h.
>
> No, it's set in stone now.  I'm not suggesting changing it.  I'm just
> suggesting that for the first time clang is showing the need for
> fixincludes.
>
>
>  Maybe it could easily be __block__ going forward, but don't you think
>> that's going to change a lot of people's libdispatch related code just
>> to make things compatible with unistd.h from glibc?
>>
>
> I never proposed that.
>
>
>  Isn't that worse
>> than fixing glibc's header to comply with C99?
>>
>
> No need to say that again.  Repeating the same argument sounds like the
> preferred strategy of people writing to glibc bugzilla, just to see Drepper
> getting upset.  But we're not on glibc bugzilla, and I'm not Drepper.
>
>
>     2) past experience has suggested a workaround, namely "fixincludes".
>>    Currently clang is being sloppy in this respect; it can afford that
>>    because it only supports a few targets (basically Linux and Mac OS
>>    X). It's a young project, so I don't have anything against that.
>>      But sooner or later you will have to deal with the consequences of
>>    not controlling the whole C99/POSIX environment.
>>
>> fixincludes was to deal with non-ANSI headers right?  How is that going
>> to help with the __block keyword?
>>
>
> Nowadays fixincludes nowadays deals with all sort of problems.  It is even
> fixing old glibc to cope with removed extensions, or backwards-incompatible
> changes in the compiler (in addition to the above example with __thread, I
> can count two cases: C99 inline semantics, and removal of lvalue casts).  It
> can run arbitrary substitutions and sed scripts.  Here is another example:
>
> /*
>  *  Incorrect sprintf declaration in X11/Xmu.h
>  */
> fix = {
>    hackname = x11_sprintf;
>    files    = X11/Xmu.h;
>    files    = X11/Xmu/Xmu.h;
>    select   = "^extern char \\*\tsprintf\\(\\);$";
>
>    c_fix     = format;
>    c_fix_arg = "#ifndef __STDC__\n%0\n#endif /* !defined __STDC__ */";
>
>    test_text = "extern char *\tsprintf();";
> };
>
> Paolo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/libdispatch-dev/attachments/20100608/de4f01cb/attachment.html>

From david-m at orbotech.com  Sun Jun 13 22:18:00 2010
From: david-m at orbotech.com (David Minor)
Date: Mon, 14 Jun 2010 08:18:00 +0300
Subject: [libdispatch-dev] OpenCL
In-Reply-To: <5EB79B06-BD2F-4BB5-8367-6368F90B5271@shadowlab.org>
Message-ID: <FCB44A2146B78C479695CF9CCA7EEA8706A88256@excg-isl01>

Is there a detailed explanation of the actual plumbing they use anywhere?


-----Original Message-----
From: Jean-Daniel Dupas [mailto:devlists at shadowlab.org] 
Sent: Tuesday, June 08, 2010 4:00 PM
To: David Minor
Cc: libdispatch-dev at lists.macosforge.org
Subject: Re: [libdispatch-dev] OpenCL

AFAIK, that's already how it works on OS X:

>From OpenCL Programming Guide:

?Apple's implementation of OpenCL is built on top of Grand Central Dispatch (GCD).?

http://developer.apple.com/mac/library/documentation/Performance/Conceptual/OpenCL_MacProgGuide/OpenCLontheMacPlatform/OpenCLontheMacPlatform.html

Le 8 juin 2010 ? 14:53, David Minor a ?crit :

> Is anybody working on the idea of getting OpenCL kernels to run under libdispatch?  A kernel is semantically similar to a block.
> I think it would be a big win to be able to do this.
>  
> David Minor
>  
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev

-- Jean-Daniel





From mark at heily.com  Mon Jun 14 21:36:32 2010
From: mark at heily.com (Mark Heily)
Date: Tue, 15 Jun 2010 00:36:32 -0400
Subject: [libdispatch-dev] libpthread_workqueue released
Message-ID: <4C170350.5010506@heily.com>

I have written a pure userspace implementation of the Darwin 
pthread_workqueue API as used by libdispatch. This is based on Stacey Son's 
port to FreeBSD [1], but does not include any of the kernel bits. It's 
portable to a wide range of POSIX-compliant platforms. I have tested it 
under Linux, and it shows major improvements in the performance and 
stability of the libdispatch test suite.

I havn't implemented overcommit or multiple priority levels yet, but the 
basic workqueue functionality is there. I have a few ideas for adding kernel 
support and will be developing a Linux kernel module to complement the 
existing userspace code.

A tarball of the source code is here:

	http://mark.heily.com/src/libpthread_workqueue-0.1.tar.gz

The SVN repository is available here:

	svn://mark.heily.com/libpthread_workqueue

To try this with libdispatch, add '-lpthread_workqueue' to the LDADD 
variable in testing/Makefile.am.

Regards,

  - Mark

[1] http://people.freebsd.org/~sson/thrworkq/

From leimy2k at gmail.com  Tue Jun 15 07:28:24 2010
From: leimy2k at gmail.com (David Leimbach)
Date: Tue, 15 Jun 2010 07:28:24 -0700
Subject: [libdispatch-dev] libpthread_workqueue released
In-Reply-To: <4C170350.5010506@heily.com>
References: <4C170350.5010506@heily.com>
Message-ID: <AANLkTilOzERTBDvtQULIOYCF-L0t-vRroBpnOERO42jq@mail.gmail.com>

This is excellent!

On Mon, Jun 14, 2010 at 9:36 PM, Mark Heily <mark at heily.com> wrote:

> I have written a pure userspace implementation of the Darwin
> pthread_workqueue API as used by libdispatch. This is based on Stacey Son's
> port to FreeBSD [1], but does not include any of the kernel bits. It's
> portable to a wide range of POSIX-compliant platforms. I have tested it
> under Linux, and it shows major improvements in the performance and
> stability of the libdispatch test suite.
>
> I havn't implemented overcommit or multiple priority levels yet, but the
> basic workqueue functionality is there. I have a few ideas for adding kernel
> support and will be developing a Linux kernel module to complement the
> existing userspace code.
>
> A tarball of the source code is here:
>
>        http://mark.heily.com/src/libpthread_workqueue-0.1.tar.gz
>
> The SVN repository is available here:
>
>        svn://mark.heily.com/libpthread_workqueue
>
> To try this with libdispatch, add '-lpthread_workqueue' to the LDADD
> variable in testing/Makefile.am.
>
> Regards,
>
>  - Mark
>
> [1] http://people.freebsd.org/~sson/thrworkq/
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/libdispatch-dev/attachments/20100615/102c7911/attachment.html>

From jkh at apple.com  Sun Jun 20 13:16:47 2010
From: jkh at apple.com (Jordan K. Hubbard)
Date: Sun, 20 Jun 2010 13:16:47 -0700
Subject: [libdispatch-dev] libpthread_workqueue released
In-Reply-To: <4C170350.5010506@heily.com>
References: <4C170350.5010506@heily.com>
Message-ID: <14223824-9B48-4550-840E-1EFC9F4A2F78@apple.com>

Hi Mark,

That's very interesting, and certainly solves the portability problem for those OSes unwilling / unable to implement pthread workqueues in kernel space.  How are you planning on implementing the system-wide load balancing behavior?  A global semaphore of some sort, I would assume, but it's really your assumptions I think we're interested in. :)

Looking further forward, is this something you would like to propose as part of the "official libdispatch sources" for making the code base more portable?

Thanks,

- Jordan

On Jun 14, 2010, at 9:36 PM, Mark Heily wrote:

> I have written a pure userspace implementation of the Darwin pthread_workqueue API as used by libdispatch. This is based on Stacey Son's port to FreeBSD [1], but does not include any of the kernel bits. It's portable to a wide range of POSIX-compliant platforms. I have tested it under Linux, and it shows major improvements in the performance and stability of the libdispatch test suite.
> 
> I havn't implemented overcommit or multiple priority levels yet, but the basic workqueue functionality is there. I have a few ideas for adding kernel support and will be developing a Linux kernel module to complement the existing userspace code.
> 
> A tarball of the source code is here:
> 
> 	http://mark.heily.com/src/libpthread_workqueue-0.1.tar.gz
> 
> The SVN repository is available here:
> 
> 	svn://mark.heily.com/libpthread_workqueue
> 
> To try this with libdispatch, add '-lpthread_workqueue' to the LDADD variable in testing/Makefile.am.
> 
> Regards,
> 
> - Mark
> 
> [1] http://people.freebsd.org/~sson/thrworkq/
> _______________________________________________
> libdispatch-dev mailing list
> libdispatch-dev at lists.macosforge.org
> http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev


From mark at heily.com  Wed Jun 23 20:53:53 2010
From: mark at heily.com (Mark Heily)
Date: Wed, 23 Jun 2010 23:53:53 -0400
Subject: [libdispatch-dev] libpthread_workqueue released
In-Reply-To: <14223824-9B48-4550-840E-1EFC9F4A2F78@apple.com>
References: <4C170350.5010506@heily.com>
	<14223824-9B48-4550-840E-1EFC9F4A2F78@apple.com>
Message-ID: <4C22D6D1.6000208@heily.com>

On 06/20/2010 04:16 PM, Jordan K. Hubbard wrote:
 >
> How are you planning on implementing the system-wide load
> balancing behavior?  A global semaphore of some sort, I would assume, but
> it's really your assumptions I think we're interested in. :)
>

There are a couple of big problems with using a global semaphore. A global 
semaphore would need to be world-writable, which leads to a trivial 
denial-of-service vulnerability. Another problem is that a thread may take 
the semaphore, and then sleep or block while waiting for I/O. This would 
starve other threads that are waiting for the semaphore to be released.

Instead of using IPC, I'm planning to manage the size of the thread pool by 
using a combination of four variables:
	
	a := the total number of pending work items on all queues
	b := the number of worker threads
	c := the 1-minute load average from getloadavg(3)
	d := the number of online CPUs

These four variables will be used to compute two ratios:

	items_per_worker        := a / b
	per_cpu_runqueue_length := c / d

If the workload is high (items_per_worker > 1) and system load is low 
(per_cpu_runqueue_length <= 1) , additional worker threads will be added to 
the pool. Conversely, if the system load is high (per_cpu_runqueue_length > 
1) or there are lots of idle workers (items_per_worker < 0.5), some of the 
worker threads will be removed from the pool.

> Looking further forward, is this something you would like to propose as
> part of the "official libdispatch sources" for making the code base more
> portable?
>

I think it makes sense to import this into the libdispatch source tree 
instead of distributing it as a separate library. Once some of the missing 
pieces are finished, I'll submit a patch to integrate it with libdispatch.

Regards,

  - Mark

From mark at heily.com  Wed Jun 23 21:02:43 2010
From: mark at heily.com (Mark Heily)
Date: Thu, 24 Jun 2010 00:02:43 -0400
Subject: [libdispatch-dev] packaging libdispatch & friends for Debian
Message-ID: <4C22D8E3.5050007@heily.com>

I am pleased to announce that libkqueue has been uploaded to the Debian
GNU/Linux package repository [1]. It is available in the 'unstable'
branch, and will be in the 'testing' branch soon.

Debian users who wish to build libdispatch from source should install
the following packages:

     libkqueue0
     libkqueue-dev

I am working on Debian packages for compiler-rt and libdispatch, and hope
to have those ready soon [2].

Cheers,

    - Mark

[1] http://packages.debian.org/sid/libkqueue0

[2] 
http://mentors.debian.net/cgi-bin/sponsor-pkglist?maintainer=mark%40heily.com