[MacRuby-devel] Strings, Encodings and IO

[Benjamin Stiglitz]

>> So plan B: We emulate Ruby 1.9 strings behavior on top of of  
>> NSString/NSData.
>
> I'm really interested in this discussion too. A little background  
> for JRuby:

Thanks for the background, Charlie. This sort of history is very  
instructive.

> * Java's strings are all UTF-16. In order to represent binary data,  
> we ended up using a "raw" encoder/decoder and only using the bottom  
> byte of each character. Wasteful, since every string was 2x as  
> large, and slow, since IO had to up/downcast byte[] contents to  
> char[] and back.

Most CFStrings use a UTF-16 internal store as well.

> * We want to move to an intermediate version, where we sometimes  
> have a byte[]-backed string and sometimes a char[]/String-backed  
> string. IronRuby does this already. This is, however, predicated on  
> the idea that byte[]-based strings rarely become char[]-based  
> strings and vice versa. I don't have any evidence for or against  
> that yet.
>
> So it's a nearly identical problem for MacRuby, as I understand it.  
> I'm interested in discussion around this topic, since we are still  
> moving forward with JRuby and would like to improve interop with  
> Java libraries. I will offer the following food for thought:
>
> * Going with 100% objc strings at first is probably a good pragmatic  
> start. You'll have the perf/memory impact of encoding/decoding and  
> wasteful string contents, but you should be able to get it  
> functioning well. And since interop is a primary goal for MacRuby  
> (where it's been somewhat secondary in JRuby) this is probably a  
> better place to start.

That’s where things stand today, and with Laurent’s ByteString work  
this all mostly works as long as you don’t try to change encodings  
around on strings.

> * Alternatively, you could only support a minimum set of encodings  
> and make it explicit that internally everything would be UTF-16 or  
> MacRoman. In MacRuby's case, I think most people would happily  
> accept that, just as a lot of JRuby users would probably accept that  
> everything's UTF-16 since that's what they get from Java normally.

This seems like a bad situation in the face of the varied encoding  
landscape on the Internet.

> Ultimately this is the exact reason I argued over a year ago that  
> Ruby 1.9 should introduce a separate Bytes class used for IO. I was  
> denied.

I was disappointed to see this turned down as well. The encoding  
situation in 1.9 feels worse than it was in 1.8, and that’s pretty  
impressive.

> It's definitely a sticky issue, and Ruby has made it even stickier  
> in 1.9 with arbitrary encoding support. None of the proposed  
> solutions across all implementations (including JRuby) have really  
> seemed ideal to me.

Laurent and I discussed this a bit tonight, and here’s what I think we  
can get away with:

By default, store all strings as NSString (UTF-16 backed) with an ivar  
to store the encoding e.
When getting bytes, convert to a ByteString in the appropriate encoding.
When doing force_encoding, convert to a ByteString in the old  
encoding, then try to convert to an NSString in the new encoding. If  
we succeed, great. If not, leave as a tagged ByteString (and probably  
whine about it).
All ASCII-8BIT strings are backed by ByteString.

There’s some simplification here; some of the ByteStrings are really  
just NSDatas, &c., but the flow is there. Sorry the list above is a  
mess, I’m up much later than I’m accustomed to.

-Ben