Repository navigation
Latin1 encoding for JSRT N-API #235
Description
After updating the Node-ChakraCore JSRT implementation, addons_napi/test_string fails because there are no JSRT string APIs for Latin1 encoding, so the JSRT napi_create_string_latin1() and napi_get_value_string_latin1() incorrectly use UTF-8 encoding instead.
Activity
jasongin commentedon Apr 26, 2017
ChakraCore doesn't have any support for Latin1 encoding, even internally. So the most straightforward way to implement this would be to depend on the character conversion APIs in Node's ICU. Unfortunately, Node-ChakraCore currently doesn't build with the ICU. But there is an effort in progress to improve the intl support of Node-ChakraCore, which will involve building in the ICU. Once that is done, N-API can easily use that to convert Latin1 strings to/from UTF-16 (without requiring any additional JSRT APIs).
addaleax commentedon Apr 26, 2017
@jasongin Luckily, that’s already pretty easy because Unicode was designed to extend the Latin1 character set. 😄 Latin1 → UTF-16 is just zero-extending the bytes, UTF-16 → Latin1 is just truncating them to an uint8_t, I don’t think we need ICU for that.
jasongin commentedon Apr 26, 2017
OK maybe it wouldn't be too bad, but I'd be concerned about edge cases when converting UTF-16 -> Latin1, for example handling characters that are encoded as more than one UTF-16 code point, and filling in placeholder chars. Sure we could code and test all that, but I think it would be cleaner just to depend on the existing implementation in the ICU.
addaleax commentedon Apr 26, 2017
If you’re talking about non-BMP characters that are encoded as 2 UTF-16 surrogates, the behaviour I described is what V8 does … I don’t care, maybe replacement characters make more sense, but if we want compatibility, the naïve approach should actually be the right one. :)
mhdawson commentedon Nov 15, 2021
Closing since Chakra is no longer active, please let us know if you think that was not the right thing to do.