[xwiki-devs] [VOTE] Make UTF-8 mandatory for a valid installation
Hi devs, We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years. I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros) There's one big site to update on our side: xwiki.org. Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8. After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all. -- Sergiu Dumitriu http://purl.org/net/sergiu
Hi, On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example For XWiki 5.x I'm not sure. To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything? Thanks -Vincent
On 12/07/2012 04:26 PM, Vincent Massol wrote:
Hi,
On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
+1 for UTF-8 only. If we want to support an encoding we need to run our test suite with it so each encoding we support multiplies the test run time and it's not going to bring features to the user's hands. +1 for waiting until 5.x at least before making it mandatory because we will have to require MySQL >= 5.5.3 and set the encoding to utf8mb4 in order to avoid errors when saving pages with 4 byte codepoints. http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html I understand that some users currently set the encoding to latin1 so MySQL will just treat the data as opaque blobs. Thanks, Caleb
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
I found some interesting explanations about why non-UTF-8 unicode is more or less a historical accident. http://www.utf8everywhere.org/ http://programmers.stackexchange.com/questions/102205/should-utf-16-be-consi... It seems that it was originally thought that 65535 chars would be "enough for anyone" but now that UTF-16 has accepted it's fate as a variable width encoding, it's basically UTF-8 with padding, lack of backward (ascii) compatibility, and endianness. The most prominent argument for UTF-16 seems to be that it's more "culturally inclusive" since Arabic and Asian languages encode with more similar efficiency. I think speakers of Asian languages would care more about not having broken content because of UTF-16 implementations which still assume char == 16 bits. Which until 1.5, included Java: http://www.ibm.com/developerworks/java/library/j-unicode/ And in some places it still does: http://docs.oracle.com/javase/6/docs/api/java/lang/StringBuffer.html#reverse...) Thanks, Caleb On 12/07/2012 04:56 PM, Caleb James DeLisle wrote:
On 12/07/2012 04:26 PM, Vincent Massol wrote:
Hi,
On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
+1 for UTF-8 only.
If we want to support an encoding we need to run our test suite with it so each encoding we support multiplies the test run time and it's not going to bring features to the user's hands.
+1 for waiting until 5.x at least before making it mandatory because we will have to require MySQL >= 5.5.3 and set the encoding to utf8mb4 in order to avoid errors when saving pages with 4 byte codepoints. http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html I understand that some users currently set the encoding to latin1 so MySQL will just treat the data as opaque blobs.
Thanks, Caleb
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
On 12/07/2012 04:56 PM, Caleb James DeLisle wrote:
On 12/07/2012 04:26 PM, Vincent Massol wrote:
Hi,
On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
+1 for UTF-8 only.
If we want to support an encoding we need to run our test suite with it so each encoding we support multiplies the test run time and it's not going to bring features to the user's hands.
+1 for waiting until 5.x at least before making it mandatory because we will have to require MySQL >= 5.5.3 and set the encoding to utf8mb4 in order to avoid errors when saving pages with 4 byte codepoints. http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html
I'm afraid we'll get errors if we do that, since indexes are still limited to a total of 1024 bytes, and we're already maxing out with 255-varchar columns + other fields. In short, MySQL sucks for serious projects, but we can't really tell our users "use Postgres, it's better". So I'd rather keep it to the current utf-8, and hope that nobody will need the extended unicode planes, until we find a better solution. To be more specific: we can't switch to 4-byte utf8 until we stop using names as primary key elements. Just tried it, and indeed trying to save characters outside the BMP will fail. Thanks for pointing this out.
I understand that some users currently set the encoding to latin1 so MySQL will just treat the data as opaque blobs.
Except that it doesn't work like that. If you use latin1, you'll get errors with the default XE xar about invalid values in the RCS table. The connector doesn't send bytes, it sends characters, and the database will try to store them, which it can't. Every piece of MySQL has an encoding, which isn't opaque. Pushing characters outside the table's charset will trigger an exception.
Thanks, Caleb
Thanks -Vincent
-- Sergiu Dumitriu http://purl.org/net/sergiu
On Dec 11, 2012, at 12:43 AM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
On 12/07/2012 04:56 PM, Caleb James DeLisle wrote:
On 12/07/2012 04:26 PM, Vincent Massol wrote:
Hi,
On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
+1 for UTF-8 only.
If we want to support an encoding we need to run our test suite with it so each encoding we support multiplies the test run time and it's not going to bring features to the user's hands.
+1 for waiting until 5.x at least before making it mandatory because we will have to require MySQL >= 5.5.3 and set the encoding to utf8mb4 in order to avoid errors when saving pages with 4 byte codepoints. http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html
I'm afraid we'll get errors if we do that, since indexes are still limited to a total of 1024 bytes, and we're already maxing out with 255-varchar columns + other fields. In short, MySQL sucks for serious projects, but we can't really tell our users "use Postgres, it's better". So I'd rather keep it to the current utf-8, and hope that nobody will need the extended unicode planes, until we find a better solution.
To be more specific: we can't switch to 4-byte utf8 until we stop using names as primary key elements.
Just tried it, and indeed trying to save characters outside the BMP will fail. Thanks for pointing this out.
I understand that some users currently set the encoding to latin1 so MySQL will just treat the data as opaque blobs.
Except that it doesn't work like that. If you use latin1, you'll get errors with the default XE xar about invalid values in the RCS table. The connector doesn't send bytes, it sends characters, and the database will try to store them, which it can't. Every piece of MySQL has an encoding, which isn't opaque. Pushing characters outside the table's charset will trigger an exception.
Reviving this thread now that 5.0 dev is going to start. xwiki.org is still running latin1 AFAIK and it's working well, including for page history so I'm not sure what the problem is. Now I'm fine to require UTF8. It would be nice to check the environment at startup. I hope we can do so. This means checking that DB and container are set up correctly. This is important also for existing users who are using latin1. They need to know they have something to do. xwiki.org is a good example. We should also document how users can migrate their DBs to UTF8 in our admin guide on xwiki.org. Does it mean we'll remove (deprecate to start with?) the xwiki.encoding config parameter? Thanks -Vincent
Hello, I thought xwiki was using UTF-8 by default?! Because in that case, we _only_ need to check the "environment", which means every supported database and containers, and maybe the system language, not sure it matters btw. I can take care of such a documentation if you want. I've heard that postgreSQL uses utf-8 by default, but need to study it anyway. 2013/2/11 Vincent Massol <vincent@massol.net>
On Dec 11, 2012, at 12:43 AM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
On 12/07/2012 04:56 PM, Caleb James DeLisle wrote:
On 12/07/2012 04:26 PM, Vincent Massol wrote:
Hi,
On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some
distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
+1 for UTF-8 only.
If we want to support an encoding we need to run our test suite with it so each encoding we support multiplies the test run time and it's not going to bring features to the user's hands.
+1 for waiting until 5.x at least before making it mandatory because we will have to require MySQL >= 5.5.3 and set the encoding to utf8mb4 in order to avoid errors when saving pages with 4 byte codepoints. http://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf8mb4.html
I'm afraid we'll get errors if we do that, since indexes are still limited to a total of 1024 bytes, and we're already maxing out with 255-varchar columns + other fields. In short, MySQL sucks for serious projects, but we can't really tell our users "use Postgres, it's better". So I'd rather keep it to the current utf-8, and hope that nobody will need the extended unicode planes, until we find a better solution.
To be more specific: we can't switch to 4-byte utf8 until we stop using names as primary key elements.
Just tried it, and indeed trying to save characters outside the BMP will fail. Thanks for pointing this out.
I understand that some users currently set the encoding to latin1 so MySQL will just treat the data as opaque blobs.
Except that it doesn't work like that. If you use latin1, you'll get errors with the default XE xar about invalid values in the RCS table. The connector doesn't send bytes, it sends characters, and the database will try to store them, which it can't. Every piece of MySQL has an encoding, which isn't opaque. Pushing characters outside the table's charset will trigger an exception.
Reviving this thread now that 5.0 dev is going to start.
xwiki.org is still running latin1 AFAIK and it's working well, including for page history so I'm not sure what the problem is.
Now I'm fine to require UTF8. It would be nice to check the environment at startup. I hope we can do so. This means checking that DB and container are set up correctly. This is important also for existing users who are using latin1. They need to know they have something to do. xwiki.org is a good example. We should also document how users can migrate their DBs to UTF8 in our admin guide on xwiki.org.
Does it mean we'll remove (deprecate to start with?) the xwiki.encoding config parameter?
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
Guillaume, I think the interest of "making it mandatory" is to write the necessary code to assert that it is ok and fail to start otherwise. E.g. write to the DB, clean any cache, then check the read is consistent (i generally use the lambda letter for this). Although PostgreSQL might make it default, it will not be the case of older Postgres some of which we may still support. Paul On 12 févr. 2013, at 16:30, Guillaume Fenollar wrote:
I thought xwiki was using UTF-8 by default?! Because in that case, we _only_ need to check the "environment", which means every supported database and containers, and maybe the system language, not sure it matters btw. I can take care of such a documentation if you want. I've heard that postgreSQL uses utf-8 by default, but need to study it anyway.
On 12/07/2012 04:26 PM, Vincent Massol wrote:
Hi,
On Dec 7, 2012, at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
Non-ASCII-compatible encodings don't work at least because of the way we read components.txt. For other encodings, the problem is that you never know when something will break. Things may appear to work, but enter a non-ASCII character and: - the data might be discarded - the database might throw an exception -- if a non-transactional database engine is used, then all future access to the document or history or object properties will fail - the document might become inaccessible since the URL is decoded incorrectly - The PDF export might break - Escaped XML entities might appear in the browser instead of characters (just from the top of my head) This isn't a proposal to change our rules, it's a proposal to make explicit what we've been doing anyway. There have been many issues and emails where our answer starts with "make sure your <component> encoding is set to UTF-8". On xwiki.org users often try to use non-ASCII characters, and that doesn't work, and so we might be losing potential users if they assume that XWiki simply doesn't support their language. -- Sergiu Dumitriu http://purl.org/net/sergiu
+1 for utf-8 only. Ok to wait till 5.0 unless there are good migrations.
To be able to answer I need to understand more. For example what currently doesn't work with any encoding the user wants to use? Shouldn't we just be transparent and use whatever encoding is specified and not hardcode anything?
Unfortunately, this has been a dream that never comes true that has never been sufficiently tested. I know for sure that both Solr and XWiki will remain imperfect at some edge cases which few people test if their application-level-configured encoding differs from the platform-encoding. XWiki may well have thought well about this since its developers pay attention to deprecations often, there's a zillion remaining other places where this is not done as cleanly. Sergiu, I understand your proposal to include a validation of the environment at startup, is that correct? Including a validation of the DB abilities? paul
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all.
For XWiki 4.x I'm -1 since it's a big change and we don't want to break our users that currently use 4.x with ISO8859-1 for example
For XWiki 5.x I'm not sure.
+1 Thanks, Marius On Fri, Dec 7, 2012 at 10:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all. -- Sergiu Dumitriu http://purl.org/net/sergiu _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
+1 Thanks, Eduard On Mon, Dec 10, 2012 at 11:41 AM, Marius Dumitru Florea < mariusdumitru.florea@xwiki.com> wrote:
+1
Thanks, Marius
On Fri, Dec 7, 2012 at 10:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all. -- Sergiu Dumitriu http://purl.org/net/sergiu _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
+1 On Fri, Dec 7, 2012 at 9:59 PM, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Hi devs,
We've moved more and more toward an UTF-8-only application, and XWiki has only been tested with this configuration for several years.
I propose that we require UTF-8 for a valid, supported installation. This means: - JVM encoding (-Dfile.encoding=UTF8) - Container default URL encoding (Tomcat has ISO-8859-1 by default) - Database encoding (MySql is still configured with latin1 on some distros)
There's one big site to update on our side: xwiki.org.
Here's my +1. This is a move toward a future web, since more and more standards require (or at least assume as a default) UTF-8.
After thinking a bit more, it would make sense to require a valid Unicode encoding, including UTF-16, which is preferable in countries that don't use a latin alphabet. However, XWiki doesn't currently work under 16-bit encodings at all. -- Sergiu Dumitriu http://purl.org/net/sergiu _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
participants (8)
-
Caleb James DeLisle -
Eduard Moraru -
Guillaume Fenollar -
Marius Dumitru Florea -
Paul Libbrecht -
Sergiu Dumitriu -
Thomas Mortagne -
Vincent Massol