[xwiki-devs] attachment's mime-types
Hello XWiki developers, can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ? thanks in advance paul
Paul Libbrecht wrote:
Hello XWiki developers,
can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ?
In web.xml. Search for css, and do something similar for your mimetypes. Of course, restart the server afterwards. -- Sergiu Dumitriu http://purl.org/net/sergiu/
thanks, worked well to one point: the mime-type is added with a charset parameter which is even iso-8859-1 although everything in my environment is set to be utf-8. Do I have a way to remove that charset for such downloads? That would be the most sensible one. thanks in advance paul Le 28-déc.-08 à 13:40, Sergiu Dumitriu a écrit :
can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ?
In web.xml. Search for css, and do something similar for your mimetypes. Of course, restart the server afterwards.
2008/12/28 Paul Libbrecht <paul@activemath.org>
can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ?
... [sergiu suggests "web.xml. Search for css, and do something similar for your mimetypes."] ...
worked well to one point: the mime-type is added with a charset parameter which is even iso-8859-1 although everything in my environment is set to be utf-8.
Do I have a way to remove that charset for such downloads? That would be the most sensible one.
Perhaps this is relevant:
to jerome@xwiki.com, XWiki Users <users@xwiki.org> date Fri, Mar 13, 2009 at 5:48 PM subject Re: [xwiki-users] [xwiki-devs] support for google sitemaps and webmaster tools? (and why do xwiki RDF's give "unsupported file format"?)
...
No matter what I did in my setup, I was getting
<?xml version="1.0" encoding="ISO-8859-1" ?>
The roller blog atom feed that *does* work correctly w/ google sitemaps returns:
<?xml version="1.0" encoding='utf-8'?>
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?). When that alone didn't work, I also added " -Djavax.servlet.request.encoding=UTF-8-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -Djavax.servlet.request.encoding= UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true)
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script... however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten||skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
With all of the above now reconfigured, I now get the correct output for http://nielsmayer.com/xwiki/bin/view/Main/WebRss?xpage=rdf : <?xml version="1.0" encoding="UTF-8" ?>
... -- Niels http://nielsmayer.com
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Unfortunately, none of the below suggestions have helped. Note, contrary to the mails of Jerome and Niels below, it's about the / download/ action here. I still get this with Curriki 1.8 branch (based on xwiki 1.8). This now breaks at least one user. Since there's no single possible thinkable way of knowing the character set of an attachment unless reading it, I insist it is a bug to add this charset to the header! So guys, any way to remove this addition? paul Le 17-mars-09 à 13:42, Niels Mayer a écrit :
2008/12/28 Paul Libbrecht <paul@activemath.org> can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ? ... [sergiu suggests "web.xml. Search for css, and do something similar for your mimetypes."] ... worked well to one point: the mime-type is added with a charset parameter which is even iso-8859-1 although everything in my environment is set to be utf-8.
Do I have a way to remove that charset for such downloads? That would be the most sensible one.
Perhaps this is relevant: to jerome@xwiki.com, XWiki Users <users@xwiki.org> date Fri, Mar 13, 2009 at 5:48 PM subject Re: [xwiki-users] [xwiki-devs] support for google sitemaps and webmaster tools? (and why do xwiki RDF's give "unsupported file format"?)
...
No matter what I did in my setup, I was getting
<?xml version="1.0" encoding="ISO-8859-1" ?>
The roller blog atom feed that *does* work correctly w/ google sitemaps returns: <?xml version="1.0" encoding='utf-8'?>
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?). When that alone didn't work, I also added "- Djavax.servlet.request.encoding=UTF-8-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m - Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m - Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 - DjavaEncoding=UTF-8 -Djava.awt.headless=true)
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script... however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten|| skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
With all of the above now reconfigured, I now get the correct output for http://nielsmayer.com/xwiki/bin/view/Main/WebRss?xpage=rdf : <?xml version="1.0" encoding="UTF-8" ?>
... -- Niels http://nielsmayer.com
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
Paul Libbrecht wrote:
Unfortunately, none of the below suggestions have helped. Note, contrary to the mails of Jerome and Niels below, it's about the /download/ action here.
I still get this with Curriki 1.8 branch (based on xwiki 1.8). This now breaks at least one user.
Since there's no single possible thinkable way of knowing the character set of an attachment unless reading it, I insist it is a bug to add this charset to the header!
So guys, any way to remove this addition?
It's added automatically. If we do: response.setContentType("application/octet-stream"), the container adds automatically the charset, and I think that this explains the behavior: http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/servlet/ServletResponse.... "The charset for the MIME body response can be specified with setContentType(java.lang.String). For example, "text/html; charset=Shift_JIS". The charset can alternately be set using setLocale(java.util.Locale). If no charset is specified, ISO-8859-1 will be used." I think the only way to get rid of it is to provide your own implementation of the ServletResponse class...
Le 17-mars-09 à 13:42, Niels Mayer a écrit :
2008/12/28 Paul Libbrecht <paul@activemath.org> can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ? ... [sergiu suggests "web.xml. Search for css, and do something similar for your mimetypes."] ... worked well to one point: the mime-type is added with a charset parameter which is even iso-8859-1 although everything in my environment is set to be utf-8.
Do I have a way to remove that charset for such downloads? That would be the most sensible one.
Perhaps this is relevant: to jerome@xwiki.com, XWiki Users <users@xwiki.org> date Fri, Mar 13, 2009 at 5:48 PM subject Re: [xwiki-users] [xwiki-devs] support for google sitemaps and webmaster tools? (and why do xwiki RDF's give "unsupported file format"?)
...
No matter what I did in my setup, I was getting
<?xml version="1.0" encoding="ISO-8859-1" ?>
The roller blog atom feed that *does* work correctly w/ google sitemaps returns: <?xml version="1.0" encoding='utf-8'?>
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?). When that alone didn't work, I also added "-Djavax.servlet.request.encoding=UTF-8-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true)
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script... however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten||skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
With all of the above now reconfigured, I now get the correct output for http://nielsmayer.com/xwiki/bin/view/Main/WebRss?xpage=rdf : <?xml version="1.0" encoding="UTF-8" ?>
... -- Niels http://nielsmayer.com
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
-- Sergiu Dumitriu http://purl.org/net/sergiu/
I wish you were right and it would be configurable by the tomcat but it's not the case. The same tomcat serves: http://i2geo.net/files/movies/index.htm with: Content-Type: text/html; charset=utf-8 so you are saying you can't do anything against it? paul Le 04-oct.-09 à 04:08, Sergiu Dumitriu a écrit :
Paul Libbrecht wrote:
Unfortunately, none of the below suggestions have helped. Note, contrary to the mails of Jerome and Niels below, it's about the /download/ action here.
I still get this with Curriki 1.8 branch (based on xwiki 1.8). This now breaks at least one user.
Since there's no single possible thinkable way of knowing the character set of an attachment unless reading it, I insist it is a bug to add this charset to the header!
So guys, any way to remove this addition?
It's added automatically. If we do:
response.setContentType("application/octet-stream"), the container adds automatically the charset, and I think that this explains the behavior:
http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/servlet/ServletResponse....
"The charset for the MIME body response can be specified with setContentType(java.lang.String). For example, "text/html; charset=Shift_JIS". The charset can alternately be set using setLocale(java.util.Locale). If no charset is specified, ISO-8859-1 will be used."
I think the only way to get rid of it is to provide your own implementation of the ServletResponse class...
Le 17-mars-09 à 13:42, Niels Mayer a écrit :
2008/12/28 Paul Libbrecht <paul@activemath.org> can someone explain me what is the strategy to define the mime- type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ? ... [sergiu suggests "web.xml. Search for css, and do something similar for your mimetypes."] ... worked well to one point: the mime-type is added with a charset parameter which is even iso-8859-1 although everything in my environment is set to be utf-8.
Do I have a way to remove that charset for such downloads? That would be the most sensible one.
Perhaps this is relevant: to jerome@xwiki.com, XWiki Users <users@xwiki.org> date Fri, Mar 13, 2009 at 5:48 PM subject Re: [xwiki-users] [xwiki-devs] support for google sitemaps and webmaster tools? (and why do xwiki RDF's give "unsupported file format"?)
...
No matter what I did in my setup, I was getting
<?xml version="1.0" encoding="ISO-8859-1" ?>
The roller blog atom feed that *does* work correctly w/ google sitemaps returns: <?xml version="1.0" encoding='utf-8'?>
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?). When that alone didn't work, I also added "-Djavax.servlet.request.encoding=UTF-8-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m - Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true)
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script... however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten||skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
With all of the above now reconfigured, I now get the correct output for http://nielsmayer.com/xwiki/bin/view/Main/WebRss?xpage=rdf : <?xml version="1.0" encoding="UTF-8" ?>
... -- Niels http://nielsmayer.com
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
-- Sergiu Dumitriu http://purl.org/net/sergiu/ _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
Paul Libbrecht wrote:
I wish you were right and it would be configurable by the tomcat but it's not the case.
The same tomcat serves: http://i2geo.net/files/movies/index.htm
with: Content-Type: text/html; charset=utf-8 so you are saying you can't do anything against it?
We can specify a different encoding, but how can we guess it? We could use a library that guesses the encoding of a file. But is it worth the (computational) effort? What problems does this cause?
Le 04-oct.-09 à 04:08, Sergiu Dumitriu a écrit :
Paul Libbrecht wrote:
Unfortunately, none of the below suggestions have helped. Note, contrary to the mails of Jerome and Niels below, it's about the /download/ action here.
I still get this with Curriki 1.8 branch (based on xwiki 1.8). This now breaks at least one user.
Since there's no single possible thinkable way of knowing the character set of an attachment unless reading it, I insist it is a bug to add this charset to the header!
So guys, any way to remove this addition?
It's added automatically. If we do:
response.setContentType("application/octet-stream"), the container adds automatically the charset, and I think that this explains the behavior:
http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/servlet/ServletResponse....
"The charset for the MIME body response can be specified with setContentType(java.lang.String). For example, "text/html; charset=Shift_JIS". The charset can alternately be set using setLocale(java.util.Locale). If no charset is specified, ISO-8859-1 will be used."
I think the only way to get rid of it is to provide your own implementation of the ServletResponse class...
Le 17-mars-09 à 13:42, Niels Mayer a écrit :
2008/12/28 Paul Libbrecht <paul@activemath.org> can someone explain me what is the strategy to define the mime-type of an attachment? I see not configurable map nor any setter on the attachment types. Is there any way I can enrich this or is it simply taking over the mime-type provided the browser (which, too often, ends up being application/octet-stream) ? ... [sergiu suggests "web.xml. Search for css, and do something similar for your mimetypes."] ... worked well to one point: the mime-type is added with a charset parameter which is even iso-8859-1 although everything in my environment is set to be utf-8.
Do I have a way to remove that charset for such downloads? That would be the most sensible one.
Perhaps this is relevant: to jerome@xwiki.com, XWiki Users <users@xwiki.org> date Fri, Mar 13, 2009 at 5:48 PM subject Re: [xwiki-users] [xwiki-devs] support for google sitemaps and webmaster tools? (and why do xwiki RDF's give "unsupported file format"?)
...
No matter what I did in my setup, I was getting
<?xml version="1.0" encoding="ISO-8859-1" ?>
The roller blog atom feed that *does* work correctly w/ google sitemaps returns: <?xml version="1.0" encoding='utf-8'?>
I fixed this issue by running java with -Dfile.encoding=UTF-8 (note the lowercase setting suggested in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Performances seems incorrect?). When that alone didn't work, I also added "-Djavax.servlet.request.encoding=UTF-8-DjavaEncoding=UTF-8" which had been suggested in solving this problem for other Tomcat users. (Now I run java with the following options:-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true)
I also saw other suggestions to set LANG="en_US.UTF-8" in the tomcat launching script... however, I'm not sure which of my changes "did" it, but i believe that following two steps I'd forgotten||skipped in http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Encoding caused the correct encoding to be used: (1) WEB-INF> diff web.xml.~1~ web.xml 23c23 < <param-value>ISO-8859-1</param-value> ---
<param-value>UTF-8</param-value>
(2) WEB-INF> diff xwiki.cfg.~2~ xwiki.cfg 29c29 < xwiki.encoding=ISO-8859-1 ---
xwiki.encoding=UTF-8
With all of the above now reconfigured, I now get the correct output for http://nielsmayer.com/xwiki/bin/view/Main/WebRss?xpage=rdf : <?xml version="1.0" encoding="UTF-8" ?>
... -- Niels http://nielsmayer.com
PS: Why not just have xwiki.cfg's default be: 'xwiki.encoding=UTF-8' ; likewise have web.xml's default for com.xpn.xwiki.web.SetCharacterEncodingFilter's 'encoding' be UTF-8. These encoding errors that oft go unnoticed are probably resulting in a number of configuration errors, and perhaps other bug-reports that aren't entirely valid, should they depend on encoding issues.
-- Sergiu Dumitriu http://purl.org/net/sergiu/
Well, if we can specify a different encoding it should be the default xwiki encoding; of course it's UTF-8 in my case. Of course ideally, it should be just absent because, as you say, you can't be sure of the charset before you actually sniff into it. Where is this coded for the download action? The problems that it cause are simple: applet parameters get garbled and thus make the applet non-working. thanks in advance paul Le 04-oct.-09 à 11:30, Sergiu Dumitriu a écrit :
I wish you were right and it would be configurable by the tomcat but it's not the case.
The same tomcat serves: http://i2geo.net/files/movies/index.htm
with: Content-Type: text/html; charset=utf-8 so you are saying you can't do anything against it?
We can specify a different encoding, but how can we guess it?
We could use a library that guesses the encoding of a file. But is it worth the (computational) effort? What problems does this cause?
Sergiu Dumitriu <sergiu.dumitriu@gmail.com>:
the container adds automatically the charset, and I think that this explains the behavior:
http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/servlet/ServletResponse....
"The charset for the MIME body response can be specified with setContentType(java.lang.String). For example, "text/html; charset=Shift_JIS". The charset can alternately be set using setLocale(java.util.Locale). If no charset is specified, ISO-8859-1 will be used."
Despite Paul's complaint that my suggestions didn't help, wouldn't the omission of an explicit locale/charset setting for the container cause Tomcat to get an inappropriate default charset from the environment? The locale settings in Tomcat include the charset, e.g. setLocale(java.util.Locale)<http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/servlet/ServletResponse.html#setLocale(java.util.Locale)> "Sets the locale of the response, setting the headers (including the Content-Type's charset) as appropriate" I'm currently not experiencing such problems, and my Tomcat config has environment variable settings like: LANG="en_US.UTF-8" (english/US locale with UTF-8 content encoding) JAVA_OPTS="-server -Xms160m -Xmx1024m -XX:PermSize=160m -XX:MaxPermSize=320m -XX:+DoEscapeAnalysis -XX:+UseBiasedLocking -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintCompilation -XX:+PrintGCDetails -XX:+TraceClassLoading -XX:+TraceClassUnloading -Djavax.servlet.request.encoding=UTF-8 -Dfile.encoding=UTF-8 -DjavaEncoding=UTF-8 -Djava.awt.headless=true -Djava.library.path=/usr/lib64" I'm pretty sure the LANG setting with encoding is the setting that does the trick. You might try dropping an "env(1)" (unix: "man 1 env") into your tomcat launch script to see what environment vars are actually being inherited from the environment (or just launch a subprocess out of java and run "env(1)"). Sometimes you might be surprised by unexpected environment variables that are being set, or having different variables set depending on whether the script is launched as a service or just run out of the user's shell. To avoid environmental dependencies, how about just using, e.g., $response.setContentType("text/xml; charset=UTF-8")? --Niels http://nielsmayer.com PS:
It's added automatically. If we do: response.setContentType("application/octet-stream"),
I agree that setContentType is extremely useful part of Xwiki API. As example, I have a ClassSheet that displays the same doc three different ways: #if( "$!request.xpage" == "plain" )##{ #if( "$!request.json" != "" )##{ $response.setContentType("text/javascript")## ... #else##}{ $response.setContentType("text/xml")## ... #end##} #else##}{ -- xpage=plain parameter not given, do "normal wikipage" ... #end##}
participants (4)
-
Niels Mayer -
Paul Libbrecht -
Sergiu Dumitriu -
Sergiu Dumitriu