[Laszlo-dev] [Updated] For Review: Change 20090608-raju-Y Summary: Fix for CSS parser uses incorrect file encoding

[P T Withington]

[cc-ing Wanli]

Wanli,

Maybe you also want to review (or have someone on your team review) as  
this issue affects you?

---

Raju,

This looks good.

I think you might have trouble trying checking in your utf-16 examples  
to svn.  I'm not sure svn will do the right thing.  It looks like it  
is treating them as binary files.  Maybe that will work.  I don't know.

It looks to me like you may also have some tab characters in your  
sources.  You need to expand tabs to check in, otherwise the pre- 
commit filter will reject your checkin.

I approve this change.  You might want to check in your fix separately  
from your tests, just in case the UTF-16 files cause svn issues.

On 2009-06-08, at 14:44EDT, Raju Bitter wrote:

> Added a "+" in debug output string concatenation.
>
> Change 20090608-raju-Y by raju at ip-90-186-160-71.web.vodafone.de on  
> 2009-06-08 20:09:02 CEST
>    in /Users/rajubitter/src/svn/openlaszlo/trunk-cssunicode
>    for http://rajubitter@svn.openlaszlo.org/openlaszlo/trunk
>
> Summary: Fix for CSS parser uses incorrect file encoding
>
> New Features: Adds an optional @charset to the stylesheet tag, in  
> case the user wants to use a CSS file in a different encoding then  
> utf-8
>
> Bugs Fixed: LPP-8045
>
> Technical Reviewer: ptw
> QA Reviewer: (pending)
> Doc Reviewer: (pending)
>
> Documentation:
>
> Release Notes:
>
> Details:
> + StyleSheetCompiler.java: Handling for @charset on stylesheet tag  
> added. Added a
> 2nd parameter with the encoding value to the CSSHandler.parse() call.
>
> + CSSHandler.java:
> parse() method takes the encoding from the LZX stylesheet tag as a  
> 2nd parameter.
> getInputSource() method does a few more things now:
>  1) checks for a possible BOM on the CSS file
>  2) checks if a possible BOM conflicts with the value of the  
> stylesheet tag's @charset
>  3) if there's a BOM, the BOM bytes are removed from the input stream
>
> + FileUtils.java:
> Added method public static String  
> detectBOMEncoding(BufferedInputStream in)
> The method returns the BOM marker interpreted as one of the  
> following strings:
>  UTF-8
>  UTF-16LE
>  UTF-16BE
>
> Tests:
> + test files in folder test/css/encoding
> The following test exist:
>  1) iso8859-1_with_charset_attr.lzx
>     Reading an iso-8859-2 encoded CSS file with some German special  
> chars
>  2) utf16BE_with_BOM.lzx
>     Reading an utf-16 BE CSS file with BOM marker
>  3) utf16LE_with_BOM.lzx
>     Reading an utf-16 LE CSS file with BOM marker
>  4) utf8_with_BOM_no_charset_attr.lzx
>     Reading an utf-8 CSS file with BOM and no charset attribute on  
> the stylesheet tag
>  5) utf8_with_BOM_conflicting_charset_attr.lzx
>     Reading a CSS with @charset value of utf-16, but CSS having a  
> UTF-8 BOM marker, will
>     throw a compile error
>
> Files:
> A       test/css/encoding
> A       test/css/encoding/iso8859-1_with_charset_attr.lzx
> A       test/css/encoding/utf16BE_with_BOM.lzx
> A       test/css/encoding/utf8_with_BOM_no_charset_attr.css
> A       test/css/encoding/utf16LE_with_BOM.css
> A       test/css/encoding/utf8_no_BOM_no_charset_attr.css
> A       test/css/encoding/utf8_with_BOM_conflicting_charset_attr.css
> A       test/css/encoding/utf8_with_BOM_no_charset_attr.lzx
> A       test/css/encoding/utf16LE_with_BOM.lzx
> A       test/css/encoding/utf8_no_BOM_no_charset_attr.lzx
> A       test/css/encoding/iso8859-1_with_charset_attr.css
> A       test/css/encoding/utf16BE_with_BOM.css
> A       test/css/encoding/utf8_with_BOM_conflicting_charset_attr.lzx
> M       WEB-INF/lps/server/src/org/openlaszlo/css/CSSHandler.java
> M       WEB-INF/lps/server/src/org/openlaszlo/utils/FileUtils.java
> M       WEB-INF/lps/server/src/org/openlaszlo/compiler/ 
> StyleSheetCompiler.java
>
> Changeset: http://svn.openlaszlo.org/openlaszlo/patches/20090608-raju-Y.tar