mirror of
1
Fork 0
forgejo/modules/charset
Giteabot b272224918
Preserve BOM in web editor () ()
Backport  by @silverwind

The `ToUTF8*` functions were stripping BOM, while BOM is actually valid
in UTF8, so the stripping must be optional depending on use case. This
does:

- Add a options struct to all `ToUTF8*` functions, that by default will
strip BOM to preserve existing behaviour
- Remove `ToUTF8` function, it was dead code
- Rename `ToUTF8WithErr` to `ToUTF8`
- Preserve BOM in Monaco Editor
- Remove a unnecessary newline in the textarea value. Browsers did
ignore it, it seems but it's better not to rely on this behaviour.

Fixes: https://github.com/go-gitea/gitea/issues/28743
Related: https://github.com/go-gitea/gitea/issues/6716 which seems to
have once introduced a mechanism that strips and re-adds the BOM, but
from what I can tell, this mechanism was removed at some point after
that PR.

Co-authored-by: silverwind <me@silverwind.io>
(cherry picked from commit b8e6cffd31)
2024-01-31 14:18:26 +01:00
..
ambiguous Replace `interface{}` with `any` () 2023-07-04 18:36:08 +00:00
invisible Replace `interface{}` with `any` () 2023-07-04 18:36:08 +00:00
ambiguous.go Ensure that Chinese punctuation is not ambiguous when locale is Chinese () 2022-12-04 17:57:30 +00:00
ambiguous_gen.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00
ambiguous_gen_test.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00
breakwriter.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00
breakwriter_test.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00
charset.go Preserve BOM in web editor () () 2024-01-31 14:18:26 +01:00
charset_test.go Preserve BOM in web editor () () 2024-01-31 14:18:26 +01:00
escape.go Add option to disable ambiguous unicode characters detection () () 2023-12-22 12:07:01 +01:00
escape_status.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00
escape_stream.go Add option to disable ambiguous unicode characters detection () () 2023-12-22 12:07:01 +01:00
escape_test.go Add option to disable ambiguous unicode characters detection () () 2023-12-22 12:07:01 +01:00
htmlstream.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00
invisible_gen.go Implement FSFE REUSE for golang files () 2022-11-27 18:20:29 +00:00