mirror of
https://github.com/MaSzyna-EU07/maszyna.git
synced 2026-11-05 10:59:02 +01:00
fix(utilities): win1250_to_utf8 turns the polish letters into utf-8, not into themselves (#252)
utilities.cpp starts with a byte order mark, so the compiler reads it as utf-8. The game's targets don't get /utf-8, though: set_target_properties(... COMPILE_OPTIONS ...) in the MSVC block of CMakeLists.txt replaces the options add_compile_options gave them. So the string literals went into the program in the code page of the system, windows-1250 on a polish one, and the map of win1250_to_utf8 turned each letter into itself. What read windows-1250 through it got no utf-8: the names, descriptions and categories of the sceneries and the names of the textures in the launcher, and the descriptions of the templates in the editor. The editor writes such a description back through utf8_to_win1250, which makes a question mark of each byte that doesn't start a utf-8 sequence, so the letters it had read unchanged didn't come back. The utf-8 sequences are written as bytes now, the way utf8_to_win1250 has them, so they don't depend on the options. The options stay as they are: with /utf-8 other literals of the files with a byte order mark would change, the "±" of the table of Bezogonkow would take two bytes and move the letters after it. The unit tests check the letters and that the text given is valid utf-8. Co-authored-by: maj00r <maj00r@host>
This commit is contained in:
26
tests/unit/test_win1250.cpp
Normal file
26
tests/unit/test_win1250.cpp
Normal file
@@ -0,0 +1,26 @@
|
||||
#include "stdafx.h"
|
||||
#include "doctest.h"
|
||||
#include "utilities/utilities.h"
|
||||
|
||||
// the polish letters, in windows-1250 and in utf-8; written as bytes, the build reads the sources in any code page
|
||||
namespace
|
||||
{
|
||||
std::string const kWin1250{"\xA5\xC6\xCA\xA3\xD1\xD3\x8C\x8F\xAF\xB9\xE6\xEA\xB3\xF1\xF3\x9C\x9F\xBF"};
|
||||
std::string const kUtf8{"\xC4\x84\xC4\x86\xC4\x98\xC5\x81\xC5\x83\xC3\x93\xC5\x9A\xC5\xB9\xC5\xBB"
|
||||
"\xC4\x85\xC4\x87\xC4\x99\xC5\x82\xC5\x84\xC3\xB3\xC5\x9B\xC5\xBA\xC5\xBC"};
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("win1250: the polish letters of a scenery file read as utf-8") {
|
||||
CHECK(win1250_to_utf8("Bia\xB3" "a Podlaska") == "Bia\xC5\x82" "a Podlaska");
|
||||
CHECK(win1250_to_utf8(kWin1250) == kUtf8);
|
||||
// the rest of the text stays as it is
|
||||
CHECK(win1250_to_utf8("tor 1, km 0.178") == "tor 1, km 0.178");
|
||||
}
|
||||
|
||||
TEST_CASE("win1250: what win1250_to_utf8 gives is valid utf-8") {
|
||||
std::u32string decoded;
|
||||
CHECK_FALSE(utf8_to_utf32(kWin1250, decoded));
|
||||
REQUIRE(utf8_to_utf32(win1250_to_utf8(kWin1250), decoded));
|
||||
CHECK(decoded.size() == kWin1250.size());
|
||||
CHECK(decoded[12] == U'\x0142');
|
||||
}
|
||||
Reference in New Issue
Block a user