17
0
mirror of https://github.com/MaSzyna-EU07/maszyna.git synced 2026-11-05 13:14:50 +01:00

fix(utilities): win1250_to_utf8 turns the polish letters into utf-8, not into themselves (#252)

utilities.cpp starts with a byte order mark, so the compiler reads it as utf-8. The game's targets don't get
/utf-8, though: set_target_properties(... COMPILE_OPTIONS ...) in the MSVC block of CMakeLists.txt replaces the
options add_compile_options gave them. So the string literals went into the program in the code page of the
system, windows-1250 on a polish one, and the map of win1250_to_utf8 turned each letter into itself.

What read windows-1250 through it got no utf-8: the names, descriptions and categories of the sceneries and the
names of the textures in the launcher, and the descriptions of the templates in the editor. The editor writes
such a description back through utf8_to_win1250, which makes a question mark of each byte that doesn't start
a utf-8 sequence, so the letters it had read unchanged didn't come back.

The utf-8 sequences are written as bytes now, the way utf8_to_win1250 has them, so they don't depend on the
options. The options stay as they are: with /utf-8 other literals of the files with a byte order mark would
change, the "±" of the table of Bezogonkow would take two bytes and move the letters after it.

The unit tests check the letters and that the text given is valid utf-8.

Co-authored-by: maj00r <maj00r@host>
This commit is contained in:
maj00r
2026-11-05 03:12:05 +01:00
committed by GitHub
parent b7ff8bab73
commit 09b1c2de10
2 changed files with 31 additions and 2 deletions

View File

@@ -0,0 +1,26 @@
#include "stdafx.h"
#include "doctest.h"
#include "utilities/utilities.h"
// the polish letters, in windows-1250 and in utf-8; written as bytes, the build reads the sources in any code page
namespace
{
std::string const kWin1250{"\xA5\xC6\xCA\xA3\xD1\xD3\x8C\x8F\xAF\xB9\xE6\xEA\xB3\xF1\xF3\x9C\x9F\xBF"};
std::string const kUtf8{"\xC4\x84\xC4\x86\xC4\x98\xC5\x81\xC5\x83\xC3\x93\xC5\x9A\xC5\xB9\xC5\xBB"
"\xC4\x85\xC4\x87\xC4\x99\xC5\x82\xC5\x84\xC3\xB3\xC5\x9B\xC5\xBA\xC5\xBC"};
} // namespace
TEST_CASE("win1250: the polish letters of a scenery file read as utf-8") {
CHECK(win1250_to_utf8("Bia\xB3" "a Podlaska") == "Bia\xC5\x82" "a Podlaska");
CHECK(win1250_to_utf8(kWin1250) == kUtf8);
// the rest of the text stays as it is
CHECK(win1250_to_utf8("tor 1, km 0.178") == "tor 1, km 0.178");
}
TEST_CASE("win1250: what win1250_to_utf8 gives is valid utf-8") {
std::u32string decoded;
CHECK_FALSE(utf8_to_utf32(kWin1250, decoded));
REQUIRE(utf8_to_utf32(win1250_to_utf8(kWin1250), decoded));
CHECK(decoded.size() == kWin1250.size());
CHECK(decoded[12] == U'\x0142');
}