PowerShell on Legacy Chinese Windows: UTF-8 with BOM or Bust

Windows PowerShell 5.1 does not guess encodings. When it loads a .ps1 file, it looks for a byte-order mark. If it finds one, it decodes accordingly. If it does not, it falls back to the system's ANSI code page — and on a Chinese Windows install, that code page is GBK (CP936). Every non-ASCII character in your script is then decoded through the wrong table, and you get mojibake that does not throw an error. It just quietly changes what your script means. This is the story of how a path check turned into a week of debugging, and the one rule that fixed it: every .ps1 file ships as UTF-8 with BOM and CRLF endings. The symptom A deployment script had a line like this: $configDir = Join-Path $env:ProgramData "我的工具\配置" if (-not (Test-Path $configDir)) { New-Item -ItemType Directory -Path $configDir -Force | Out-Null } Enter fullscreen mode Exit fullscreen mode On the developer's machine it worked. On the build server it created a directory named something like 我的工具 — or sometimes it created nothing at all, and Test-Path returned $false for a path that clearly existed when inspected manually in Explorer. The reason is straightforward once you see the bytes. The file was saved by an editor as UTF-8 without BOM. PowerShell 5.1 read it as GBK. The UTF-8 bytes for 我 are E6 88 91. Decoded as GBK, those three bytes are a valid two-byte character plus a stray byte, producing a different string entirely. PowerShell then passed that corrupted string to the Windows API, which dutifully created a directory with a nonsense name. Test-Path on the intended name failed because the intended name was never used. No exception. No warning. Just a different program than the one you wrote. Detecting it: hex-dump the first bytes The fastest way to confirm the diagnosis is to look at the raw bytes at the start of the file. # Show the first 8 bytes of a script file $bytes = [System.IO.File]::ReadAllBytes(".\deploy.ps1")[0..7] ($bytes | ForEach-Object { $_.ToString("X2") }) -join ' ' Enter fullscreen mode Exit fullscreen mode Interpretation: EF BB BF — UTF-8 with BOM. PowerShell 5.1 decodes correctly. FF FE — UTF-16 LE with BOM. Also fine. FE FF — UTF-16 BE with BOM. Fine, rare. Anything else — no BOM. PowerShell 5.1 falls back to the ANSI code page. On a GBK system that means GBK. If you see E6 88 91 at the start of a file containing 我, you have a BOM-less UTF-8 file, and 5.1 is about to misread it. A quick Python check across a whole directory is often easier than doing this by hand: from pathlib import Path BOMS = { b"\xef\xbb\xbf": "utf-8-sig", b"\xff\xfe": "utf-16-le", b"\xfe\xff": "utf-16-be", } def sniff(path: Path) -> str: head = path.read_bytes()[:4] for bom, enc in BOMS.items(): if head.startswith(bom): return enc # No BOM. Try strict UTF-8; if it fails, it is almost certainly ANSI/GBK. try: path.read_bytes().decode("utf-8") return "utf-8 (NO BOM) None: raw = path.read_bytes() if raw.startswith(b"\xef\xbb\xbf"): return text = raw.decode(source_encoding) text = text.replace("\r\n", "\n").replace("\n", "\r\n") path.write_bytes(b"\xef\xbb\xbf" + text.encode("utf-8")) for p in Path(sys.argv[1]).rglob("*.ps1"): try: to_utf8_bom(p, "utf-8") except UnicodeDecodeError: to_utf8_bom(p, "gbk") print(f"converted {p}") Enter fullscreen mode Exit fullscreen mode The fallback to gbk on UnicodeDecodeError handles mixed repos where some files were saved as ANSI by an older editor. Once every file is UTF-8 with BOM, the ambiguity is gone. PowerShell 7 behaves differently PowerShell 7 (Core) changed the defaults. Get-Content and the script loader default to UTF-8 without requiring a BOM, and Set-Content writes UTF-8 without BOM by default. So the same BOM-less file that 5.1 mangles will load correctly in 7. That is a feature, and also a trap. If your scripts run under both 5.1 and 7 — which is common on servers that still have Windows PowerShell installed alongside Core — you need the file to work under the stricter reader. UTF-8 with BOM works in both. BOM-less UTF-8 works only in 7. So the rule is not "use UTF-8"; the rule is "use UTF-8 with BOM" if any 5.1 host might touch the file. Note also that PowerShell 7 still honors the BOM when present, so adding it does not break anything under Core. A pre-flight check before you ship The fix is cheap; the discipline is what fails. Put a validator in your build or pre-commit step so a BOM-less file never reaches a 5.1 host. #!/usr/bin/env python3 """Fail the build if any .ps1 is not UTF-8 with BOM and CRLF.""" import sys from pathlib import Path def check(path: Path) -> list[str]: errors = [] raw = path.read_bytes() if not raw.startswith(b"\xef\xbb\xbf"): errors.append("missing UTF-8 BOM") else: try: raw[3:].decode("utf-8") except UnicodeDecodeError as exc: errors.append(f"not valid UTF-8: {exc}") # CRLF check: reject lone LF outside of the final byte. body = raw[3:] if raw.startswith(b"\xef\xbb\xbf") else raw if b"\n" in body.replace(b"\r\n", b""): errors.append("contains LF-only line endings; expected CRLF") return errors failed = False for p in sorted(Path(".").rglob("*.ps1")): errs = check(p) if errs: failed = True for e in errs: print(f"{p}: {e}") sys.exit(1 if failed else 0) Enter fullscreen mode Exit fullscreen mode Drop this into CI as a gate. The failure message is specific enough that the fix is obvious: save the file as UTF-8 with BOM, convert line endings to CRLF, commit. If you want the validator to auto-fix instead of fail, reuse the conversion functions above and re-run the check. In a pre-commit hook, auto-fixing is usually the better default — it removes the human from the loop entirely. The rule Every .ps1 file that might ever run under Windows PowerShell 5.1 must be saved as UTF-8 with BOM and CRLF line endings. BOM-less UTF-8 is a valid encoding and a broken script on a GBK system. UTF-16 also works but is unusual in a repo and confuses diffs and tooling. Stick to UTF-8 with BOM. The failure mode — silent mojibake, locale-dependent behavior, no exception — is what makes this worth a hard rule rather than a convention. A linter catches it in a second; a human catches it after a week of Test-Path returning $false on a path that exists. More notes like this ship every week on this site.