A header-only C++23 library for using std::u8string as the one true string
type: formatting, printing, and std::expected-based error handling, with
adapters for the std::string edges (third-party libraries, iostreams).
C++20 introduced char8_t/std::u8string to mark UTF-8 text in the type
system, but the standard library has not caught up: std::format and
std::print accept neither u8 format strings nor u8 arguments, ostreams
deleted their char8_t overloads, and nothing helps carry UTF-8 error
messages through std::expected pipelines. In practice that pushes code back
to std::string and erases the encoding guarantee the type was created for.
u8io fills that gap so a codebase can be u8string-first everywhere, dropping
to std::string only at the edges where third-party APIs demand it.
#include <u8io/u8io.hpp>
std::expected<std::u8string, u8io::error> load_greeting(const std::filesystem::path& p) {
auto text = u8io::read_file(p); // expected<u8string, error>
if (!text) return u8io::fail_with(text.error(), u8"greeting unavailable");
return u8io::format(u8"héllo, {}!", *text); // compile-time checked
}
int main() {
auto greeting = load_greeting(u8"greeting.txt");
u8io::println(u8"{}", greeting); // value, or "unexpected(<message chain>)"
}The standard leaves no legal extension point: std::formatter<std::u8string, char> specializes a standard template purely with standard types (ill-formed,
no diagnostic required), and char8_t is not a supported format context
character type. u8io therefore transcodes at the boundary — which is free,
see below — and forwards to std::vformat. Compile-time format-string
checking is preserved: u8io::u8format_string's consteval constructor
re-runs the standard library's own checker against the mapped argument types.
- C++23 with
<format>and<expected>(MSVC 19.37+, GCC 14+, Clang/libc++ 18+, Apple Clang 16+). Note: Clang 18 paired with libstdc++ does not work — libs 8E75 tdc++ disablesstd::expectedfor Clang older than 19 (__cpp_conceptsfeature gate); use Clang 19+, or libc++ via-stdlib=libc++. - The ordinary literal encoding must be UTF-8. Enforced by
static_assert; on MSVC compile with/utf-8(the CMake target adds it automatically). This is what makeschar8_t↔charreinterpretation free andstd::print's Unicode console path work on Windows. - Trust by convention:
u8stringcontents are presumed valid UTF-8.as_u8/to_u8stringdo not validate. Callu8io::validate()where bytes enter from outside;read_filevalidates by default.
| Header | Contents |
|---|---|
u8io/cast.hpp |
as_char, as_u8 (zero-cost views), to_string, to_u8string (copies), validate |
u8io/format.hpp |
format, format_to, formatted_size, runtime_format, the arg_mapper extension point |
u8io/print.hpp |
print, println to stdout or any FILE*, via std::vprint_unicode when available |
u8io/error.hpp |
error (message + source_location + error_code + cause chain), fail, fail_with, u8unexpected, from_edge, from_error_code, formatting for error and std::expected |
u8io/io.hpp |
write_to(ostream, …), read_file, write_file, read_line (console-aware on Windows), formattable std::filesystem::path |
u8io/text.hpp |
code_points(u8string_view) — a forward range of char32_t scalar values (ill-formed input yields U+FFFD) |
u8io/transcode.hpp |
UTF-8 ↔ UTF-16/UTF-32: to_u16string, to_u32string, to_u8string — strict (expected<…, decode_error>) and _lossy (U+FFFD) flavors; encode(char32_t)/append for single code points; on Windows to_wstring, as_wide/as_u16 |
u8io/ascii.hpp |
constexpr, locale-free ASCII classification and case conversion for char8_t (is_alpha, is_digit, to_lower, …) |
u8io/charconv.hpp |
to_chars / from_chars for char8_t* buffers and u8string_view input; parse<T> → expected<T, error> for whole-string numbers |
u8io/hash.hpp |
string_hash (transparent), u8string_map<T>, u8string_set — heterogeneous lookup without temporary keys |
Formatting accepts u8string, u8string_view, u8 literals, and lone
char8_t alongside all ordinary std::format argument types; string format
specs (width, precision, alignment) apply to the mapped UTF-8 code units.
std::expected<T, E> formats as the bare value (nothing, for void) or
unexpected(<error>); u8io::error formats as its full cause chain
(top: mid: root).
To make your own type formattable through u8io::format, specialize
u8io::arg_mapper<T> to map it to a char-formattable type.
- iostreams: C++20 deleted
operator<<forchar8_ton purpose. Writeos << u8io::as_char(text)oru8io::write_to(os, text). std::u8fstream:u8iodeliberately provides no stream drop-ins. The standard leftbasic_fstream<char8_t>unusable on purpose — no standard locale supplies acodecvt<char8_t, char, mbstate_t>facet, so a real replacement means reimplementingfilebufand the stream layer, and inheriting the failbit/badbit error model thatu8io'sstd::expected-based errors exist to avoid. The committee's own direction for new code isstd::print/std::println(C++23) andstd::scan(P1729), not streams. Useread_file/write_filefor whole files,read_linefor line input,u8io::printto aFILE*for incremental output, andwrite_towhen an API hands you anostream.- The Windows API boundary (
wchar_t): Win32 W-functions speak UTF-16.to_wstring(text)outbound,to_u8string(wide)inbound — strict variants returnexpectedand report the offset of the first ill-formed sequence,_lossyvariants substitute U+FFFD and always succeed.as_wide/as_u16reinterpret betweenwchar_tandchar16_tviews for free. All of it is locale-free, deterministic, and needs no<windows.h>. stdin:u8io::read_line()reads one line as UTF-8. On Windows, real console input is read natively viaReadConsoleWand transcoded — the input-side counterpart ofstd::print's Unicode console path — so typed text arrives as UTF-8 regardless of the console code page. Redirected input is read as raw bytes and validated by default, likeread_file.- Third-party APIs taking/returning
std::string:as_char/to_stringoutbound,to_u8string/validateinbound,from_edgeforstd::expected<T, std::string>/std::error_codereturns. std::filesystem::path: construct fromu8stringand read back with.u8string()— already lossless in the standard;u8ioadds formatting.- Regex:
u8iodeliberately does not wrap<regex>. Byte-levelstd::regexgives wrong answers on non-ASCII text for anything beyond literal patterns (., character classes, and case-insensitivity operate on code units), and its performance is poor. For ASCII-literal patterns,u8io::as_char()the input; for real Unicode matching, use RE2, PCRE2 (PCRE2_UTF), or ICU.
cmake -B build && cmake --build build && ctest --test-dir buildWith just installed, the common chores are one word each:
just test # configure, build, and run the tests
just format # clang-format all headers and sources in-place
just install [prefix] # install headers + CMake package (find_package(u8io))just alone lists the recipes; builddir and config can be overridden,
e.g. just config=Release test.
- A lone
char8_targument formats as a single code unit (byte), which is only a whole character for ASCII. error_code::message()text is used as-is; with non-English system locales on Windows it may not be UTF-8 (standard categories emit ASCII).- No grapheme/normalization/width awareness — that is ICU's job. Format
width/precision count code units, exactly as
std::formatdoes forchar;code_points()yields scalar values, which are still not user-perceived characters. ascii::classification and case conversion are ASCII-only by design; non-ASCII bytes are in no class and case-convert to themselves.read_line's Windows console path transcodes lossily (unpaired UTF-16 surrogates become U+FFFD) and treats Ctrl-Z at the start of a line as end of input, matching console conventions.
Release history is in CHANGELOG.md. u8io follows Semantic Versioning; while the version is below 1.0.0 the public API may change between minor versions.
MIT.