Boost.Locale: Default Encoding under Microsoft Windows

| Boost.Locale

Default Encoding under Microsoft Windows

All modern operating systems use Unicode.

- Unix operating system family use UTF-8 encoding by default.
- Microsoft Windows have migrated to Wide/UTF-16 API. The narrow encodings have been deprecated and the native OS API became the so called "Wide API"

As a result of radically different approaches, it is very hard to write portable Unicode aware applications.

Boost Locale fully supports both narrow and wide APIs. The default character encoding is assumed to be UTF-8 on Windows.

So if the default operating system Locale is "English_USA.1252" the default locale for Boost.Locale on Windows would be "en_US.UTF-8".

When the created locale object is installed globally, then any libraries that use `std::codecvt` for conversion between narrow API and the native wide API would handle UTF-8 correctly.

A good example of such library is Boost.Filesystem v3.

For example

#include <boost/locale.hpp>
#include <boost/filesystem/path.hpp>
#include <boost/filesystem/fstream.hpp>
int main()
{
// Create and install global locale
std::locale::global(boost::locale::generator().generate(""));
// Make boost.filesystem use it
boost::filesystem::path::imbue(std::locale());
// Now Works perfectly fine with UTF-8!
boost::filesystem::ofstream hello("שלום.txt");
}
boost::locale::generator
the major class used for locale generation
Definition: generator.hpp:101

However such behavior may break existing software that assumes that the current encoding is single byte encodings like code page 1252.

The boost::locale::generator class has a property use_ansi_encoding() that allows changing the behavior to the legacy one and selecting an ANSI code page as the default system encoding.

So, when the current locale is "English_USA.1252" and the `use_ansi_encoding` is turned on, then the default locale would be "en_US.windows-1252".

Note
  The `winapi` backend does not support ANSI encodings; thus UTF-8 encoding is always used for narrow characters.

- © Copyright 2009-2012 Artyom Beilis, Distributed under the Boost Software License, Version 1.0.