path: root/documentation/src/unicode.dox

/**

 \page unicode Unicode and UTF-8 Support

FLTK provides comprehensive Unicode support through UTF-8 encoding, allowing your applications to handle international text and be easily localized for users worldwide.

\section unicode_overview Overview

Starting with version 1.3.0, FLTK uses UTF-8 as its primary text encoding. This means:
- All text in FLTK is expected to be UTF-8 encoded
- Your application can display text in any language
- File operations work correctly with international filenames
- Most existing ASCII code continues to work unchanged

\note Unicode support in FLTK is functional but still evolving. Some advanced features like bidirectional text and complex script shaping are not yet implemented.

\section unicode_quick_start Quick Start

For most applications, you simply need to ensure your text is UTF-8 encoded:

\code
// These all work automatically with UTF-8:
Fl_Window window(400, 300, "Hello 世界");  // Mixed ASCII and Chinese
button->label("Café");                      // Accented characters
fl_fopen("документ.txt", "r");             // Cyrillic filename
\endcode

\section unicode_background What is Unicode and UTF-8?

__Unicode__ is a standard that assigns a unique number to every character used in human languages - from Latin letters to Chinese characters to emoji. Each character has a "code point" like U+0041 for 'A' or U+4E2D for '中'.

__UTF-8__ is a way to store Unicode characters as bytes. It's backward-compatible with ASCII and efficient for most text:
- ASCII characters (like 'A') use 1 byte
- European accented characters use 2 bytes
- Most other characters (Chinese, Arabic, etc.) use 3 bytes
- Rare characters and emoji may use 4 bytes

FLTK chose UTF-8 because it works well with existing C string functions and doesn't break legacy ASCII code.

\section unicode_functions Unicode Functions in FLTK

\subsection unicode_validation Text Validation and Analysis

Functions to check and analyze UTF-8 text:

fl_utf8test() - Check if a string contains valid UTF-8
\code
const char* text = "Hello 世界";
int result = fl_utf8test(text, strlen(text));
// Returns: 0=invalid, 1=ASCII, 2=2-byte chars, 3=3-byte chars, 4=4-byte chars
\endcode

fl_utf8len() - Get the byte length of a UTF-8 character
\code
char ch = '\xE4';  // First byte of a 3-byte UTF-8 sequence
int len = fl_utf8len(ch);  // Returns 3 (or -1 if invalid)
\endcode

fl_utf8locale() - Check if system uses UTF-8 encoding
\code
if (fl_utf8locale()) {
    // System uses UTF-8, no conversion needed
} else {
    // May need to convert from local encoding
}
\endcode

fl_utf_nb_char() - Count UTF-8 characters in a buffer
\code
const char* text = "Hello 世界";
int char_count = fl_utf_nb_char((unsigned char*)text, strlen(text));
// Returns 8 (number of characters, not bytes)
\endcode

fl_utf8bytes() / fl_unichar_to_utf8_size() - Get bytes needed for Unicode character
\code
unsigned int unicode_char = 0x4E2D;  // Chinese character '中'
int bytes_needed = fl_utf8bytes(unicode_char);  // Returns 3
\endcode

fl_nonspacing() - Check if character is non-spacing (combining character)
\code
unsigned int accent = 0x0300;  // Combining grave accent
if (fl_nonspacing(accent)) {
    // This is a combining character, doesn't take visual space
}
\endcode

\subsection unicode_conversion Text Conversion

Functions to convert between encodings:

fl_utf8decode() / fl_utf8encode() - Convert between UTF-8 and Unicode values
\code
// Decode UTF-8 to Unicode code point
const char* utf8_char = "中";
int len;
unsigned int unicode = fl_utf8decode(utf8_char, utf8_char + 3, &len);
// unicode = 0x4E2D, len = 3

// Encode Unicode back to UTF-8
char buffer[5];
int bytes = fl_utf8encode(0x4E2D, buffer);  // Returns 3
buffer[bytes] = '\0';  // Now buffer contains "中"
\endcode

fl_utf8froma() / fl_utf8toa() - Convert between UTF-8 and single-byte encodings
\code
// Convert ISO-8859-1 to UTF-8
char utf8_buffer[200];
fl_utf8froma(utf8_buffer, sizeof(utf8_buffer), "café", 4);

// Convert UTF-8 to single-byte (non-representable chars become '?')
char ascii_buffer[100];
fl_utf8toa("café", 5, ascii_buffer, sizeof(ascii_buffer));
\endcode

fl_utf8fromwc() / fl_utf8towc() - Convert between UTF-8 and wide characters
\code
// Convert wide string to UTF-8
wchar_t wide_text[] = L"Hello 世界";
char utf8_buffer[100];
fl_utf8fromwc(utf8_buffer, sizeof(utf8_buffer), wide_text, wcslen(wide_text));

// Convert UTF-8 to wide string
const char* utf8_text = "Hello 世界";
wchar_t wide_buffer[50];
fl_utf8towc(utf8_text, strlen(utf8_text), wide_buffer, 50);
\endcode

fl_utf8toUtf16() - Convert UTF-8 to UTF-16
\code
const char* utf8_text = "Hello 世界";
unsigned short utf16_buffer[100];
unsigned int result = fl_utf8toUtf16(utf8_text, strlen(utf8_text),
                                    utf16_buffer, 100);
// Converts to UTF-16, handling surrogate pairs on Windows
\endcode

fl_utf2mbcs() - Convert UTF-8 to local multibyte encoding
\code
const char* utf8_text = "Hello 世界";
char* local_text = fl_utf2mbcs(utf8_text);
// Converts to system's local encoding (Windows CP, etc.)
// Remember to free the returned pointer
free(local_text);
\endcode

fl_utf8from_mb() / fl_utf8to_mb() - Convert between UTF-8 and local multibyte
\code
// Convert from local multibyte to UTF-8
char utf8_buffer[200];
fl_utf8from_mb(utf8_buffer, sizeof(utf8_buffer), local_text, strlen(local_text));

// Convert from UTF-8 to local multibyte
char local_buffer[200];
fl_utf8to_mb(utf8_text, strlen(utf8_text), local_buffer, sizeof(local_buffer));
\endcode

\subsection unicode_navigation Text Navigation

Functions to move through UTF-8 text safely:

fl_utf8back() / fl_utf8fwd() - Find character boundaries
\code
const char* text = "Café";
const char* start = text;
const char* end = text + strlen(text);
const char* e_pos = text + 3;  // Points to 'é'

// Move to previous character
const char* c_pos = fl_utf8back(e_pos, start, end);  // Points to 'f'

// Move to next character
const char* next_pos = fl_utf8fwd(e_pos, start, end);  // Points after 'é'
\endcode

\subsection unicode_string_ops String Operations

UTF-8 aware string functions:

fl_utf8strlen() - Count UTF-8 characters (not bytes)
\code
const char* text = "Café";  // 5 bytes, 4 characters
int chars = fl_utf8strlen(text);  // Returns 4
int bytes = strlen(text);         // Returns 5
\endcode

fl_utf_strcasecmp() / fl_utf_strncasecmp() - Compare strings ignoring case
\code
int result = fl_utf_strcasecmp("Café", "CAFÉ");  // Returns 0 (equal)
int result2 = fl_utf_strncasecmp("Café", "CAFÉ", 2);  // Compare first 2 chars
\endcode

fl_tolower() / fl_toupper() - Convert case for individual Unicode characters
\code
unsigned int lower_a = fl_tolower(0x41);   // 'A' -> 'a' (0x61)
unsigned int upper_e = fl_toupper(0xE9);   // 'é' -> 'É' (0xC9)
\endcode

fl_utf_tolower() / fl_utf_toupper() - Convert case for UTF-8 strings
\code
const char* text = "Café";
char lower_buffer[20];
fl_utf_tolower((unsigned char*)text, strlen(text), lower_buffer);
// lower_buffer now contains "café"
\endcode

\subsection unicode_file_ops File Operations

Cross-platform file functions that handle UTF-8 filenames correctly:

__Basic file operations:__
\code
// These work with international filenames on all platforms:
FILE* f = fl_fopen("测试文件.txt", "r");        // Open file
int fd = fl_open("документ.bin", O_RDONLY);     // Open with file descriptor
int result = fl_stat("файл.dat", &stat_buf);   // Get file info
\endcode

__File access and properties:__
\code
fl_access("测试文件.txt", R_OK);           // Check if file is readable
fl_chmod("文档.dat", 0644);               // Change file permissions
fl_unlink("临时文件.tmp");                // Delete file
fl_rename("旧名.txt", "新名.txt");        // Rename file
\endcode

__Directory operations:__
\code
fl_mkdir("新文件夹", 0755);               // Create directory
fl_rmdir("旧文件夹");                     // Remove directory
char current_dir[1024];
fl_getcwd(current_dir, sizeof(current_dir));  // Get current directory
\endcode

__Path operations:__
\code
fl_make_path("新目录/子目录/深层目录");          // Create directory path
fl_make_path_for_file("路径/到/新文件.txt");     // Create path for file
\endcode

__Process and system operations:__
\code
fl_execvp("程序名", argv);                     // Execute program
fl_system("echo 'Hello 世界'");               // Execute system command
char* value = fl_getenv("环境变量");           // Get environment variable
\endcode

\section unicode_best_practices Best Practices

\subsection unicode_practices_files File Handling
- Always use fl_fopen(), fl_open(), etc. for file operations with international names
- Save source code files as UTF-8 with BOM if your editor requires it
- Test with international filenames during development

\subsection unicode_practices_strings String Processing
- Use fl_utf8strlen() instead of strlen() for character counts
- Use fl_utf8fwd()/fl_utf8back() when iterating through text character by character
- Validate user input with fl_utf8test() if accepting external data
- Be careful when truncating strings - use character boundaries, not arbitrary byte positions

\subsection unicode_practices_display Display and UI
- Test your interface with text in various languages (especially long German words or wide Asian characters)
- Consider that text length varies greatly between languages when designing layouts
- Ensure your chosen fonts support the characters you need to display

\subsection unicode_practices_performance Performance Notes
- ASCII text has no performance overhead compared to single-byte encodings
- UTF-8 functions are optimized for common cases (ASCII and Western European text)
- File operations may be slightly slower on Windows due to UTF-16 conversion

\section unicode_troubleshooting Common Issues and Solutions

\subsection unicode_problem_display "My international text shows up as question marks"
__Solution:__ Ensure your text is UTF-8 encoded and your font supports the characters. If reading from files, verify they're saved as UTF-8.

\subsection unicode_problem_files "File operations fail with international names"
__Solution:__ Use FLTK's Unicode file functions instead of standard C functions:
\code
// Instead of:
FILE* f = fopen("файл.txt", "r");  // May fail on Windows

// Use:
FILE* f = fl_fopen("файл.txt", "r");  // Works correctly
\endcode

\subsection unicode_problem_length "String length calculations are wrong"
__Solution:__ Use UTF-8 aware functions:
\code
// Wrong - counts bytes, not characters:
int len = strlen("Café");  // Returns 5

// Correct - counts characters:
int len = fl_utf8strlen("Café");  // Returns 4
\endcode

\subsection unicode_problem_truncation "Text gets corrupted when I truncate it"
__Solution:__ Don't truncate UTF-8 strings at arbitrary byte positions:
\code
// Wrong - may cut in middle of character:
char truncated[10];
strncpy(truncated, utf8_text, 9);

// Correct - find proper character boundary:
const char* end = utf8_text;
int char_count = 0;
while (char_count < max_chars && *end) {
    end = fl_utf8fwd(end, utf8_text, utf8_text + strlen(utf8_text));
    char_count++;
}
int safe_length = end - utf8_text;
\endcode

\section unicode_error_handling Error Handling

FLTK handles invalid UTF-8 sequences gracefully using configurable behavior:

__Error handling modes (compile-time configuration):__
- __ERRORS_TO_CP1252__ (default): Treats bytes 0x80-0x9F as CP1252 characters
- __STRICT_RFC3629__: Strict UTF-8 validation according to RFC 3629
- __ERRORS_TO_ISO8859_1__ (default): Invalid bytes returned as-is, otherwise returns Unicode replacement character (U+FFFD)

\note You can configure these with compiler flags like -DERRORS_TO_CP1252=0

This design allows FLTK to handle legacy text files that mix encodings, making it more robust in real-world scenarios.

\section unicode_limitations Current Limitations

FLTK's Unicode support covers most common use cases but has some limitations:

__Text Processing:__
- No automatic text normalization (combining characters are treated separately)
- No complex script shaping (may affect some Arabic, Indic scripts)
- No bidirectional text support (right-to-left languages like Arabic/Hebrew)

__Character Range:__
- Full Unicode range supported (U+000000 to U+10FFFF)
- Some legacy APIs may be limited to 16-bit characters (Basic Multilingual Plane)

__Sorting and Comparison:__
- String comparison is byte-based, not linguistically correct
- Use system locale functions for proper collation when needed for sorting

__Composed Characters:__
- Composed characters (base + combining accents) are treated as separate characters
- No automatic character composition or decomposition

Most applications won't encounter these limitations in practice. The Unicode support in FLTK is sufficient for displaying and processing international text in the majority of real-world scenarios.

\htmlonly
<hr>
<table summary="navigation bar" width="100%" border="0">
<tr>
  <td width="45%" align="LEFT">
    <a class="el" href="advanced.html">
    [Prev]
    Advanced FLTK
    </a>
  </td>
  <td width="10%" align="CENTER">
    <a class="el" href="index.html">[Index]</a>
  </td>
  <td width="45%" align="RIGHT">
    <a class="el" href="enumerations.html">
    Constants and Enumerations
    [Next]
    </a>
  </td>
</tr>
</table>
\endhtmlonly

*/