1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
/**
\page unicode Unicode and UTF-8 Support
This chapter explains how FLTK handles international
text via Unicode and UTF-8.
Unicode support was only recently added to FLTK and is
still incomplete. This chapter is Work in Progress, reflecting
the current state of Unicode support.
\section unicode_about About Unicode, ISO 10646 and UTF-8
The summary of Unicode, ISO 10646 and UTF-8 given below is
deliberately brief, and provides just enough information for
the rest of this chapter.
For further information, please see:
- http://www.unicode.org
- http://www.iso.org
- http://en.wikipedia.org/wiki/Unicode
- http://www.cl.cam.ac.uk/~mgk25/unicode.html
\par The Unicode Standard
The Unicode Standard was originally developed by a consortium of mainly
US computer manufacturers and developers of mult-lingual software.
It has now become a defacto standard for character encoding,
and is supported by most of the major computing companies in the world.
Before Unicode, many different systems, on different platforms,
had been developed for encoding characters for different languages,
but no single encoding could satisfy all languages.
Unicode provides access to over 100,000 characters
used in all the major languages written today,
and is independent of platform and language.
Unicode also provides higher-level concepts needed for text processing
and typographic publishing systems, such as algorithms for sorting and
comparing text, composite character and text rendering, right-to-left
and bi-directional text handling.
<i>There are currently no plans to add this extra functionality to FLTK.</i>
\par ISO 10646
The International Organisation for Standardization (ISO) had also
been trying to develop a single unified character set.
Although both ISO and the Unicode Consortium continue to publish
their own standards, they have agreed to coordinate their work so
that specific versions of the Unicode and ISO 10646 standards are
compatible with each other.
The international standard ISO 10646 defines the
<b>Universal Character Set</b> (UCS)
which contains the characters required for almost all known languages.
The standard also defines three different implementation levels specifying
how these characters can be combined.
<i>There are currently no plans for handling the different implementation
levels or the combining characters in FLTK.</i>
In UCS, characters have a unique numerical code and an official name,
and are usually shown using 'U+' and the code in hexadecimal,
e.g. U+0041 is the "Latin capital letter A".
The UCS characters U+0000 to U+007F correspond to US-ASCII,
and U+0000 to U+00FF correspond to ISO 8859-1 (Latin1).
The UCS also defines various methods of encoding characters as
a sequence of bytes.
UCS-2 encodes Unicode characters into two bytes,
which is wasteful if you are only dealing with ASCII or Latin1 text,
and insufficient if you need characters above U+00FFFF.
UCS-4 uses four bytes, which lets it handle higher characters,
but this is even more wasteful for ASCII or Latin1.
\par UTF-8
The Unicode standard defines various UCS Transformation Formats.
UTF-16 and UTF-32 are based on units of two and four bytes.
UTF-8 encodes all Unicode characters into variable length
sequences of bytes. Unicode characters in the 7-bit ASCII
range map to the same value and are represented as a single byte,
making the transformation to Unicode quick and easy.
All UCS characters above U+007F are encoded as a sequence of
several bytes. The top bits of the first byte are set to show
the length of the byte sequence, and subseqent bytes are
always in the range 0x80 to 8x8F. This combination provides
some level of synchronisation and error detection.
<table summary="Unicode character byte sequences" align="center">
<tr>
<td>Unicode range</td>
<td>Byte sequences</td>
</tr>
<tr>
<td><tt>U+00000000 - U+0000007F</tt></td>
<td><tt>0xxxxxxx</tt></td>
</tr>
<tr>
<td><tt>U+00000080 - U+000007FF</tt></td>
<td><tt>110xxxxx 10xxxxxx</tt></td>
</tr>
<tr>
<td><tt>U+00000800 - U+0000FFFF</tt></td>
<td><tt>1110xxxx 10xxxxxx 10xxxxxx</tt></td>
</tr>
<tr>
<td><tt>U+00010000 - U+001FFFFF</tt></td>
<td><tt>11110xxx 10xxxxxx 10xxxxxx 10xxxxxx</tt></td>
</tr>
<tr>
<td><tt>U+00200000 - U+03FFFFFF</tt></td>
<td><tt>111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx</tt></td>
</tr>
<tr>
<td><tt>U+04000000 - U+7FFFFFFF</tt></td>
<td><tt>1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx</tt></td>
</tr>
</table>
Moving from ASCII encoding to Unicode will allow all new FLTK
applications to be easily internationalized and and used all
over the world. By choosing UTF-8 encoding, FLTK remains
largely source-code compatible to previous iteration of the
library.
\section unicode_in_fltk Unicode in FLTK
FLTK will be entirely converted to Unicode in UTF-8 encoding.
If a different encoding is required by the underlying operatings
system, FLTK will convert string as needed.
It is important to note that the initial implementation of
Unicode and UTF-8 in FLTK involves three important areas:
- provision of Unicode character tables and some simple related functions;
- conversion of char* variables and function parameters from single byte
per character representation to UTF-8 variable length characters;
- modifications to the display font interface to accept general
Unicode character or UCS code numbers instead of just ASCII or Latin1
characters.
The current implementation of Unicode / UTF-8 in FLTK will impose
the following limitations:
- FLTK will only handle single characters, so composed characters
consisting of a base character and floating accent characters
will be treated as multiple characters;
- FLTK will only compare or sort strings on a byte by byte basis
and not on a general Unicode character basis;
- FLTK will not handle right-to-left or bi-directional text;
\par TODO:
\li more doc on unicode, add links
\li write something about filename encoding on OS X...
\li explain the fl_utf8_... commands
\li explain issues with Fl_Preferences
\li why FLTK has no Fl_String class
\par DONE:
\li initial transfer of the Ian/O'ksi'D patch
\li adapted Makefiles and IDEs for available platforms
\li hacked some Unicode keybard entry for OS X
\par ISSUES:
\li IDEs:
- Makefile support: tested on Fedora Core 5 and OS X, but heaven knows
on which platforms this may fail
- Xcode: tested, seems to be working (but see comments below on OS X)
- VisualC (VC6): tested, test/utf8 works, but may have had some issues
during merge. Some additional work needed (imm32.lib)
- VisualStudio2005: tested, test/utf8 works, some addtl. work needed
(imm32.lib)
- VisualCNet: sorry, I have no longer access to that IDE
- Borland and other compiler: sorry, I can't update those
\li Platforms:
- you will encounter problems on all platforms!
- X11: many characters are missing, but that may be related to bad
fonts on my machine. I also could not do any keyboard tests yet.
Rendering seems to generally work ok.
- Win32: US and German keyboard worked ok, but no compositing was
tested. Rendering looks pretty good.
- OS X: redering looks good. Keyboard is completely messed up, even in
US setting (with Alt key)
- all: while merging I have seen plenty of places that are not
entirley utf8-safe, particularly Fl_Input, Fl_Text_Editor, and
Fl_Help_View. Keycodes from the keyboard conflict with Unicode
characters. Right-to-left rendered text can not be marked or edited,
and probably much more.
\htmlonly
<hr>
<table summary="navigation bar" width="100%" border="0">
<tr>
<td width="45%" align="LEFT">
<a class="el" href="advanced.html">
[Prev]
Advanced FLTK
</a>
</td>
<td width="10%" align="CENTER">
<a class="el" href="main.html">[Index]</a>
</td>
<td width="45%" align="RIGHT">
<a class="el" href="enumerations.html">
FLTK Enumerations
[Next]
</a>
</td>
</tr>
</table>
\endhtmlonly
*/
|
