101035 中文信息处理Chinese NLP.pptVIP

Big5 Big5 is the most widely implemented character set standard used in Taiwan and is used for traditional Chinese characters. * 国际编码方式International Encoding Method Unicode and ISO 10646-1. We need to develop a multilingual character set combining the majority of the world’s writing systems and character sets into a Universal Character Set (UCS) or Unicode. Character Set Encoding Method Unicode and ISO 10646-1 UCS-2, UCS-4 UTF-7, UTF-8, UTF-16 * BMP The first plane (plane 0), the Basic Multilingual Plane (BMP), is where most characters have been assigned so far. The BMP contains characters for almost all modern languages, and a large number of special characters. * UCS-2 A 16-bit representation can end up to 65,536 (=216) unique code points. It allocates the entire encoding space for characters (0x0000-0xFFFF). UCS-2 and Unicode encodings are identical for most of Chinese characters. 00-FF 00-FF UCS-2 First byte range Second byte range 0-255 0-255 * UCS-4 It is a four byte (actually, 31-bit) representation which can encode up to 2,147,483,648 (=231) code points (00x7FFFFFFF). It allocates the entire encoding space for characters (0x0000-0xFFFF). 00-7F 00-FF 00-FF 00-FF UCS-4 First byte range Second byte range Third byte range Fourth byte range 0-127 0-256 0-256 0-256 * UCS-2 vs UCS-4 * 0000 … … … FFFF 65,536 code points UCS-2 (16-bit) Unicode (17-plane) 17? 256?256= 1,114,112 characters 2,147,483,648 code points… … … 0000FFFF UCS-4 (31-bit)… … … 7FFFFFFF Can only encode BMP Plane Sufficient to encode all 17 Planes of Unicode Set UTF-16 In essence, UTF-16 encodes the BMP according to UCS-2 (16 bits) encoding (compatible). But it also allows the next 16 planes, which are normally only accessible through UCS-4 (32 bits) encoding. The surrogates area is defined with UTF-16 to allow for expansion beyond the 16-bit code space. * UCS-2 vs UCS-4 vs UTF-16 *