UTF-8 text processing (UTF8PAT legacy)

The UTF-8 encoding library allows you to process data files that contain UTF-8 encoding.

You can use the pattern macro utf8-char to match UTF-8 characters in data.

You can use the pattern macro single-byte-utf8-char to match only ASCII UTF-8 characters in data.

You can use the pattern macro multi-byte-utf8-char to match only double-byte UTF-8 characters in data.

You can use the function utf8-char-number to convert a UTF-8 character (that is, a sequence of bytes that represents a character in UTF-8) to its binary character value.

You can use the function utf8-encoded to convert a binary character value to UTF-8 (that is, to that sequence of bytes which represents that character value in UTF-8).

Example:

  include "utf8pat.xin"
    process
    repeat scan "flamb%195#%169#"
         match single-byte-utf8-char+ => c
              output c
         match multi-byte-utf8-char => c
              local integer n initial {utf8-char-number c}
              do when n > 255
                   output "&#x" || "16rud" % n || ";"
              else
                   output "b" % n
              done
      again

To use these functions, you must include the following code near the top of your program:

  include "utf8pat.xin"

Functions