Practical .NET · Lesson 19

UTF-8 bytes and text boundaries

Measure encoded data separately from the number of UTF-16 code units.

How it works

A .NET string stores UTF-16 code units, while an exported text payload may use UTF-8 bytes. String.Length and the byte count of an encoded message therefore answer different questions. A character outside the basic ASCII range can occupy more than one UTF-8 byte, and some Unicode scalar values require two UTF-16 code units in a string.

Choose the unit that matches the limit or protocol. A user-facing character limit is not automatically a byte limit, and neither is automatically the same as a count of Unicode text elements. This lesson focuses on a concrete encoding boundary: converting a prepared string to UTF-8 and decoding it back. No network or file access is needed to reason about the bytes.

Walk through the example. The string contains an ASCII letter followed by an accented letter. Its UTF-16 length is two code units, while UTF-8 encoding produces three bytes. The hexadecimal representation makes those bytes visible. Decoding the bytes reconstructs the original string, and ordinal equality confirms the round trip. The explicit UTF8Encoding instance requests exceptions for invalid byte sequences rather than silently substituting replacement text.

Watch the boundary. Cutting a byte array at an arbitrary size can split a multibyte character. Likewise, cutting a string at an arbitrary UTF-16 index can split a surrogate pair. Do not implement a text-length policy by truncating whichever representation happens to be nearby. Keep data as bytes only where the byte-level contract requires it, and choose an explicit error policy for invalid encoded input.

Practice independently. Compare UTF-16 length and UTF-8 byte length for an ASCII word, an accented word, and an emoji. Predict which measurements differ before running. Then attempt strict decoding of an invalid prepared byte sequence and handle the decoding error explicitly. Explain why preserving an entire encoded character is different from preserving a complete user-perceived grapheme such as a letter with a combining mark.

A working example

using System;
using System.Collections.Generic;
using System.Linq;
using System.Globalization;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Text.RegularExpressions;

string text = "Aé";
var utf8 = new UTF8Encoding(false, true);
byte[] bytes = utf8.GetBytes(text);
Console.WriteLine(text.Length);
Console.WriteLine(bytes.Length);
Console.WriteLine(Convert.ToHexString(bytes));
Console.WriteLine(utf8.GetString(bytes).Equals(text, StringComparison.Ordinal));

Expected output

2
3
41C3A9
True
Try this
State whether a limit concerns UTF-16 units, encoded bytes, Unicode scalars, or text elements.