Data and Formats · Lesson 8

UTF-8 is bytes

Distinguish a string’s characters from its encoded length.

10 min guided lesson · Read, predict, then run

Text and encoded bytes are different representations

A .NET string contains text represented through UTF-16 code units, while a byte-oriented format requires an encoding. UTF-8 maps Unicode text into a sequence of bytes suitable for storage or transport. Character positions and byte counts are not interchangeable measurements. This sample uses a short word with one non-ASCII letter so the difference appears without any network or file operations. A limit expressed in bytes must be checked after choosing the encoding, rather than inferred from a string’s displayed length.

Count the bytes of the actual word

The literal café here uses a precomposed accented e. The ASCII letters c, a, and f each require one UTF-8 byte, while that accented letter requires two, giving a total of five. GetBytes creates the encoded byte array, and bytes.Length prints 5. GetString decodes those bytes using the same encoding and prints café. This is a representation round trip for the given valid text; the first line is not the number of letters shown on the screen.

var bytes = Encoding.UTF8.GetBytes(text);

A round trip needs a compatible encoding

Encoding and decoding must agree about how bytes represent text. A byte array does not carry an automatic universal guarantee that every consumer will interpret it as UTF-8. A protocol or file format needs to state that contract, including whether an optional marker or other metadata is relevant. This sample keeps the encoding explicit on both operations, so the relationship is visible. The same principle applies when a browser sends bytes elsewhere: a correct local conversion cannot repair a receiver that assumes an incompatible representation.

One visible symbol may have several internal units

Even counting Unicode scalar values is not always the same as counting user-perceived characters. Some visible text uses combining marks or several scalars, and supplementary scalars require two UTF-16 code units. Do not generalize this word’s four-letter appearance into a rule that string.Length always equals a display count. The browser’s invariant globalization configuration also should not be used to promise culture-dependent normalization behavior. For this lesson, keep the claim narrow: explicit UTF-8 encoding of this exact literal produces five bytes and decodes back to the same text.

Practice: compare ASCII and supplementary text

Replace café with cafe and predict four bytes. Then use a string containing A followed by one supplementary smiling-face scalar and compare UTF-16 length with UTF-8 byte length. Record the units beside every number rather than calling both measurements characters. Restore the baseline afterward. These independent inputs expose two reasons text length and byte length differ: a non-ASCII scalar can need multiple encoded bytes, and a supplementary scalar also occupies more than one UTF-16 position.

  • Use Encoding.UTF8.GetByteCount when only the byte budget is needed, avoiding an unnecessary byte-array allocation.
  • Do not split an arbitrary string at a UTF-16 index and assume a surrogate pair remains intact.

Practice: decide how invalid byte input is handled

For a separate boundary exercise, try decoding a deliberately invalid UTF-8 byte sequence and inspect the decoder’s fallback policy. Compare the usual replacement behavior with an explicitly configured strict UTF8Encoding that throws on invalid bytes. Decide which policy an import format requires, and report malformed representation separately from valid but unacceptable domain text. A successful decode with replacement characters can be useful for display, but it should not silently prove that the original wire data was fully valid under a strict protocol.

  • Keep malformed-byte tests separate from valid text that merely contains non-ASCII symbols.
  • State the encoding and fallback policy in the format contract rather than relying on a convenient default.

Name the unit at every data boundary

Encoding work becomes easier to verify when variables and tests say whether they concern UTF-16 units, Unicode scalars, displayed text, or encoded bytes. The baseline uses deterministic managed APIs available in SharpRune and makes no desktop-only assumptions. Extend it by defining a real byte budget or decode policy, then test exact input representations. The most important invariant is not that text always has one obvious length, but that every component uses the same agreed encoding and interprets each reported count in the correct unit.

  • Baseline output is 5 and then café.
  • The accented letter contributes two UTF-8 bytes in this exact literal.
  • Encoding, decoding fallback, normalization, and display-character segmentation are separate policies.

A working example

using System;
using System.Linq;
using System.Collections.Generic;
using System.Globalization;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;

var text = "café";
var bytes = Encoding.UTF8.GetBytes(text);
Console.WriteLine(bytes.Length);
Console.WriteLine(Encoding.UTF8.GetString(bytes));
Open in playground

Expected output

5
café
Try this
A byte limit should measure encoded bytes, not string length.

Further reading