Postgres "Invalid Byte Sequence for Encoding UTF8" on Data Import

This error means the data you're importing contains byte sequences that aren't valid UTF-8 — Postgres, when configured with UTF-8 encoding (the common, recommended default), correctly refuses to store genuinely malformed byte sequences rather than silently accepting or misinterpreting them, and the fix requires converting the source data to genuine, valid UTF-8 before import.

The Problem

Importing data — via COPY, an application insert, or a bulk load tool — fails partway through:

ERROR:  invalid byte sequence for encoding "UTF8": 0xe9 0x73
CONTEXT:  COPY customers, line 4521

The source file might appear to display correctly in some text editors or contexts, which can be misleading — some tools are lenient and attempt to guess or auto-correct the encoding when displaying content, masking the fact that the underlying bytes aren't genuinely valid UTF-8.

Why It Happens

UTF-8 is a specific, well-defined encoding scheme, and not every arbitrary sequence of bytes is valid UTF-8 — data originally encoded in a different character encoding (like Latin-1/ISO-8859-1, or Windows-1252) contains byte sequences that are simply invalid when interpreted as UTF-8, even though those same bytes were perfectly valid in their original, different encoding. Common sources of this mismatch:

• A CSV or text file exported from a different system (an older application, a Windows-based tool, a legacy database) using a different default character encoding than UTF-8, without an explicit conversion step before import
• Data that mixes multiple encodings within the same file, sometimes from being concatenated or merged from multiple sources with different original encodings
• A text editor or tool that saved a file with an encoding different from what was assumed or intended, especially common when files pass through multiple different tools or systems before reaching the final import step

The Fix

First, identify the source file's actual current encoding, rather than assuming — the file command can often provide a reasonable guess, though it's not always definitive for ambiguous cases:

file -i source_data.csv
source_data.csv: text/plain; charset=iso-8859-1

If the detected encoding is something other than UTF-8 (Latin-1/ISO-8859-1 is a very common culprit, particularly for data originating from older systems or certain European locales), convert the file to genuine UTF-8 using iconv before attempting the import:

iconv -f ISO-8859-1 -t UTF-8 source_data.csv > source_data_utf8.csv

Then import the converted file rather than the original:

\copy customers FROM 'source_data_utf8.csv' WITH CSV HEADER;

If file -i's detected encoding isn't accurate or confident enough (common with genuinely ambiguous byte sequences, since automatic encoding detection is inherently a best-effort heuristic, not a perfectly reliable determination), and you know the actual original source system or context, use that known information to determine the correct source encoding rather than relying purely on automatic detection:

# If you know the data came from a Windows-based export, for example
iconv -f WINDOWS-1252 -t UTF-8 source_data.csv > source_data_utf8.csv

If the file genuinely mixes multiple encodings (a harder scenario to fix cleanly), you may need to identify and handle the specific problematic sections separately, or, if the affected rows are a small enough number relative to the overall dataset, consider whether excluding those specific genuinely malformed rows (after review) is an acceptable resolution rather than attempting a complex mixed-encoding conversion across the entire file.

For future imports from the same recurring source, address the encoding issue at the source if you control it — configuring whatever system originally generates the export to use UTF-8 consistently prevents this issue from recurring on every subsequent import, rather than needing a conversion step every single time.

If you're building an ongoing import pipeline (not a one-time fix), add an explicit encoding validation and conversion step as part of the pipeline itself, so any future file with encoding issues is caught and handled consistently and automatically, rather than causing an import failure that requires manual investigation each time it recurs:

# As part of an automated import script
iconv -f "$(file -b --mime-encoding "$input_file")" -t UTF-8 "$input_file" > "$converted_file" 2>/dev/null || echo "Conversion failed, manual review needed"

Still Not Working?

If iconv conversion itself fails or produces unexpected results, the source file may contain genuinely corrupted or truly invalid byte sequences that don't correspond cleanly to any standard encoding at all — in this case, use iconv's error-handling options to skip or substitute invalid sequences rather than failing the conversion entirely, accepting some data loss for the specific malformed portions in exchange for being able to successfully import the remainder of otherwise valid data:

iconv -f ISO-8859-1 -t UTF-8//IGNORE source_data.csv > source_data_utf8.csv

The //IGNORE suffix causes iconv to silently drop characters it can't convert rather than failing outright — review the resulting file afterward to understand the extent of what was actually dropped, and confirm that's an acceptable tradeoff for your specific data before relying on this approach for anything where data completeness genuinely matters.