Introduction to Data Cleaning Before Building a Lao RAG — How to Approach Text Normalization

Introduction to Data Cleaning Before Building a Lao RAG — How to Approach Text Normalization

Introduction

Before feeding Lao-language documents into RAG, you need to carry out a continuous sequence of natural language processing preprocessing steps—character encoding unification, removal of notation variants, word segmentation, and validation—and the search accuracy varies depending on which step is skipped. This article, aimed at AI implementation personnel handling Lao-language internal documents and customer data, presents concrete procedures ranging from encoding detection, dictionary-based unification of variant notations, and language-specific segmentation, through to validation using metrics such as recall@10. By the end of this article, you will be able to apply each step of text normalization to your own dataset and verify the quality of the processing yourself.

If you proceed to normalization and segmentation while overlooking disorder in character encoding and variation in notation, rework will occur repeatedly in later stages. Before starting work, you must always separate and grasp whether there are issues with the document's script and encoding, as well as spelling variation due to dialect or generational differences.

Check the Character Encoding and Input Method of Target Documents

First, check whether an already-declared encoding remains in the document. If there is no mention in the file's metadata or in the header of an email attachment, the encoding may be implicitly determined by the sending system or OS settings, so it cannot be judged by appearance alone.

If there is no declaration, use a tool that estimates it from the byte sequence. In Python, use chardet to read a sample file and check the estimation result and confidence level.

python
1import chardet 2 3with open("sample.txt", "rb") as f: 4raw = f.read() 5result = chardet.detect(raw) 6print(result) # {'encoding': 'UTF-8', 'confidence': 0.99, ...}

If the confidence is low, or if the estimated result is something other than UTF-8 (such as a different code page, which is common in documents created with Windows-based tools), do not proceed based on automatic detection alone. After decoding by trusting the estimation result, visually check a few samples to confirm that Lao characters are displayed correctly. Rather than checking just one file, building a sample set by extracting from multiple files from different sources makes it easier to spot encoding biases by acquisition route.

Also check the input method (whether it originates from OCR, keyboard input, or copy-and-paste). Documents converted via OCR tend to have character misrecognition and extraneous symbols mixed in, and have a different type of noise than documents entered via keyboard, so listing the origin of each document provides a basis for judging which rules to prioritize in subsequent noise removal and segmentation. For OCR documents with low scan image quality, it is safer to increase the number of visual checks compared to documents from other input methods.

Understand Common Notation Variation Patterns Specific to Lao

It is generally said that the 1975 spelling reform organized the Lao orthographic system into phonologically consistent rules, but old documents retaining pre-reform notation, and documents with spelling that differs by generation, still exist today. When feeding old contracts and meeting minutes accumulated internally into RAG, you need to infer from the document's creation date which notation system—pre-reform or post-reform—is mixed in, and confirm this with samples.

Notation variation in loanwords is another point that is easy to overlook. There are loanwords, such as vocabulary derived from French, where the same word is written with multiple spellings; if this is not normalized before being passed through embedding, the same word is treated as a separate vector, causing search omissions. Place names, personal names, and technical terms similarly tend to have multiple coexisting notations, so identifying which vocabulary is prone to variation for each document group can reduce omissions in dictionary construction in later stages.

Another point to note is that spaces between words do not necessarily correspond to word boundaries. In Lao sentences, spaces are sometimes used to mark clause or phrase boundaries, and may not function as word-level delimiters. Since this characteristic directly affects the subsequent word segmentation process, you need to understand at this stage that word boundaries cannot be determined by spacing alone.

The frequency of these patterns differs by document. By listing, on a count basis, the degree to which pre-reform notation, dialect, and loanword notation variation are mixed for each document type—contracts, meeting minutes, customer inquiries, and so on—you can more easily prioritize dictionary construction and rule design in later stages.

Step 1: How to Unify Character Codes and Encoding?

Step 1: How to Unify Character Codes and Encoding?

Unifying character encoding can maintain reproducibility by proceeding in the fixed order of detection, decoding, mojibake correction, and normalization. If you skip this order and proceed to dictionary unification or segmentation, the same word will remain treated as a different string downstream.

Steps to Apply Unicode Normalization (NFC/NFD)

Even when Lao characters look identical, they may be internally represented either as a single precomposed character or as a combination of a base character and a combining mark, and if the sequence of code points differs, the strings are treated as distinct even though they appear the same. If this representational variation is not unified after decoding, the same word can be converted into different vectors, causing omissions in embedding search. Unicode normalization is used to unify this, and in practice, the basic policy is to use NFC (Normalization Form Canonical Composition), which aligns everything to the precomposed representation.

In Python, you apply NFC to the decoded string using the standard library's unicodedata.normalize.

python
1import unicodedata 2 3text = unicodedata.normalize("NFC", decoded_text)

When dealing with large-scale data or multilingual mixed corpora, another option is to use ICU's Normalizer2. Since the older Normalizer API has been deprecated, new code should use the Normalizer2-based API. It is important to decide on one approach for the project in advance and avoid mixing the two partway through.

The order of application also requires attention. Normalization should be performed after mojibake correction and before dictionary-based variant unification. If normalization is applied while mojibake remains, the corrupted sequence of code points will be normalized as-is, risking the creation of an unrecoverable string. Conversely, if normalization is deferred until after dictionary unification, representational differences that existed before normalization will slip past the dictionary matching, leaving unresolved variants behind.

Detecting and Converting Mixed Character Encodings

Within a single file, correctly decoded strings and strings that were loaded with a different encoding and became corrupted (mojibake) can sometimes coexist. This tends to occur when documents collected from multiple systems are merged, or when text that was once decoded with the wrong encoding is re-saved under yet another encoding; in such cases, detection with chardet alone may yield split confidence scores among candidates, making it impossible to reach a clear judgment.

In this situation, a practical approach is to have chardet list multiple encoding candidates and then use ftfy to restore the corrupted sequence of code points. ftfy detects patterns where a string that should have been correct became corrupted by passing through a different encoding, and restores it to a form close to the original.

python
1import ftfy 2 3fixed_text = ftfy.fix_text(mojibake_text)

However, since ftfy's correction results are not perfect, you should visually inspect the restored strings—using samples that include edge cases—to confirm that no unnatural sequences of code points remain as Lao characters. For lines that cannot be fully restored, it is safer to extract them on a per-file basis and trace back through the original data's acquisition route for manual verification.

As criteria for judgment, check whether, after applying ftfy, the Lao character code points fall within a consistent range, and whether fragmentary symbols or control characters—typical signs of mojibake—remain. For files where the proportion of unrestorable lines exceeds a certain threshold, rather than relying on automated processing, it is better to check the output settings of the original system and re-acquire the data, which prevents rework in later stages. Keeping a record of the restoration success rate for each acquisition route provides a useful basis for judgment when documents continue to arrive from the same source.

Step 2: How to Remove Notation Variations and Noise?

Step 2: How to Remove Notation Variations and Noise?

Control character and whitespace variation, as well as inconsistent spellings of the same word, disperse embeddings for separate reasons. Removal rules and dictionary-based unification must be applied in a defined order and priority, unifying the notation into one form while preserving meaning.

Defining Rules for Removing Whitespace, Line Breaks, and Control Characters

Since half-width spaces, full-width spaces, line breaks, tabs, and control characters each serve different purposes, uniformly deleting them would break the document structure. Based on Unicode's White_Space property, you need to decide, for each document type, which whitespace to preserve and which to collapse.

Line breaks at the end of a line indicate paragraph or item boundaries, so they should be preserved in principle. On the other hand, consecutive half-width spaces within text are often remnants of copy processing or layout adjustments, and it is fine to collapse them into a single character. Control characters (U+0000–U+001F, U+007F–U+009F) carry no meaning in the body text and should, in principle, be removed. In documents originating from OCR, invisible control characters resulting from recognition errors tend to be mixed in, and if left unaddressed, they cause the sequence of code points for the same word to differ, leading embedding search to treat it as a different word.

python
1import re 2 3text = re.sub(r"[\x00-\x1f\x7f-\x9f]", "", text) 4text = re.sub(r" +", " ", text) # Full-width spaces (\u3000) are excluded from collapsing and judged separately 5text = re.sub(r"\n{3,}", "\n\n", text)

Full-width spaces used as bullet-point markers or table cell separators can lose the document's structural information if collapsed in the same way as consecutive whitespace. Such layout-dependent whitespace should be extracted separately for each document type before applying collapsing rules, and you should decide whether to treat it differently.

Once removal rules are decided, it is important to apply them consistently across the entire document set without changing the criteria midway. If a rule is changed, all already-processed files should be reprocessed so that a mixture of old and new rules does not remain.

Unifying Variant Spellings of the Same Word Using a Dictionary

Build the normalization dictionary with a simple structure that maps multiple variant spellings to a single canonical form. Start with vocabulary prone to variation identified for each document type—place names, personal names, technical terms, loanwords, and so on—using any published vocabulary lists as a foundation, and add variant spellings extracted from your own corpus.

python
1normalization_dict = { 2"VariantA": "CanonicalForm", 3"VariantB": "CanonicalForm", 4} 5for variant, canonical_form in normalization_dict.items(): 6text = text.replace(variant, canonical_form)

The order of application also requires attention. This process is performed after character encoding unification and Unicode normalization, and before word segmentation. Therefore, as shown above, variant unification is done via substring matching against the entire string, which has not yet been split into word units. If the dictionary is applied to a sequence of code points that has not been normalized, variants that look the same but have different code points will not be caught, leaving unification gaps.

Ambiguity in the dictionary is resolved through priority rules. When a single variant has multiple candidate canonical forms, determine the priority based on the document's origin and context, and keep a record of which conversion was applied. Without such a record, it becomes impossible to later verify whether dialectal differences were mistakenly unified.

The dictionary's coverage cannot be completed in a single pass. Any unregistered variants found during visual inspection should be added to the dictionary each time, and if there are previously processed files, they should be reprocessed with the updated dictionary. Setting up an operation where, each time an entry is added, you review whether it introduces conflicting conversions with existing entries will help maintain consistency in conversion results even as the dictionary grows. Since loanwords tend to have multiple coexisting spelling variants, you should design the operation on the premise that the dictionary will be reviewed periodically whenever new sets of documents are incorporated.

Step 3: How to Handle Word Segmentation?

Step 3: How to Handle Word Segmentation?

In Lao, where spaces between words do not align with word boundaries, simple whitespace splitting or subword tokenizers alone cannot restore word-level segmentation. It is essential to select a language-specific segmenter that uses a dictionary-based or trained model, and to verify the segmentation accuracy numerically.

Selecting and Configuring a Lao Tokenizer

LaoNLP is a candidate that comes up as a language-specific segmenter. It is an open-source toolkit with dictionary-based segmentation capability, able to restore Lao word boundaries more precisely than simple whitespace splitting.

python
1from laonlp.tokenize import word_tokenize 2 3tokens = word_tokenize(text)

A general-purpose subword tokenizer such as SentencePiece is also an option, but since it splits within words based on frequency, it is not suited for the purpose of restoring word boundaries themselves. Even when subwording is needed just before embedding generation, splitting into word units first with a language-specific segmenter such as LaoNLP, then passing the result to the subword tokenizer, avoids carrying over any breakdown of word boundaries.

As a criterion for selection, it is worth checking in advance how well a dictionary-based segmenter can handle unknown words (new loanwords or technical terms). Whether terminology unified in a normalization dictionary can be registered into the segmenter's own dictionary depends on the tool's mechanism, and if it cannot be registered, the design should assume an operation where segmentation results are corrected by downstream rules.

When the document volume is large and errors from dictionary-based segmentation stand out, introducing a language-specific segmentation model fine-tuned with CRF or a transformer from research implementations is also worth considering. However, since this increases operational load, a more realistic approach is to first check LaoNLP's segmentation results on a small sample, determine whether practical accuracy is achieved for the document types used in actual operations, and then expand the scope of adoption.

How to Verify Word Segmentation Results

Whether the segmentation results reach a practical level of accuracy cannot be judged by visual inspection alone. Confirmation is done by creating gold-standard annotated data for boundary-level evaluation and measuring Precision, Recall, and F1 by comparing it against the segmenter's output.

Gold-standard annotated data is created by extracting samples for each document type and manually re-marking the correct word boundaries. The number of samples can be small, and it varies depending on project scale and document diversity, but as a validation set usable for comparing segmenters, it is desirable to secure around 1,000 samples as a minimum guideline. When document types are mixed, such as contracts, meeting minutes, and customer inquiries, create annotated data separately for each type, and check individually whether segmentation accuracy drops depending on the type.

The validation process proceeds as follows.

  1. Extract samples for each document type and create gold-standard annotated data with correct word boundaries marked manually
  2. Run the samples through a segmenter such as LaoNLP and compare the output against the gold-standard annotations at the boundary level
  3. Calculate Precision, Recall, and F1 for each document type
  4. Identify document types with low accuracy and investigate causes such as a high number of unknown words or dialectal differences
  5. Address the issues through vocabulary additions to the dictionary or downstream correction rules, then re-evaluate

If a type with low accuracy is found, check whether terminology unified in the normalization dictionary can be additionally registered into the segmenter's dictionary; for tools with a mechanism that does not allow this, switch to an operation where downstream correction rules absorb the mis-segmentation.

Step 4: How to Validate the Cleaned Data?

Step 4: How to Validate the Cleaned Data?

Even if segmentation accuracy can be confirmed numerically, this alone does not reveal whether appropriate documents can be retrieved during search. After visually confirming that no noise remains, a step remains in which search performance is measured under the actual retriever settings, numerically linking the quality of processing to the search results.

Conducting Visual Checks Through Sample Extraction

Mixing random samples with edge cases in the extraction determines the accuracy of the visual check. Random samples alone, drawn at equal intervals or by random number from the full set, will miss low-frequency defects. Deliberately add edge cases where errors tend to concentrate—such as documents with low OCR image quality, non-standard loanword notations, and customer inquiries with strong dialectal differences—when building the extraction set.

Items to check are divided into three types.

Check itemTypical defectWhere to address
Remaining mojibakeFragmentary symbols, unnatural sequences of code pointsRe-apply encoding detection / ftfy
Normalization errorsSame word remaining as different code point sequencesCheck for missed NFC application
Mis-tokenizationWord boundaries cut across clause or phrase boundariesReview the segmenter's dictionary / correction rules

When a defect is found manually, don't just fix it on the spot—reflect the cause in the dictionary or rules. If an alternative spelling is added to the normalization dictionary, also check that it doesn't conflict with existing entries; if the issue is a segmenter error, reflect it in the correction rules. Keeping checking and dictionary updates in a back-and-forth cycle makes it possible to catch similar defects early in the next batch of documents arriving from the same source. Decide the extraction count for each document type, check the defect detection rate from your own extraction results, and consider adjusting by increasing the sample size and tightening the standard for types with a high detection rate.

Verifying Impact on RAG Search Accuracy with Small-Scale Tests

By preparing pairs of gold questions and their correct documents, then running them through the retriever on both the pre-cleaning and post-cleaning data and comparing recall@10, MRR, and nDCG, you can confirm numerically how the preprocessing changes affected search results. Use questions that closely resemble what customers or employees actually ask, and identify the correct documents or chunks by hand in advance.

For comparison, measure by changing settings one at a time—chunk size, embedding model, whether normalization is applied, and so on. Start with a chunk size of around 256–512 tokens, and check within the search results whether context before and after splitting has been lost.

Comparison ConditionMetric to CheckWhat It Reveals
Before/after applying normalization dictionaryrecall@10Whether unifying variant spellings reduced missed retrievals
Before/after applying NFCMRRWhether code point variation affected ranking
Difference in tokenizer/segmenternDCGWhether word boundary accuracy was reflected in top results

Setting changes that don't improve the numbers should be treated as a reason to reexamine the quality of the validation set itself. By advancing to the stage of confirming whether fixes found through visual inspection actually show up as improvements in search metrics, it becomes easier to apply the same preprocessing priorities to the next set of documents.

What Are Common Pitfalls in Data Cleaning?

What Are Common Pitfalls in Data Cleaning?

Normalization dictionaries and noise-removal rules can harm search results whether they are applied too aggressively or not aggressively enough. Neither type of failure is necessarily noticed right after the work is done—often it only becomes apparent during search testing—so the boundary line needs to be made clear at the rule-design stage.

Cases Where Excessive Noise Removal Damages Meaning

If rules for removing control characters and whitespace are extended to cover diacritical marks or combining characters as well, the distinction in meaning or pronunciation of words themselves is lost. Lao script has combining characters that attach tone or vowel information to a base character, and if these are mistakenly treated as visual noise and included in the removal target, the resulting sequence of code points gets converted into a different word.

Do not extend the scope of removal targets—defined at the stage of setting rules for removing whitespace, line breaks, and control characters—to include diacritical marks or combining characters. Checking the list of code points before application is an essential step.

Zero-width characters require similar caution. Invisible characters such as zero-width joiners may have been intentionally inserted when the document was created, and can be involved in how a given word is displayed or segmented. If these are uniformly included in the removal target, a word whose representation was supposed to be unified through Unicode normalization ends up remaining as a different code point sequence.

As a decision criterion, add symbols to the removal target one at a time, after identifying the scope of impact for each.

  • Compare words that include a combining character against those that don't
  • Check whether the word could be read as a different word after removal
  • Perform this check separately for each type of symbol

If multiple symbols are added to the removal rule all at once, it becomes impossible to later identify which symbol caused the loss of meaning.

Decisions to extend removal rules should be made only after confirming the impact through visual sampling, symbol by symbol. Since it is difficult to restore a file after removal has already been applied, insert a step—before applying the rule to the entire production dataset—in which strings before and after removal are compared side by side on a small sample to check the differences. Inserting this step allows you to catch meaning-altering changes before they are applied to production.

Cases Where Dialects or Colloquial Expressions Are Mistakenly Unified

If spelling differences due to dialect or generational variation are treated as mere notational inconsistency and converted en masse to a canonical form, the information about word meaning and regionality that the document carries is lost. If region-specific phrasing found in customer inquiries is forced to conform to the standard spelling used in internal documents, the original question text—which includes dialect—and the normalized document wording will no longer match at search time, which can actually cause missed retrievals. The same applies when multiple spelling variants of a loanword coexist: if, without establishing a priority rule, everything is mechanically unified to whichever spelling is found first, a spelling that was correct in a different context may end up being wrongly rewritten.

Decisions about dictionary unification should treat variant spellings that don't change a word's meaning (simple spelling variation) separately from alternate expressions that reflect dialect or generational differences. As decision criteria, individually check, for each document type, the document's origin (which region or customer segment it was collected from) and whether the meaning and contextual naturalness are preserved after conversion. If you keep a record of conversion history and the rationale for priority in the normalization dictionary, you can later identify, document type by document type, whether a particular conversion mistakenly unified a dialectal difference.

If a questionable conversion is found, a practical response is not to delete the dictionary entry on the spot, but to review its priority and narrow the scope of application by document type. Preparing separate dictionary sets for customer-facing documents and internal documents, and dividing the scope of unification between them, is also an option.

Frequently Asked Questions

Frequently Asked Questions

This summarizes the points that implementation personnel should check regarding tool selection, the scope of handling based on data volume, and the decision between handling things in-house versus outsourcing.

Q1: Are There Automated Cleaning Tools for Lao?

Individual tools exist for each step of the process, but there is no single tool that handles everything in one pass. For encoding correction, chardet and ftfy cover the task; for code point unification, unicodedata and ICU's Normalizer2 are available; for word segmentation, LaoNLP covers the relevant scope. At the chunking stage, text splitters from Hugging Face or LangChain can be combined for use. In all cases, the underlying assumption is an operational approach that does not simply adopt the output of automated processing as-is, but instead combines visual sample checks with ongoing updates to the dictionary.

Q2: Is Cleaning Necessary Even for Small Amounts of Data?

It is necessary. Even with a small number of documents, variations in character encoding and inconsistent notation get embedded directly into the embeddings, becoming a cause of missed search results. The approach of first building the skeleton of a normalization dictionary and rule set on a small-scale dataset, then applying that same dictionary and ruleset as documents increase, avoids the trouble of having to review the entire document set later on.

Q3: Should Cleaning Be Done In-House or Outsourced?

The decision depends on the volume of data and the scale of risk involved.

Cases where in-house handling is sufficient: This applies when target documents are centered on a single format, low-quality OCR-derived documents are few, and document types are limited to a few categories such as contracts and meeting minutes. If a script combining chardet, ftfy, and LaoNLP, along with visual checks via sample extraction, can be managed by one or two staff members, then an approach of growing the dictionary and rules in-house is suitable.

Cases where collaboration with external partners should be considered: This applies when multiple document formats are mixed together, the volume of OCR conversion is large, and variation in dialects or loanwords differs significantly across document types. Partnering with an external organization or research institution experienced in building language resources allows for creating gold-standard annotated data and evaluating segmentation models through a combination of semi-automated processing and manual verification. If the scale makes the decision unclear, it is reasonable to first try in-house processing on a small-scale dataset, and once document types are identified where Precision, Recall, and F1 for the word segmentation results are not stable, consider the scope of external collaboration at that point.

Conclusion

Conclusion

Processing Lao-language documents before feeding them into RAG involves less rework if approached as one continuous flow, from encoding detection through to verification. After checking character encoding, input methods, and notational variation in dialects and loanwords, align the character encoding into a consistent, variation-free state by following this order: encoding detection and decoding in line with common chardet usage, mojibake correction with ftfy, and NFC normalization (one of the Unicode normalization forms). Next, define rules for removing control characters and whitespace, unify inconsistent notation using a normalization dictionary, and then perform word segmentation using a language-specific segmenter such as LaoNLP, measuring Precision, Recall, and F1 against gold-standard annotated data to verify segmentation accuracy. Finally, combine visual sample checks with small-scale search tests using recall@10, MRR, and nDCG to numerically confirm how changes in processing are reflected in search results. Proceed with caution against going too far in ways that undermine meaning, such as removing diacritical marks or uniformly flattening dialectal distinctions. Start by taking a small number of documents at hand, trying out the flow from encoding detection through verification on a small scale once, and confirming whether the same procedure can be run on your own organization's data.

References

  1. docs.databricks.com/aws/en/ai-search/retrieval-qualityAccessed November 20, 2026
  2. ebooklao.gov.la/wp-content/uploads/2024/03/ລາວສຶກສາ.pdfAccessed November 20, 2026
  3. learn.microsoft.com/azure/ai-studio/concepts/retrieval-augmented-generationAccessed November 20, 2026
  4. trec.nist.gov/pubs/trec25/papers/DUTH-CX.pdfAccessed November 20, 2026

Author & Supervisor

Chi

Chi

Majored in Information Science at the National University of Laos, where he contributed to the development of statistical software, building a practical foundation in data analysis and programming. He began his career in web and application development in 2021, and from 2023 onward gained extensive hands-on experience across both frontend and backend domains. At our company, he is responsible for the design and development of AI-powered web services, and is involved in projects that integrate natural language processing (NLP), machine learning, and generative AI and large language models (LLMs) into business systems. He has a voracious appetite for keeping up with the latest technologies and places great value on moving swiftly from technical validation to production implementation.