CharacterTextSplitter/RecursiveCharacterTextSplitter corrupt chunk content when merging with a non-lookaround regex separator and keep_separator=False

[Soban-2004]

Submission checklist

• This is a bug, not a usage question.
• I added a clear and descriptive title that summarizes this issue.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
• This is not related to the langchain-community package.
• I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-openrouter
• langchain-perplexity
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Related Issues / PRs

This is a long-standing, recurring problem with a partial fix already merged, and I want to be
explicit about that history:

• text-splitters: Fix regex separator merge bug in CharacterTextSplitter #31137 (merged) added detection for zero-width "lookaround" regex separators
  ((?=, (?<=, (?!, (?<!) in CharacterTextSplitter.split_text, forcing merge_sep="" for those
  specifically -- but its own comment ("Detect zero-width lookaround so we never re-insert it")
  scopes the fix explicitly to lookaround patterns. Any ordinary consuming regex separator
  (e.g. \s+, [,;]+, \n{2,}) still falls through to using the raw pattern string as the literal
  merge separator -- the exact bug text-splitters: Fix regex separator merge bug in CharacterTextSplitter #31137 fixed, just for a narrower trigger.
• RecursiveCharacterTextSplitter never received even that narrow fix -- its merge path
  (_split_text in character.py) has no lookaround detection at all.

Prior reports of this same underlying issue: #23394 (closed "not planned" by a stale-issue bot in
Oct 2024, not fixed), #10840, #9843, and closed/unmerged fix attempts #37664 and #23397. None of
these are currently open. I'm filing fresh with a reproduction against current master and a precise
trace of exactly where the (partial) existing fix stops covering the problem.

Reproduction Steps / Example Code (Python)

from langchain_text_splitters import CharacterTextSplitter, RecursiveCharacterTextSplitter

text = "AAA   BBB\tCCC\n\nDDD    EEE"
splitter = CharacterTextSplitter(
    chunk_size=15, chunk_overlap=0, add_start_index=True,
    separator=r"\s+", is_separator_regex=True, keep_separator=False,  # keep_separator=False is the class default
)
for doc in splitter.create_documents([text]):
    print(repr(doc.page_content), doc.metadata)

# 'AAA\\s+BBB\\s+CCC' {'start_index': -1}   <- corrupted content: literal "\s+" embedded as text
# 'DDD\\s+EEE' {'start_index': -1}          <- same corruption; start_index is -1 because this
#                                              string never actually appears in the source at all

text2 = "one   two\tthree\n\nfour     five"
splitter2 = RecursiveCharacterTextSplitter(
    chunk_size=15, chunk_overlap=0, separators=[r"\s+"], is_separator_regex=True, keep_separator=False,
)
print(splitter2.split_text(text2))
# ['one\\s+two', 'three\\s+four', 'five']   <- same corruption, no lookaround-only carve-out exists here

Error Message and Stack Trace (if applicable)

None raised -- silent content corruption. The returned page_content is not a substring of the
input text at all (contains the literal regex pattern text instead of the matched whitespace/
separator), and add_start_index correctly reports -1 since the corrupted string genuinely can't be located in the source, but nothing surfaces this as an error.

Description

When a regex separator is used (is_separator_regex=True) with keep_separator=False
(CharacterTextSplitter's default) and multiple split pieces need to be re-joined into one chunk,
both splitters use the raw separator string as the literal text to join pieces with:

• CharacterTextSplitter.split_text: merge_sep = self._separator (only suppressed to "" when the
  pattern is a detected zero-width lookaround, per text-splitters: Fix regex separator merge bug in CharacterTextSplitter #31137)
• RecursiveCharacterTextSplitter._split_text: separator_ = "" if self._keep_separator else separator
  (no lookaround detection at all)

For any regex that actually consumes variable text -- which is most real-world regex separators,
e.g. \s+ matching a differing amount of whitespace at each occurrence -- the pattern string itself
is not equivalent to what it matched, so re-inserting it produces output that (a) contains literal
regex syntax as visible garbage text, and (b) is not a substring of the source document at all.

This is a real correctness/data-integrity bug, not a cosmetic one: any RAG or embedding pipeline
using a regex separator with the class default (keep_separator=False) silently gets corrupted
chunk text fed into it, with no error or warning anywhere in the pipeline.

I see two possible directions for a fix, and I'd like a maintainer's input on which is preferred
before I put together a PR, given multiple past attempts at this haven't landed:

1. Minimal/safe: raise a clear error when is_separator_regex=True and keep_separator=False and
   the separator isn't a detected zero-width lookaround -- i.e. extend text-splitters: Fix regex separator merge bug in CharacterTextSplitter #31137's existing lookaround
   detection so the fallback for the un-handled case is a loud ValueError pointing the user at
   keep_separator=True, instead of silent corruption. Small diff, doesn't attempt to reconstruct
   correct output, but guarantees nothing silently corrupts.
2. Complete: capture the actual matched separator text during the initial split (similar to how
   _split_text_with_regex already does for keep_separator=True, via a capturing group), and use
   the real per-occurrence matched text when re-joining pieces in _merge_splits, instead of a single
   uniform merge string. More correct, more invasive, and is presumably why prior attempts (fix(text-splitters): regex separator pattern used as literal text when merging chunks #37664,
   text-splitters: fix text_splitter keep_seprator git bug #23397) struggled to land.

Happy to open a PR for whichever direction is preferred, plus tests covering both CharacterTextSplitter
and RecursiveCharacterTextSplitter, if a maintainer can weigh in and assign me to this issue.

System Info

System Information

OS: Windows
OS Version: 10.0.26200
Python Version: 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]

Package Information

langchain_core: 1.4.9
langchain: 1.3.13
langchain_community: 0.4.2
langsmith: 0.10.2
langchain_classic: 1.0.8
langchain_groq: 1.1.3
langchain_huggingface: 1.2.2
langchain_protocol: 0.0.18
langchain_text_splitters: 1.1.2
langgraph_sdk: 0.4.2

Optional packages not installed

deepagents
deepagents-cli

Other Dependencies

aiohttp: 3.13.2
anyio: 4.9.0
async-timeout: 4.0.3
cachetools: 4.2.2
distro: 1.9.0
groq: 0.37.1
httpx: 0.28.1
httpx-sse: 0.4.3
huggingface-hub: 1.23.0
jsonpatch: 1.33
langgraph: 1.2.9
numpy: 1.26.4
orjson: 3.11.9
packaging: 24.2
pydantic: 2.13.4
pydantic-settings: 2.13.1
pyyaml: 6.0.2
requests: 2.32.5
requests-toolbelt: 1.0.0
rich: 14.3.3
sentence-transformers: 5.6.0
sniffio: 1.3.1
sqlalchemy: 2.0.51
tenacity: 9.1.2
tokenizers: 0.22.2
transformers: 5.13.1
typing-extensions: 4.14.1
uuid-utils: 0.17.0
websockets: 15.0.1
wrapt: 1.17.2
xxhash: 3.8.1
zstandard: 0.24.0

[chelsealong]

I'd like to take this one if it's still open — happy to put up a PR.

[SnowingFox]

Opened a PR to address this: #39596

[SnowingFox]

I have a ready implementation and would like to submit it for review (PR #39596, currently auto-closed pending assignment). The regression test reproduces the corruption on master and passes with the change.

Root cause: when keep_separator=False and is_separator_regex=True with a consuming separator (e.g. \s+, [,;]+, \n{2,}), both CharacterTextSplitter and RecursiveCharacterTextSplitter re-join merged pieces using the raw regex pattern string as a literal separator. The pattern is not the text the regex matched, so merged chunks contain literal regex syntax (e.g. "AAA\s+BBB"), which is not a substring of the source — add_start_index then reports -1.

#31137 only special-cased zero-width lookarounds; this change extends the same protection to every other consuming pattern.

Approach: add a _regex_is_literal() helper in character.py; when the separator is a consuming, non-literal regex and keep_separator=False, merge with "" (the documented meaning of keep_separator=False), leaving literal-regex and lookaround behavior unchanged. Happy to adjust the merge strategy if you prefer a different design.