Find answers from the community

Updated 3 months ago

Are there any tools in Llama_Index or

Are there any tools in Llama_Index or Langchain that can be used for cleaning up nodes before embedding? I'm looking for something to help remove white space, special characters, etc.

12 comments

LLogan M

Isn't that as simple as just applying a regex to your text? 👀

LLogan M

text = re.sub(r'[^0-9A-Za-z ]", "", text)

LLogan M

something like that

PPocketColin

yeah I should have been more clear about what I'm looking for haha. How do I do that with SimpleNodeParser or do I have to use a different node parser that gives me access to each node as it parses?

PPocketColin

Or should I literally just iterate over every single node and update them after SimpleNodeParser returns the nodes?

LLogan M

I would just iterate after you get the parsed nodes yea 🙂

PPocketColin

laaaaaame

PPocketColin

haha ok thanks!

PPocketColin

maybe I'll make a PR to allow a custom parser to be passed to SimpleNodeParser

LLogan M

mmm maybe wait a day or two, something new coming out soon that will make this easier 🙂

PPocketColin

ooooo can't wait!

LLogan M

I'll even add this as an example! 💪

Add a reply