AIbase
Product LibraryTool Navigation

Scrape-Tokenize-whole-website-for-LLMs

Public

Extract the urls from a website's sitemap, scrape the text from each URL, clean the text, and prepare it for use in a Large Language Model (LLM) by tokenizing the text.

Creat2024-09-15T01:32:52
Update2024-09-15T01:37:36
0
Stars
0
Stars Increase

Related projects