{"id":57503,"date":"2026-06-21T11:07:12","date_gmt":"2026-06-21T15:07:12","guid":{"rendered":"https:\/\/overcentral.com\/en\/?p=57503"},"modified":"2026-06-21T11:07:12","modified_gmt":"2026-06-21T15:07:12","slug":"crawlee-python-rag-chunk-export-pipeline","status":"publish","type":"post","link":"https:\/\/overcentral.com\/en\/crawlee-python-rag-chunk-export-pipeline\/","title":{"rendered":"Crawlee for Python Builds Web Crawling Pipeline with RAG Chunk Export"},"content":{"rendered":"

Crawlee for Python<\/a>, the web scraping and automation framework, now enables developers to build end-to-end crawling pipelines that export structured data directly into Retrieval-Augmented Generation (RAG) ready chunk formats. The latest demonstration pipeline showcases how a single coordinated crawl can extract content using three distinct parsing strategies \u2014 BeautifulSoup, Parsel, and Playwright \u2014 normalize the results into a product catalog, build a site link graph, and output JSONL files containing semantically segmented text chunks suitable for feeding into vector databases and LLM<\/a>a-powered retrieval systems.<\/p>\n

Why a Multi-Parser Pipeline Matters for RAG Data Preparation<\/h2>\n

Modern web crawling rarely demands a single parsing approach. Static HTML pages yield quickly to lightweight parsers like BeautifulSoup or Parsel, while JavaScript-heavy single-page applications require a full browser engine such as Playwright. By combining all three in a single pipeline, developers can cover the spectrum of site architectures without maintaining separate crawlers. The RAG export step then transforms whatever raw text the parsers collect into clean, fixed-size chunks that preserve source metadata \u2014 a critical requirement for grounding LLM responses in verifiable, attributable content.<\/p>\n

The Three Crawling Strategies in the Pipeline<\/h2>\n

The pipeline runs three crawlers sequentially against a target base URL, each producing a list of row dictionaries containing fields such as source<\/strong>, page_type<\/strong>, title<\/strong>, url<\/strong>, text_preview<\/strong>, rendered_text<\/strong>, and description<\/strong>.<\/p>\n

BeautifulSoup Crawl: Fast Static Extraction<\/h3>\n

The BeautifulSoup pass handles traditional server-rendered pages. It parses the raw HTML, extracts links and structured data from meta tags, heading elements, and common content containers, and returns rows with minimal latency. This pass is ideal for sites that do not rely on client-side JavaScript to render their primary content.<\/p>\n

Parsel Crawl: Precision Selectors for Listings<\/h3>\n

The Parsel pass employs XPath and CSS selectors tuned for product listings and catalog pages. It targets specific structural patterns \u2014 price spans, stock availability badges, SKU attributes, and rating widgets \u2014 to extract normalized product data with higher field-level accuracy than a general-purpose BeautifulSoup parse typically achieves.<\/p>\n

Playwright Crawl: Dynamic Page Rendering<\/h3>\n

The Playwright pass launches a headless Chromium browser to capture pages that require JavaScript execution. It waits for network idle, takes a full-page screenshot for visual verification, and extracts the rendered DOM after all client-side frameworks have finished loading. This ensures that content injected by React, Vue, or Angular applications is not missed.<\/p>\n

How the Pipeline Converts Crawled Data into RAG Chunks<\/h2>\n

The function make_rag_chunks<\/strong> is the core mechanism that transforms raw extraction rows into retrievable text segments. It iterates over every row collected by the three parsers, selects the best available text field (preferring text_preview<\/strong> over rendered_text<\/strong> over description<\/strong>), normalizes whitespace and encoding, and then splits the text at sentence boundaries using a regular expression pattern that respects sentence-ending punctuation.<\/p>\n

Each chunk respects a configurable max_chars<\/strong> threshold \u2014 defaulting to 700 characters \u2014 while keeping complete sentences intact. The result is a list of dictionaries, each containing a chunk_id<\/strong> generated from a SHA-1 hash of the URL combined with the chunk text (truncated to 12 characters for readability), the original url<\/strong>, source<\/strong>, page_type<\/strong>, title<\/strong>, and the chunk text<\/strong> itself.<\/p>\n

This design addresses a common failure point in RAG pipelines: chunks that split mid-sentence break semantic coherence and confuse retrieval scoring. By segmenting at sentence boundaries and carrying full metadata, the output is immediately compatible with embedding pipelines and vector stores such as Chroma, Pinecone, or Weaviate.<\/p>\n

Analysis, Visualization, and Export Features<\/h2>\n

After crawling completes, the analyze_outputs<\/strong> function produces a comprehensive summary of the run. It aggregates all rows from the three parsers, flattens product data into a structured DataFrame, and computes numeric fields such as price, stock, and rating. It also calculates an inventory_value<\/strong> column \u2014 the product of price and stock \u2014 which provides a quick signal for catalog valuation.<\/p>\n

The function builds a directed link graph using NetworkX, writing the result to a GraphML file for later visualization in tools such as Gephi or yEd. It then generates:<\/p>\n