Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.tensorlake.ai/llms.txt

Use this file to discover all available pages before exploring further.

View Source Code

Check out the full source code for this example on GitHub.
This tutorial demonstrates how to build a production-grade Web Scraper that crawls websites, processes content into clean Markdown, generates embeddings using Voyage AI, and stores them in MongoDB Atlas Vector Search.

​
Overview

This application showcases the power of Tensorlake’s parallel processing capabilities:
  1. Parallel Crawling: Uses Breadth-First Search (BFS) with Tensorlake’s .map() to fetch multiple pages concurrently at each depth level.
  2. Headless Browsing: Utilizes PyDoll (based on Chromium) to render JavaScript-heavy websites.
  3. Content Cleaning: Converts HTML and PDFs to clean Markdown, automatically removing boilerplate like headers, footers, and ads.
  4. Vector Embeddings: Generates high-quality embeddings for document chunks using Voyage AI.
  5. Vector Search: Stores the processed chunks and embeddings directly into MongoDB Atlas for RAG applications.

​
Prerequisites

  • Python 3.11+
  • Tensorlake Account and CLI installed.
  • MongoDB Atlas cluster URI.
  • Voyage AI API Key.

​
Implementation

The application is defined in a single file, scraper_to_atlas.py. It defines two custom runtime images: one for scraping (with Chromium) and one for embedding (lightweight).

​
1. Define Dependencies and Images

from tensorlake.applications import Image

# Image with Chromium, pydoll, and dependencies for web scraping
scraper_image = (
    Image(name="scraper-to-atlas-image", base_image="python:3.11.0")
    .env("DEBIAN_FRONTEND", "noninteractive")
    .run("apt-get update && apt-get install -y chromium ...") # System deps
    .run("pip install pydoll-python tensorlake beautifulsoup4 markdownify pymupdf4llm")
)

# Image for embedding and MongoDB operations
embedding_image = (
    Image(name="embedding-image", base_image="python:3.11.0")
    .run("pip install tensorlake voyageai pymongo")
)

​
2. Main Scraping Logic (scraper_to_atlas.py)

The @application entry point orchestrates the crawling process. It manages the BFS queue and dispatches parallel tasks using fetch_and_convert.map(). 
@application()
@function(secrets=["VOYAGE_API_KEY", "MONGO_URI"])
def scrape_and_embed(input: ScrapeAndEmbedInput) -> dict:
    # ... setup BFS ...

    # Phase 1: Parallel BFS
    for depth in range(max_depth + 1):
        # ... deduce URLs to fetch ...
        
        # Parallel fetch all URLs at this depth level using map()
        results = fetch_and_convert.map(urls_to_fetch)
        
        # ... process results and collect new links ...

    # Phase 2: Process PDFs in parallel
    if pdf_urls:
        pdf_results = fetch_and_convert_pdf.map(list(pdf_urls))

    # Phase 3: Generate embeddings and store
    embed_and_store(all_documents, ...)

​
3. Page Fetching and Conversion

The fetch_and_convert function runs in the scraper_image and uses PyDoll to render pages. 
@function(image=scraper_image, timeout=120, memory=4)
def fetch_and_convert(url: str) -> dict:
    return asyncio.run(_fetch_and_convert_async(url))

async def _fetch_and_convert_async(url: str) -> dict:
    async with Chrome() as browser:
        page = await browser.new_page()
        await page.goto(url)
        html = await page.content()
        # ... extract title and links ...
    
    # Convert to clean markdown
    markdown = _html_to_markdown(html)
    chunks = _chunk_text(markdown)
    return {"url": url, "chunks": chunks, ...}

​
4. Embedding and Storage

The embed_and_store function runs in the embedding_image and handles interaction with Voyage AI and MongoDB. 
@function(image=embedding_image, secrets=["VOYAGE_API_KEY", "MONGO_URI"])
def embed_and_store(documents, mongo_uri, voyage_api_key, ...):
    # Initialize Voyage AI
    vo = voyageai.Client(api_key=voyage_api_key)
    
    # Generate embeddings
    embeddings = vo.embed(texts=[d["text"] for d in documents], model="voyage-4-large")
    
    # Store in MongoDB
    client = pymongo.MongoClient(mongo_uri)
    collection = client[db_name][col_name]
    collection.insert_many([{...} for ...])

​
Running Locally

  1. Set your environment variables: 
    export MONGO_URI="mongodb+srv://..."
export VOYAGE_API_KEY="voyage-..."
  2. Run the application: 
    python scraper_to_atlas.py

​
Deploying to Tensorlake

Deploy your scalable scraper to the cloud. 
tl secrets set MONGO_URI="mongodb+srv://..."
tl secrets set VOYAGE_API_KEY="voyage-..."
tl deploy scraper_to_atlas.py
Your scraper will now run in the cloud, automatically scaling to handle hundreds of pages in parallel!