Tensorlake Documentation

Qdrant is a high-performance vector database built for AI applications. When combined with Tensorlake’s document parsing, you get RAG systems with complete document understanding—including table summaries, figure descriptions, and metadata filtering that traditional parsing misses. This integration is essential for academic research, financial reports, legal documents, and technical documentation where visual content matters.

Run this end-to-end in Colab:

Why Use Tensorlake + Qdrant?

The Problem:

Traditional parsing loses tables, figures, and reading order
Text-only embeddings miss critical visual content
Generic chunking breaks document structure and context
No way to filter by document metadata before searching

The Solution: Tensorlake preserves tables, figures, and structure during parsing. Qdrant stores embeddings with rich metadata for filtering. Together, they create RAG systems that understand complete documents. Key Benefits:

Richer embeddings - Include table summaries and figure descriptions, not just text
Advanced filtering - Search within specific authors, dates, or document types
Better chunking - Semantic sections (abstract, methods, results) instead of arbitrary splits
Accurate results - Complete context leads to more relevant retrieval

Installation

pip install tensorlake qdrant-client sentence-transformers

Quick Start

Step 1: Parse Documents with Tensorlake

Configure Tensorlake to extract structured data, tables, and figures in one API call:

from tensorlake.documentai import (
    DocumentAI,
    ParsingOptions,
    StructuredExtractionOptions,
    EnrichmentOptions,
    ChunkingStrategy,
    TableParsingFormat,
    TableOutputMode
)

# Initialize client
doc_ai = DocumentAI()

# Define schema for structured extraction
json_schema = {
    "title": "ResearchPaper",
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "authors": {"type": "array", "items": {"type": "string"}},
        "abstract": {"type": "string"},
        "conference_name": {"type": "string"},
        "publication_year": {"type": "integer"}
    }
}

# Configure parsing
parsing_options = ParsingOptions(
    chunking_strategy=ChunkingStrategy.SECTION,  # Chunk by semantic sections
    table_parsing_strategy=TableParsingFormat.TSR,
    table_output_mode=TableOutputMode.MARKDOWN,
)

structured_extraction_options = [StructuredExtractionOptions(
    schema_name="ResearchPaper",
    json_schema=json_schema,
)]

enrichment_options = EnrichmentOptions(
    figure_summarization=True,
    figure_summarization_prompt="Summarize this figure in the context of the research paper.",
    table_summarization=True,
    table_summarization_prompt="Summarize this table's data and significance to the research.",
)

# Parse document
file_url = "https://example.com/research_paper.pdf"
parse_id = doc_ai.parse(
    file_url,
    parsing_options,
    structured_extraction_options,
    enrichment_options
)

result = doc_ai.wait_for_completion(parse_id)

What You Get:

Markdown chunks preserving reading order and structure
Structured metadata (title, authors, conference, year)
Table summaries that capture data meaning
Figure descriptions explaining visual content

Step 2: Create Embeddings and Store in Qdrant

Transform parsed content into embeddings and store with metadata for filtering:

from qdrant_client import QdrantClient, models
from sentence_transformers import SentenceTransformer
from uuid import uuid4

# Initialize Qdrant and embedding model
qdrant_client = QdrantClient(":memory:")  # Use QdrantClient(url="...") for production
model = SentenceTransformer("all-MiniLM-L6-v2")

collection_name = "research_papers"

# Create collection
qdrant_client.create_collection(
    collection_name=collection_name,
    vectors_config=models.VectorParams(
        size=model.get_sentence_embedding_dimension(),
        distance=models.Distance.COSINE
    )
)

# Extract structured metadata
structured_metadata = result.structured_data[0].data if result.structured_data else {}

# Create embeddings for text chunks
points = []

for chunk in result.chunks:
    embedding = model.encode(chunk.content).tolist()
    payload = {
        **structured_metadata,
        'content': chunk.content,
        'type': 'text'
    }
    points.append(models.PointStruct(
        id=str(uuid4()),
        vector=embedding,
        payload=payload
    ))

# Create embeddings for table summaries
for page in result.pages:
    for fragment in page.page_fragments:
        if fragment.fragment_type == "table" and fragment.content.summary:
            embedding = model.encode(fragment.content.summary).tolist()
            payload = {
                **structured_metadata,
                'content': fragment.content.summary,
                'type': 'table',
                'page': page.page_number
            }
            points.append(models.PointStruct(
                id=str(uuid4()),
                vector=embedding,
                payload=payload
            ))

# Create embeddings for figure summaries
for page in result.pages:
    for fragment in page.page_fragments:
        if fragment.fragment_type == "figure" and fragment.content.summary:
            embedding = model.encode(fragment.content.summary).tolist()
            payload = {
                **structured_metadata,
                'content': fragment.content.summary,
                'type': 'figure',
                'page': page.page_number
            }
            points.append(models.PointStruct(
                id=str(uuid4()),
                vector=embedding,
                payload=payload
            ))

# Upload to Qdrant
qdrant_client.upsert(collection_name=collection_name, points=points)

# Create index for filtering by author
qdrant_client.create_payload_index(
    collection_name=collection_name,
    field_name="authors",
    field_schema="keyword",
)

print(f"Uploaded {len(points)} embeddings to Qdrant")

Key Insight: Table and figure summaries get their own embeddings. When users search for data, they’ll retrieve both text explanations and visual content summaries.

Step 3: Query with Filtering

Combine semantic search with metadata filtering for precise results:

# Query with author filter
query = "Does computer science education improve problem solving skills?"
query_embedding = model.encode(query).tolist()

results = qdrant_client.query_points(
    collection_name=collection_name,
    query=query_embedding,
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="authors",
                match=models.MatchValue(value="William G. Griswold"),
            )
        ]
    ),
    limit=5,
).points

# Display results
for point in results:
    print(f"Title: {point.payload.get('title', 'Unknown')}")
    print(f"Authors: {point.payload.get('authors', 'Unknown')}")
    print(f"Score: {point.score:.4f}")
    print(f"Content: {point.payload.get('content')[:200]}...")
    print("-" * 80)

Output Example:

Title: Teaching Problem Solving Through CS Education
Authors: ['William G. Griswold', 'Jane Smith']
Score: 0.8752
Content: Our study demonstrates that computer science courses significantly 
improve students' problem-solving abilities across multiple domains...
--------------------------------------------------------------------------------

Step 4: Build an Intelligent Agent

Let AI decide when to apply filters based on query intent:

from openai import OpenAI

client = OpenAI()

def smart_search(query: str):
    """Use LLM to extract filters and search Qdrant."""
    
    # Extract filters using LLM
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Extract author names from the query if mentioned. Return JSON."
            },
            {"role": "user", "content": query}
        ]
    )
    
    filters_json = response.choices[0].message.content
    # Parse and build Qdrant filter...
    
    # Search with extracted filters
    results = qdrant_client.query_points(
        collection_name=collection_name,
        query=model.encode(query).tolist(),
        query_filter=build_filter(filters_json),
        limit=5
    )
    
    return results

# Example queries
smart_search("What did John Doe publish about neural networks?")
smart_search("Recent papers on transformer architectures from 2024")

How Rich Embeddings Work

Traditional RAG only embeds text chunks. Tables and figures are ignored or poorly represented. This integration changes the data flow:

During parsing: Tensorlake extracts tables and generates summaries like “Comparison of model accuracy across three datasets showing 5-10% improvement”
During embedding: Both text chunks and table/figure summaries get vectorized separately
During storage: Metadata (authors, year, conference) is stored as filterable payload fields
During retrieval: Queries match against text AND visual content summaries
During response: Results include both explanatory text and data from tables/figures

The key insight: Visual content becomes searchable through AI-generated summaries, dramatically improving RAG accuracy for documents with tables and figures.

Use Cases

Academic Research

Search through research papers with complex layouts. Retrieve both methodology text and experimental results from tables in a single query.

Financial Reports

Parse earnings reports, balance sheets, and regulatory filings. Filter by company, quarter, and fiscal year while searching across narrative and tabular data.

Legal Documents

Handle contracts and regulatory documents with proper structure. Filter by contract type, date range, or parties while searching clause content.

Technical Documentation

Process API docs, manuals, and specifications. Search across text explanations and data tables showing parameters, configurations, or benchmarks.

Medical Literature

Parse clinical studies with methods, results, and patient data tables. Filter by study type, date, or authors while retrieving complete experimental context.

Best Practices

1. Optimize Chunking Strategy

Use semantic chunking by section rather than fixed token limits. Include section headers for context.

parsing_options = ParsingOptions(
    chunking_strategy=ChunkingStrategy.SECTION,  # Not FIXED_SIZE
)

2. Create Strategic Indices

Index frequently filtered fields for performance:

# Common filters
qdrant_client.create_payload_index(collection_name, "authors", "keyword")
qdrant_client.create_payload_index(collection_name, "publication_year", "integer")
qdrant_client.create_payload_index(collection_name, "conference_name", "keyword")

3. Handle Large Tables Intelligently

For tables spanning multiple pages, create summaries with proper context:

enrichment_options = EnrichmentOptions(
    table_summarization_prompt="""Summarize this table's data including:
    1. What the table measures
    2. Key findings or patterns
    3. How it relates to the paper's main argument"""
)

4. Combine Multiple Filter Conditions

Build complex queries that narrow results effectively:

query_filter=models.Filter(
    must=[
        models.FieldCondition(key="publication_year", range=models.Range(gte=2020)),
        models.FieldCondition(key="conference_name", match=models.MatchValue(value="NeurIPS"))
    ]
)

5. Validate Embeddings Quality

Spot-check that table and figure summaries are meaningful:

# Review summaries before embedding
for page in result.pages:
    for fragment in page.page_fragments:
        if fragment.fragment_type == "table":
            print(f"Table summary: {fragment.content.summary}")
            # Ensure it's descriptive, not just "A table of data"

Complete Example

Try the full working example with research paper search:

RAG with Filtering Notebook

Complete code walkthrough including agent-based filtering and result ranking

What’s Next?

Build on this foundation:

ChromaDB Integration - Add citation tracking to results
Chonkie Integration - Advanced semantic chunking strategies

Learn more about document AI:

Blog: Fix Broken Context in RAG - Why chunking matters

Resources

Need Help?

Join our community to discuss RAG architectures:

Slack Community

Documentation Index

​Why Use Tensorlake + Qdrant?

​Installation

​Quick Start

​Step 1: Parse Documents with Tensorlake

​Step 2: Create Embeddings and Store in Qdrant

​Step 3: Query with Filtering

​Step 4: Build an Intelligent Agent

​How Rich Embeddings Work

​Use Cases

​Academic Research

​Financial Reports

​Legal Documents

​Technical Documentation

​Medical Literature

​Best Practices

​1. Optimize Chunking Strategy

​2. Create Strategic Indices

​3. Handle Large Tables Intelligently

​4. Combine Multiple Filter Conditions

​5. Validate Embeddings Quality

​Complete Example

RAG with Filtering Notebook

​What’s Next?

​Resources

​Need Help?

Why Use Tensorlake + Qdrant?

Installation

Quick Start

Step 1: Parse Documents with Tensorlake

Step 2: Create Embeddings and Store in Qdrant

Step 3: Query with Filtering

Step 4: Build an Intelligent Agent

How Rich Embeddings Work

Use Cases

Academic Research

Financial Reports

Legal Documents

Technical Documentation

Medical Literature

Best Practices

1. Optimize Chunking Strategy

2. Create Strategic Indices

3. Handle Large Tables Intelligently

4. Combine Multiple Filter Conditions

5. Validate Embeddings Quality

Complete Example

What’s Next?

Resources

Need Help?