Key Concepts

Indexify represents data-intensive AI workflows as graphs, where each node is a function operating on data, and edges represent data flow between functions. Here are the main components -

​

Graphs

These are multi-step workflows created by connecting multiple functions together. Some attributes of Graphs-

• Node: Represents a function that operates on data.
• Start Node: which is the first funciton that is executed when the graph is invoked.
• Edges: Represents data flow between functions.
• Conditional Edge: Evaluates input data from the previous function and decide which edges to take. They are like if-else statements in programming.

​

Functions

They are regular Python functions, decorated with @indexify_function() decorator.

Function can be executed in a distributed manner, and the output is stored so that if downstream functions fail, they can be resumed from the output of the function.

There are various other parameters, in the decorator that can be used to configure retry behaviour, placement constraints, and more.

​

Namespaces

Namespaces as logical abstractions for storing related content. This allows for effective data partitioning based on security requirements or organizational boundaries.

​

Programming Model

​

Map

Automatically parallelize functions across multiple machines when a function returns a sequence and the downstream function accepts only a single element of that sequence.

@indexify_function()
def fetch_urls(num_urls: int) -> list[str]:
    return [
        'https://example.com/page1',
        'https://example.com/page2',
        'https://example.com/page3',
    ]

# scrape_page is called in parallel for every element of fetch_url across
# many machines in a cluster or across many worker processes in a machine
@indexify_function()
def scrape_page(url: str) -> str:
    content = requests.get(url).text
    return content

Use Cases: Generating Embedding from every single chunk of a document.

​

Reducing/Accumulating from Sequences

Reduce functions in Indexify aggregate outputs from one or more functions that return sequences. They operate with the following characteristics:

• Lazy Evaluation: Reduce functions are invoked incrementally as elements become available for aggregation. This allows for efficient processing of large datasets or streams of data.
• Stateful Aggregation: The aggregated value is persisted between invocations. Each time the Reduce function is called, it receives the current aggregated state along with the new element to be processed.

@indexify_function()
def fetch_numbers() -> list[int]:
    return [1, 2, 3, 4, 5]

class Total(BaseModel):
    value: int = 0

@indexify_function(accumulate=Total)
def accumulate_total(total: Total, number: int) -> Total:
    total.value += number
    return total

Use Cases: Aggregating a summary from hundreds of web pages.

​

Dynamic Routing

Functions can route data to different nodes based on custom logic, enabling dynamic branching.

@indexify_function()
def handle_error(text: str):
    # Logic to handle error messages
    pass

@indexify_function()
def handle_normal(text: str):
    # Logic to process normal text
    pass

# The function routes data into the handle_error and handle_normal based on the
# logic of the function.
@indexify_router()
def analyze_text(text: str) -> List[Union[handle_error, handle_normal]]:
    if 'error' in text.lower():
        return [handle_error]
    else:
        return [handle_normal]

Use Cases: Processing outputs differently based on classification results.