Durable Execution

​

Request Replay API

curl \
"https://api.tensorlake.ai/applications/$APPLICATION_NAME/requests/$REQUEST_ID/replay" \
-H "Authorization: Bearer $TENSORLAKE_API_KEY" \
--json '{}'

​

Application code upgrade

​

Replay modes

import random
from tensorlake.applications import application, function

@function()
def foo():
    print("foo")

@function()
def bar():
    print("bar")

@application()
@function()
def my_workflow_app():
    # succeeded in the original request run,
    # skipped in the replayed run
    foo()
    if random.random() < 0.5:
        # never called in the original request run,
        # called in the replayed run, Tensorlake detects this
        bar()
    # ... more Tensorlake function calls

• Conditional execution of code depending on current time, database state, values returned by external APIs, etc.
• Changing order of Tensorlake function calls depending on duration of external API calls, LLM calls, etc (aka race conditions).
• Change of Tensorlake function calls in upgraded application code.

​

Adaptive replay

​

Strict replay

​

How function calls are matched

• Function call type (i.e. “function_call”, “map”, “reduce”).
• Function name.
• Parent function call fingerprint.
• Function call sequence number in the parent function call.
• Other information to ensure that changes in function call tree structures are detected.

• Changing function parameters in application code doesn’t affect replay behavior. A new function call with different parameters still matches the previous function call. This enables seemless application code upgrades without affecting replays.
• Passing different values (e.g., random numbers, current time) as function parameters doesn’t affect replay behavior. A function call with a different random number passed into it still matches its previous function call where the random number was different.
• If sequence of function calls changed in the latest application code then the replayed function calls will not match the previous function calls. In this case the replay behavior depends on the selected replay mode (adaptive or strict).
• If function calls are started in an arbitrary order (i.e. with a random delay) then the order of function calls would differ between the original request run and the replayed run even without application code changes. In this case the replay behavior depends on the selected replay mode (adaptive or strict). Application code should avoid arbitrary function call ordering to ensure consistent behavior during request replays and reuse of previously completed work.

​

Automatic retries

​

Disabling durable execution

from tensorlake.applications import application, function
from magic_llm import ask_llm

@function(durable=False)
def get_current_weather() -> str:
    return ask_llm("What's the weather now in San Francisco?")

​

Best practices for durable Tensorlake applications

• Wrap every external call (LLM, API, database, etc.) in a Tensorlake function to make these calls durable and avoid repeating work. If a framework is doing these calls then use framework customization points (e.g., callbacks, hooks, decorators, etc.) to wrap the calls in Tensorlake functions.
• Design your application code to be deterministic to ensure that replays follow the same execution path and thus reuse previously finished work.
• If your Tensorlake functions have external side effects (e.g., sending emails, modifying databases), ensure that these side effects are idempotent or can be safely retried without causing issues.
• Disable durability for functions that must always run on request replay or retry.
• If strict mode and code upgrade to latest are used in a replay then the latest application code needs to be fully backward compatible with the original request code to avoid failing the replay.

​

Human in the loop and external events

from tensorlake.applications import application, function, RequestError
from approval_system import wait_for_approval_request, get_approval_message, create_approval_request, ApprovalDeniedError

@function()
def create_approval(user_id: str, action: str) -> str:
    """Wraps create_approval_request call to make it durable."""
    return create_approval_request(user_id, action)


@function(timeout=300)  # 5 minute timeout
def wait_approval(approval_id: str) -> str:
    """Waits for an approval request to be completed and returns its approval message.

    Raises RequestError if the approval was denied. This fails the current request immediately.
    """
    try:
        wait_for_approval_request(approval_id)
    except ApprovalDeniedError as e:
        raise RequestError(f"Approval '{approval_id}' was denied: {e}") from e
    return get_approval_message(approval_id)


@application()
@function()
def user_authorization_workflow(user_id: str, action: str) -> str:
    approval_id: str = create_approval(user_id, action)
    approval_message: str = wait_approval(approval_id)
    return f"Approval received: {approval_message}"

​

Comparison with Temporal

​

Next