Overview

Tensorlake Workflows enable orchestrating data ingestion and transformation. The workflows are distributed and can handle ingesting and processing large volumes of data in parallel. They are durable so it’s guaranteed that all ingested data will be processed.

​

Overview of Workflows

Workflows are defined by writing Python functions which processes data. The functions are connected to each other to form a workflow which captures the end-to-end data ingestion and transformation logic.

We use Graphs to represent the workflow, to enable parallel execution of disjoint parts of the workflows.

​

Tensorlake Functions

Tensorlake functions are the building blocks of workflows. They are Python functions decorated with the @tensorlake_function decorator.

from tensorlake import tensorlake_function

image = Image()
  .run("pip install transformers")
  .build()

@tensorlake_function(image=image, input_encoding="json")
def my_function(data: str) -> int:
    return len(data)

You can write any Python code inside the function, and depend on any Python package. The functions are run inside a container of an image built from the specification of the Image object.

​

Graphs

You can string together multiple functions to form a workflow.

g = Graph(start_node=my_function, name="my_workflow", description="My workflow")
g.add_edge(my_function, my_function1)
g.add_edge(my_function, my_function2)

In the above example, my_function is the start node of the workflow. The input to the workflow is passed to my_function.

Workflows are exposed as HTTP endpoints, the body of the request will be passed to the start node of the workflow, in this case my_function.

curl -X POST https://api.tensorlake.com/workflows/my_workflow \
  -H "Content-Type: application/json" \
  -d '{"data": "Hello, world!"}'

​

Outputs

Tensorlake workflows allow retrieving the outputs of any function in the workflow.

from tensorlake import RemoteGraph

g = RemoteGraph(name="my_workflow")

g.outputs(my_function1)

​

Automatic ScaleOuts

When a function returns a list, Tensorlake automatically processes each item in parallel by creating multiple instances of the function receiving the items.

@tensorlake_function(image=image, input_encoding="json")
def chunk_essay(essay: str) -> List[str]:
    return essay.split("\n\n")

@tensorlake_function(image=image, input_encoding="json")
def count_words(text: str) -> int:
    return len(text.split(" "))

In the above example, count_words will be called in parallel for each chunk of the essay. If you are using tools like Airflow, you would need to use something like Spark, Ray or Dask to parallelize the processing. Tensorlake does this automatically.