Files larger than 10MB can be efficiently uploaded to Tensorlake via the API by first requesting a pre-signed URL. Once you have the pre-signed URL, you can upload the file to the URL using any HTTP client.

You can try this out on the API Playground here.

If you are using the Python SDK, you can ignore this page, as the SDK internally uses this method to upload large files.

1

Request a pre-signed URL

Initiate the request with some information about the file you want to upload. Pass the file_size in bytes, filename and mime_type of the file. You will also have to pass the sha256_checksum of the file so that we can verify the integrity of the file after the upload is complete.

curl --request POST \
--url https://api.tensorlake.ai/documents/v1/files_large \
--header 'Authorization: Bearer tl_apiKey_XXX' \
--header 'Content-Type: application/json' \
--data '{
"file_size": 1000,
"filename": "myfile.pdf",
"sha256_checksum": "XXX",
"mime_type": "application/pdf"
}'

The response will contain a presigned_url that you can use to upload the file to.

{
    "id":"tl-presigned-XXX",
    "presigned_url":"https://inkwell-server-prod-XXX.s3.us-east-1.amazonaws.com/XXX"}
2

Upload the file to the pre-signed URL

Once you have the pre-signed URL, you can upload the file to the URL using any HTTP client.

curl -X PUT 
-H 'Content-Type: multipart/form-data'
-F 'file=@myfile.pdf' 'https://inkwell-server-prod-XXX.s3.us-east-1.amazonaws.com/XXX'
3

Complete the upload

Once you have uploaded the file, you need to call the /files_large/{presigned_url_id} endpoint to complete the upload. Use the id from the response of the previous step to complete the upload.

curl --request POST \
--url https://api.tensorlake.ai/documents/v1/files_large/tl-presigned-XXX \
--header 'Authorization: Bearer tl_apiKey_XXX'