Skip to content
Data Ocean

Upload a multipart file

A multipart upload allows you to upload a file in parts. Choose multipart uploads when you have a slower connection or are working with large files. Multipart uploads are resumable uploads. Performing a multipart upload is the only way for the cloud service to keep the bytes you sent if a connection is interrupted.

Diagram

After each part is uploaded, the complete action recompiles the parts. The files is available under a single file ID and path with a single download URL.

  • The minimum size for a multipart upload part is 5 MB (except for the last part, which may be less than 5 MB).
  • The maximum part size is 5 GB.
  • The number of parts of a multipart upload cannot exceed 10,000.

Use the following steps to upload a multipart file to Data Ocean.

The steps below provide an outline. See Data Ocean API definition for parameter details.

1. Create a new file object

Plan your file structure ahead of time. Use the path parameter to easily and immediately organize files into a directory structure. Include a path before the file name to automatically create a directory the first time that path is used. Upload additional files using the same base path to group files into a directory.

  • API CALL:
POST {base URL}/api/files

  • HEADERS:

    • Accept: application/json
    • Content-Type: application/json
    • Authorization: Bearer {bearer token}

    Include Trimble Identity v3 OAuth Bearer token for your application.

  • BODY:

    • path: {name of file and location of file}
    • regions: {regions}
    • multipart: true

Base your body on the example request code sample below. Use the path to name your file and set the file location to organize your files. This example shows the minimum required request body, see Data Ocean API definition for parameter details and additional optional parameters.

{
    "file": {
        "path": "/user/newfile.tiff",
        "regions": ["us1"],
        "multipart": true
    }
}

After the operation is completed, the JSON response includes the id and full path values for the file and an upload URL. This URL is used to upload file parts in the next step.

Example response:

{
    "file": {
        "created_at": "2019-07-08T23:15:36.625Z",
        "fileset": false,
        "id": "555ca431-55ac-55ac-129f-45d79b65bc6b",
        "malware_status": "UNSCANNED",
        "md5": null,
        "metadata": {},
        "multipart": true,
        "multipart_id": "minf4FMlKxmhdsASUQarKfNaaXYevVBM7yFa5EY4WXcE7mrVooi8GaxjmWcXckN07MdAUEgrF5DwKCp42MLK36B0QI4ifBDh44OaVCPNF.tsAPxn49Pttsya8e7mqhHCNLPDoyhUwhMGit6uQL1y5w--",
        "path": "555cs1a68dea48b68eaf16156c070155:/multipart-test.zip",
        "region_preferences": ["us1"],
        "regions": [],
        "restricted_access": true,
        "size": 6291456,
        "status": "UPLOADABLE",
        "upload": {
            "type": "MULTIPART",
            "url": "https://fs-us1.tdata-cdndev.com/s/555ca431-55ac-55ac-129f-45d79b65bc6b.1?partNumber=*&uploadId=minf4FMlKxmhdsASUQarKfNaaXYevVBM7yFa5EY4WXcE7mrVooi8GaxjmWcXckN07MdAUEgrF5DwKCp42MLK36B0QI4ifBDh44OaVCPNF.tsAPxn49Pttsya8e7mqhHCNLPDoyhUwhMGit6uQL1y5w--&Policy={policy}&Signature={signature}&Key-Pair-Id={key-pair}"
            },
        "version": 1
    }
}

2. Upload file parts

There is a query string parameter “partNumber=*” in the upload URL from the previous request. Substitute the “*” value with the number of the current part you are uploading.

Separate the file into parts, and upload each to the URL with parameter value substituted. Part numbering starts with "1" (not zero).

  • API CALL:
PUT {UPLOAD URL}
  • BODY
{PART DATA}

Your response will include an ETag in the header. Keep track of the ETag responses for each upload. You will need them for the completion call.

3. Run assemble API call

After all of your parts are uploaded, run the API call to complete the multipart upload. Use the file id from the first step.

  • API CALL:
POST {base URL}/api/files/{id}/assemble

  • HEADER:

    • Accept: application/json
    • Value: Bearer {bearer token}

    Include Trimble Identity OAuth Bearer token for your application (see get access token instructions for more information).

  • BODY: * parts: {etag, part_number}

Base your body on the example request code sample below. Use the part number and ETag information you received for each file part in the previous step.

{
    "upload": {
        "parts": [
            {
                "etag": "fr57idd4h75de5hrr578994jifee4422",
                "part_number": 4
            },
            {
                "etag": "fr44466y7urr4wf4h8k2ee467fr432e4",
                "part_number": 3
            },
            {
                "etag": "t5543334reewg66qq3r5ffy78fr4tf33",
                "part_number": 2
            },
            {
                "etag": "33g567u8ii44t6ygr543fr323009ir83",
                "part_number": 1
            }
        ]
    }
}

Example response:

{
  "id": "68d29087-0121-4d70-b831-1581b95e9ad6",
  "path": "b247170dfdd3400f9be4d4217f99b8b8:/multipartfile",
  "size": 203912,
  "restricted_access": true,
  "status": "UPLOADABLE",
  "regions": [
    "us1"
  ],
  "region_preferences": [
    "us1"
  ],
  "fileset": false,
  "version": 4,
  "malware_status": "CLEAN",
  "last_scanned_at": "2019-07-09T20:33:18Z",
  "created_at": "2019-07-09T20:33:18Z",
  "updated_at": "2019-07-09T20:33:18Z",
  "expires_at": "2018-12-13T22:48:59Z",
  "download": {
    "url": "https://s3.amazon.com/sflkjesfe",
    "us1": "https://s3.amazon.com/sflkjesfe",
    "eu1": "https://s3.amazon.com/3kbi239dm"
  },
  "path_template": "string",
  "upload": {
    "type": "MULTIPART",
    "url": "https://s3.amazon.com/0j3nbkig"
  }
}

4. Validate the upload

Run a GET API call to check the status. The value should be AVAILABLE.

  • API CALL:
GET {base URL}/api/files/{id}

—OR—

GET {base URL}/api/files/api/files?path={file_path}

  • HEADER:

    • Accept: application/json

    This may be automatically added by your development tool. Postman and curl add this header to all requests.

    • Value: Bearer {bearer token}

    Include Trimble Identity OAuth Bearer token for your application.

 

Reference Materials

Data Ocean API reference