External Storage Troubleshooting

Registration Request Errors

When sending API requests to register items, various errors can be encountered. The following lists common errors and their causes:

• 401 - NOT_AUTHENTICATED: The API key used for the request was invalid, or lacks sufficient permissions. API keys need to be passed in a specific format in request headers, for example:

headers = {
    "Content-Type": "application/json",
    "Accept": "application/json",
    "Authorization": "ApiKey YOUR_API_KEY_HERE"
}

API keys used to register items require the Add data to datasets permission:

• Not found: This indicates that the value of at least one of the following fields could not be found:

  • team_slug: Must match your sluggified team name

  • dataset_slug: Must match your sluggified dataset name

  • storage_name: Must match the name of your storage integration in your configuration. For example:

    

• 422 - INVALID_REQUEST: The request either contained an unexpected field, was missing an expected field, or contains a syntax error. The error will contain information about the specifics of the problem. Please ensure you are using the correct API endpoint since the read-only & read-write endpoints expect different payload fields. If you need more guidance, please refer to this article and bear in mind the differences between the read-write & read-only registration payloads

• ALREADY_EXISTS: This indicates that an item being registered was trying to be assigned a name that is already taken at the same directory location within the V7 dataset

In-Platform Errors

If your registration requests are successful, but items aren't available in the platform then please see the below for common issues and their causes:

• Items stuck in processing: Although read-only items will never enter processing, read-write items require some processing before they are available. Most items will process immediately, but videos & larger files can take some time to complete. If you are experiencing excessive processing times, please get in touch with our support team at [email protected]

• Items register in error: If items are ending up in the Error state, you can click on Details for further information about the error:

The most common error here is AccessDenied:

This is most often caused by either:

• 1: A mismatch between a storage key and a file's true location in external storage
• 2: Lack of permissions

In both cases, please review your external storage location's permissions against our guides (AWS, Azure, GCP) and ensure that your storage keys are correctly formatted.

• If items can be opened in the workview but do not load, it's possible that:
  • Files are no longer available - After successfully registering external files, all relevant files must not be moved or deleted within your external storage. Please check that this is the case
  • V7 lacks permissions to access your external files. Please verify all permissions against our guide for your cloud provider: AWS, Azure, GCP

AWS S3 KMS Encryption

If you are using S3 KMS encryption then you must include the following permissions as well as GetObject and/or PutObject:

"Action": [  
        "kms:Encrypt",  
        "kms:Decrypt",  
        "kms:ReEncrypt*",  
        "kms:GenerateDataKey*",  
        "kms:DescribeKey"  
    ]

AWS S3 Object Ownership

To ensure that Darwin has access to all objects to be registered, please ensure that ownership ACLs are disabled. If this is enabled, objects written to your bucket by other accounts may not be accessible by Darwin:

Alternatively, you can enable ownership ACLs, but you'll need to whitelist a set of V7 server IP addresses in addition to the IPs of the users that will access the items in V7. Please get in touch with our support team at [email protected] for further information.

📘

Further Support

If you still need assistance then reach out to our Support by emailing [email protected]