Bring Your Own Model

Learn how to use your own externally hosted model in V7

Darwin allows you to bring the models that you host on your infrastructure and use them within the V7 platform.

These models behave the same as the ones trained using Darwin's own AutoML. Most importantly, they allow you to customize your workflow stages. With this, you can register a model that is exposed via HTTP and manage it the same way you do the ones trained using Darwin.

Minimal requirements

For the integration to be meaningful to Darwin, your model needs to conform to some specific requirements. It has to:

  • Expect application/json request payloads
  • Respond with application/json responses
  • Both request and response payloads conforming to the specific JSON schemas
  • Handle the POST /infer requests, accepting images as input and responding with the list of results
  • Handle the GET /classes, responding with the list of label types (along with class names) encoded as JSON

Registering the model via the REST API

It's possible to register the model via the API.

The following snippet from shell shows how to do it:

APIKEY="<your-key-here>"

# the following assumes that you have `jq` installed:
TEAM_ID=$(
  curl \
    -XGET \
    -s \
    -H "content-type: application/json" \
    -H "authorization: ApiKey $APIKEY" \
    "https://darwin.v7labs.com/api/users/token_info" \
    | jq ".selected_team.id"
)

# the rest you'll need to specify yourself:
MODEL_NAME="your-model-name"
EXTERNAL_URL="http://your.externalmodel.ai/api"

# the following are optional, feel free to omit them in the
# payload below if not needed:
BASIC_AUTH_USERNAME="some-user-1"
BASIC_AUTH_PASSWORD="some-password-1"
AUTH_SECRET="some-secret-1"
AUTH_SECRET_HEADER="X-MY-AUTH"

# you'll need to have your model deployment respond to the
# mandatory /classes endpoint. here, you'll need to get the
# classes response and pass it into Darwin.
# 
# again, feel free to leave out parts of authentication that
# don't apply in your case:
CLASSES=$(
  curl \
  -XGET \
  -H "content-type: application/json" \
  -H "authorization: Basic $(echo "$BASIC_AUTH_USERNAME:$BASIC_AUTH_PASSWORD" | base64)" \
  -H "$AUTH_SECRET_HEADER: $AUTH_SECRET" \
  -s "$EXTERNAL_URL/classes"
)

# now that we have all the parameters, let's actually register
# the model in Darwin:
API_URL="https://darwin.v7labs.com/ai"

curl  --request POST \
  --header "Content-Type: application/json" \
  --header "authorization: ApiKey $APIKEY" \
  --url "$API_URL/trained_models" \
  --data "{
        \"name\": \"$MODEL_NAME\",
        \"team_id\": \"$TEAM_ID\",
        \"external_url\": \"$EXTERNAL_URL\",
        \"basic_auth_username\": \"$BASIC_AUTH_USERNAME\",
        \"basic_auth_password\": \"$BASIC_AUTH_PASSWORD\",
        \"auth_secret\": \"$AUTH_SECRET\",
        \"auth_secret_header\": \"$AUTH_SECRET_HEADER\",
        \"is_external\": true,
        \"classes\": $CLASSES
      }"

🚧

API Key Permissions

You need to make sure that the API Key used in registering the model has permission to create models

Security

The form allows you to choose from between two forms of authentication supported currently:

  • HTTP Basic Authentication
  • Secret Key

🚧

Consider using SSL!

You should always use SSL, otherwise, a "man in the middle" attack could be performed, sniffing the username and password or a secret key.

The "secret key" authentication scheme allows you to specify the HTTP header name and its value. As an example, your exposed model could expect requests with an X-Auth header and some specific, secret value.

The credentials that you provide at this step are always securely stored in Darwin's database and encrypted at rest.

Communication schema

GET {base-url}/classes

When your model is being registered, a GET {base-url}/classes request is made from Darwin. The response determines the type of annotations and class names that will be available to link in the workflows.

Classes response schema

Here’s an example response for a simple classification model with two classes - valid and invalid:

[
  {
    "name": "valid", 
    "type": "tag"
  }, 
  {
    "name": "invalid", 
    "type": "tag"
  }
]

The list of supported annotation types is as follows:

  • bounding_box
  • cuboid
  • ellipse
  • line
  • keypoint
  • polygon
  • skeleton
  • tag

POST {base-url}/infer

Whenever Darwin needs to run inference using your model, it makes a POST {base-url}/infer request. This happens when e. g:

  • You run an inference from the model’s playground in the UI
  • Your model is used in an "AI Model stage" of the annotation workflow

In the case of the AI Model stage, each eligible image is sent to inference through this POST request. Darwin encapsulates the images or videos within the request payload and expects a response that conforms to a known Darwin JSON derived schema.

If the request times out or fails for whatever reason - the inference moves on to the next item in the dataset. In the case of the AI Model stage, such items would remain at the same state so it’s very easy to filter and see the items that the model failed to respond to.

Request and response schemas

The JSON response is validated using the JSON Schema as specified below.

Inference request JSON schema

$id: https://darwin.v7labs.com/schemas/external-models/inference-request.schema.json
$schema: https://json-schema.org/draft/2020-12/schema
title: Inference request
description: Provides an image or a video to run the inference on
type: object
oneOf:
- required:
  - image
- required:
  - video
properties:
  image:
    $schema: https://json-schema.org/draft/2020-12/schema
    anyOf:
    - required:
      - base64
    - required:
      - url
    description: An image to run inference on
    properties:
      base64:
        type: string
      url:
        type: string
    type: object
  video:
    $schema: https://json-schema.org/draft/2020-12/schema
    description: A video to run inference on
    oneOf:
    - required:
      - url
    - required:
      - frame_urls
    properties:
      frame_urls:
        items:
          oneOf:
          - required:
            - base64
          - required:
            - url
          properties:
            base64:
              type: string
            url:
              type: string
          type: object
        type: array
      url:
        type: string
    type: object

Inference Request Example

Below is an example inference request. If the request is as a part of the Auto Annotate feature then there will be bounding box coordinates as shown. Otherwise, the params will be an empty dictionary.

{
    "message": "infer request",
    "details": {
        "image": {
            "url": "<image url>
        },
        "params": {
            "bbox": {
                "h": 400,
                "w": 300,
                "x": 550,
                "y": 120
            },
            "id": "d0dca498-dd0e-4150-b94e-253f06b9caf2",
            "image": {
                "url": "<image url>"
            }
        }
    }
}

Note: The request.json payload will only include the image and paramsobjects

Inference response JSON schema

$id: https://darwin.v7labs.com/schemas/external-models/inference-response.schema.json
$schema: https://json-schema.org/draft/2020-12/schema
title: Inference response
description: JSON response to an inference request. Encapsulates a list of results
type: object
properties:
  results:
    items:
      properties:
        attributes:
          $ref: '#/$defs/attributes'
        bounding_box:
          $ref: '#/$defs/bounding_box'
        confidence:
          $ref: '#/$defs/confidence'
        cuboid:
          $ref: '#/$defs/cuboid'
        directional_vector:
          $ref: '#/$defs/directional_vector'
        ellipse:
          $ref: '#/$defs/ellipse'
        keypoint:
          $ref: '#/$defs/keypoint'
        line:
          $ref: '#/$defs/line'
        name:
          type: string
        polygon:
          $ref: '#/$defs/polygon'
        skeleton:
          $ref: '#/$defs/skeleton'
        tag:
          type: object
        text:
          $ref: '#/$defs/text'
      oneOf:
      - required:
        - bounding_box
      - required:
        - cuboid
      - required:
        - directional_vector
      - required:
        - ellipse
      - required:
        - line
      - required:
        - keypoint
      - required:
        - polygon
      - required:
        - skeleton
      - required:
        - text
      - required:
        - tag
      required:
      - name
      - confidence
      type: object
    type: array
  status:
    enum:
    - succeeded
    - failed
    type: string
required:
- status
$defs:
  attributes:
    properties:
      attributes:
        items:
          type: string
    required:
    - path
    type: object
  bounding_box:
    properties:
      h:
        type: number
      w:
        type: number
      x:
        type: number
      y:
        type: number
    required:
    - h
    - w
    - x
    - y
    type: object
  confidence:
    maximum: 1
    minimum: 0
    type: number
  cuboid:
    properties:
      back:
        $ref: '#/$defs/bounding_box'
      front:
        $ref: '#/$defs/bounding_box'
    required:
    - back
    - front
    type: object
  directional_vector:
    properties:
      angle:
        minimum: 0
        type: number
      length:
        minimum: 0
        type: number
    required:
    - angle
    - length
    type: object
  ellipse:
    properties:
      angle:
        minimum: 0
        type: number
      center:
        $ref: '#/$defs/keypoint'
      radius:
        $ref: '#/$defs/keypoint'
    required:
    - angle
    - center
    - radius
    type: object
  keypoint:
    properties:
      x:
        minimum: 0
        type: number
      y:
        minimum: 0
        type: number
    required:
    - x
    - y
    type: object
  line:
    properties:
      path:
        items:
          $ref: '#/$defs/keypoint'
    required:
    - path
    type: object
  polygon:
    $ref: '#/$defs/line'
  skeleton:
    properties:
      nodes:
        items:
          properties:
            occluded:
              type: boolean
            x:
              minimum: 0
              type: number
            y:
              minimum: 0
              type: number
          required:
          - occluded
          - x
          - y
          type: object
        type: array
    required:
    - nodes
    type: object
  text:
    properties:
      text:
        type: string
    required:
    - text
    type: object

Inference Response Example

Below is an example inference response for a complex polygon annotation produced by an external model:

{
  "results":[
    {
      "confidence":1,
      "label":"background",
      "name":"background",
      "polygon":{
        "path":[
          {
            "x":677.0,
            "y":896.0
          },
          {
            "x":676.0,
            "y":896.0
          },
          {
            "x":675.0,
            "y":897.0
          },
          {
            "x":674.0,
            "y":897.0
          },
          {
            "x":675.0,
            "y":897.0
          },
          {
            "x":676.0,
            "y":898.0
          },
          {
            "x":675.0,
            "y":899.0
          },
          {
            "x":674.0,
            "y":899.0
          },
          {
            "x":671.0,
            "y":902.0
          }
        ],
        "additional_paths":[
            [
          {
            "x":1,
            "y":1
          },
          {
            "x":2,
            "y":2
          },
          {
            "x":3,
            "y":3
          }
          ]
        ]
      }
    }
  ],
  "status":"succeeded"
}

Note: additional_paths contains other paths in a complex polygon (such as the hole in a donut). It should also be noted that path is a list whereas additional_paths is a list of lists.

The example above is indicative of the JSON structure and not a typical annotation.

Model Setup and Mapping Classes

Once the above is complete, you should now enable the model and map your external classes to those in V7.

For auto-annotate the instructions are here.

For the AI model stage the instructions are here.

Example model integration

Here’s a simplified listing of a “Bring Your Own Model” that handles inference requests via HTTP using Flask:

import base64
import tempfile
import urllib

from PIL import Image
from flask import Flask, request, jsonify
from your_model import Model


model = Model()


def resolve_image(image_spec: dict[str, str]) -> Image:
    if "url" in image_spec:
        with tempfile.NamedTemporaryFile() as file:
            urllib.request.urlretrieve(image_spec["url"], file.name)
            return Image.open(file.name).convert("RGB")
    elif "base64" in image_spec:
        with tempfile.NamedTemporaryFile() as file:
            file.write(base64.decodebytes(image_spec["base64"].encode("utf-8")))
            return Image.open(file.name).convert("RGB")
    else:
        raise ValueError("Invalid image spec")

@app.route("/api/classes", methods=["GET"])
def classes():
    return jsonify(model.classes)

@app.route("/api/infer", methods=["POST"])
def infer():
    payload = request.json

    try:
        image = resolve_image(payload["image"])
        class_name, confidence = model(image)

        return jsonify(
            status="succeeded",
            results=[
                {
                    "name": class_name,
                    "label": class_name,
                    "confidence": confidence,
                    "tag": {},
                }
            ],
        )
    except:
        print(traceback.format_exc())
        return jsonify(status="failed", results=[])

if __name__ == "__main__":
    app.run(debug=True, host="0.0.0.0")

🚧

External Model Latency Requirements

To avoid timeout errors, your model should take less than 30 seconds to complete.