Inference Server REST APIs
Since the NuPIC Inference Server is built on top of the Triton Inference Server, we can make use of Triton's REST APIs.
Here are a few that you might frequently use. For simplicity, the examples below assume that the client and Inference Server are running on the same machine. If you are using remote clients, please replace localhost
with your Inference Server's IP address.
Check whether Inference Server is running
curl -v http://localhost:8000/v2/health/live
Expected output:
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> GET /v2/health/live HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain
<
* Connection #0 to host localhost left intact
List all models
curl -X POST -v http://localhost:8000/v2/repository/index
Expected output:
* Host localhost:8000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying [::1]:8000...
* connect to ::1 port 8000 from ::1 port 46418 failed: Connection refused
* Trying 127.0.0.1:8000...
* Connected to localhost (127.0.0.1) port 8000
> POST /v2/repository/index HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 1908
<
{ [1908 bytes data]
100 1908 100 1908 0 0 1321k 0 --:--:-- --:--:-- --:--:-- 1863k
* Connection #0 to host localhost left intact
[
{
"name": "bert-base-cased-tokenizer",
"version": "1",
...
{
"name": "zephyr-7b-v0-wtokenizer"
}
]
Load a model
curl -X POST http://localhost:8000/v2/repository/models/nupic-sbert.base-v3/load
Get model metadata
curl http://localhost:8000/v2/models/nupic-sbert.base-v3
Expected output showing input payload structure:
{
"name": "nupic-sbert.base-v3",
"versions": [
"1"
],
"platform": "ensemble",
# Expected input payload structure
"inputs": [
{
"name": "TEXT",
"datatype": "BYTES",
"shape": [
-1
]
}
],
"outputs": [
{
"name": "encodings",
"datatype": "FP32",
"shape": [
-1,
-1
]
}
]
}
Inference
Create payload.json
based on model metadata:
{
"inputs": [
{
"name": "TEXT",
"shape": [1],
"datatype": "BYTES",
"data": ["This is a test sentence."]
}
]
}
Send payload:
curl -X POST -H "Content-Type: application/json" \
-d @payload.json \
http://localhost:8000/v2/models/nupic-sbert.base-v3/infer
Expected output:
{
"model_name": "nupic-sbert.base-v3",
"model_version": "1",
"parameters": {
"sequence_id": 0,
"sequence_start": false,
"sequence_end": false
},
"outputs": [
{
"name": "encodings",
"datatype": "FP32",
"shape": [
1,
768
],
"data": [
0.49580085277557375,
0.1195397824048996,
0.06556889414787293,
-0.42177122831344607,
...
Unload a model
curl -X POST http://localhost:8000/v2/repository/models/nupic-sbert.base-v3/unload
Updated 3 months ago