Inference Server CLI
The NuPIC Inference Server is started using the nupic/nupic_inference.sh
script.
To use this script, make sure it is made executable using chmod +x nupic/nupic_inference.sh
, then use the following syntax:
./nupic_inference.sh start [OPTIONS] | stop
Commands
Command | Description |
---|---|
start | Starts a new NuPIC Inference Server along with monitoring tools (Grafana, Prometheus). |
stop | Stops and removes the running NuPIC Inference Server and monitoring tools. |
Options
Option | Value | Description |
---|---|---|
--cpus | CPU indices | Defines the explicit CPUs which NuPIC Inference Server will run on. If not specified, NuPIC Inference Server runs on all available CPU cores. This can be useful when running NuPIC Inference Server concurrently with other workloads on the same machine. |
--expose | None | Expose NuPIC Inference Server's HTTP port (8000), GRPC" port (8001), and metrics port (8002) to all hosts. If not specified, all these ports are exposed on the loopback interface. This is required to allow remote clients to make requests to the Inference Server. |
--gpus | None | Allow NuPIC Inference Server to use all GPU devices. If not specified, NuPIC Inference Server runs with CPU only. Note that each model will still have to be individual configured to use the GPU using [instance_group.kind](doc:configuration#instance-group) on the respective config.pbtxt . |
--log-level | Integer | Set custom log verbosity for NuPIC Inference Server. The level is an integer value. 0 means verbose logging is disabled. 1 and 2 provides different level of verbose loggings. Level 2 enables most detailed logging. If not specified, the default value is 0. |
--shm-size | Bytes | Set custom shared memory size (/dev/shm) for NuPIC Inference Server. If not specified, the default shared memory size is 512 MB (512m). |
--enable-caching | None | Enable response caching in NuPIC Inference Server. If not specified, caching is disabled. |
--cache-size | Bytes | Set custom cache size for NuPIC Inference Server. If not specified, the default cache size is 1GB. |
--additional_args | Various | Any additional arguments to pass to Triton Server; e.g., NUMA node configurations. |
Updated 3 months ago