Guides
Log In
Guides

Connecting with GRPC over SSL

Suggest Edits

By default, most of the tutorials in this documentation show the NuPIC inference client connecting to the Inference Server over a HTTP connection.

While a HTTP connection provides a user-friendly REST API, in some cases there are good reasons to instead use the GRPC protocol that is also supported by the NuPIC Inference Server. For example, a GRPC connection might be preferable when you require higher throughput and/or lower latency.

Beyond performance, data privacy is also a key feature of NuPIC. To further ensure the security of data, the Inference Server supports in-flight encryption over SSL (specifically TLS).

This page shows you how to connect a remote Python client to the NuPIC Inference Server over GRPC and SSL.

Before You Start

Please ensure that you have already installed the Inference Server and Python clients. If not, please refer to the links below.

Setting Up a GRPC Connection

Before we start, please we need to stop any running container instances of the Inference Server. Run the following command from the nupic/ directory:

./nupic_inference.sh --stop

Now we want to restart the inference server, but with the GRPC protocol enabled:

./nupic_inference.sh start --expose

The --expose flag allows ports (including the default 8001 used for GRPC) to be exposed on all network interfaces of your server machine. This allows remote clients to connect over the internet. If you're using local clients, you do not need to run this flag since 8001 is available on localhost by default.

You can view the Inference Server container logs to verify that the port has been expose using docker logs nupic_inference_server:

I0520 01:03:56.338793 62 grpc_server.cc:2445] Started GRPCInferenceService at 0.0.0.0:8001
I0520 01:03:56.338964 62 http_server.cc:3555] Started HTTPService at 0.0.0.0:8000
I0520 01:03:56.380235 62 http_server.cc:185] Started Metrics Service at 0.0.0.0:8002

Next, we'll use the benchmarking example to demonstrate a GRPC connection. Navigate to the following directory and open benchmark.py in a text editor.

cd <your_nupic_dir>/nupic.examples/examples/benchmark

Looking for the URL and PROTOCOL constants, and edit them as such:

URL = "your_server_ip:8001"
PROTOCOL = "grpc"

The IP address should be that of your Inference Server. This might be the IP address of an on-premise machine or a virtual machine on a public cloud. If you're not sure, speak to your infrastructure administrator. You may also need the admin's help to ensure that your firewall (if any) allows communications between the server and remote client over port 8001.

Now try running benchmark.py

python benchmark.py

Once you start seeing inference progress bars, that's how we can tell GRPC is working!

Generating embeddings for nupic-sbert.base-v3 model for Sentence Similarity use case...

100%|██████████████████████████████████████████████████████| 1500/1500 [05:24<00:00,  4.63it/s]
100%|██████████████████████████████████████████████████████| 1500/1500 [05:24<00:00,  4.63it/s]

Sentence similarity (nupic-sbert.base-v3)
Total time: 648.3879029750824

Encrypting with SSL

Let's bring this to the next level by adding SSL encryption to our GRPC connection. Note that the sequence of the steps below is important, so please follow carefully!

For this, you'll need to access the machine hosting the Inference Server through a terminal.

Navigate to the following directory on the server machine:

cd <your_nupic_dir>/inference/certificates

By default, this directory should be empty. It's intended to contain your SSL key and certificate. Let's start by generating the SSL key.

openssl genrsa -out server.key 2048

Using this key, we'll now generate a certificate. Importantly, please fill in the placeholders below (YOUR_IP_ADDRESS) with your Inference Server's public IPV4 address. In other words, this should be the address that the remote client uses to reference the Inference Server.

openssl req -newkey rsa:2048 -nodes -keyout server.key -x509 -days 30 -out server.crt -subj "/C=US/ST=CA/L=RWC/O=Numenta/OU=Engineering/CN=YOUR_IP_ADDRESS" -addext "subjectAltName=IP:YOUR_IP_ADDRESS"

This produces a server.crt file, which you will need to manually transfer to the remote client. For this tutorial, we can place it in the benchmarking example folder (<your_nupic_dir>/nupic.examples/examples/benchmark),

Bringing our attention back to the Inference Server, we will have to restart the Inference Server container so that it can pick up the newly generated key and certificate. From your nupic/ directory:

cd <your_nupic_dir>
./nupic_inference stop
./nupic_inference start --expose

Now let's open benchmark.py on the client machine. We need to validate the certificate against the Inference Server in order to connect over SSL:

CERTIFICATES = {"root_certificates": "server.crt"}

This constant references the server.crt that you had earlier placed in the same directory as benchmark.py.

Now you if you re-run benchmark.py, you'll be using both GRPC and SSL, giving you a fast and secure connection to the Inference Server.

❗️

Couldn't get the SSL connection working?

You may want to check the following:

  1. Did you specify the correct IP address when generating server.crt? If your server has a dynamic IP address, you will have re-generate the certificate every time the IP changes.
  2. Did you restart the Inference Server after generating the SSL key and certificate?
  3. Did you add the --expose flag when starting the Inference Server?
  4. Did you transfer the latest version of server.crt to the client?
  5. Is the client connecting to the correct Inference Server IP and GRPC port? The default port is 8001.
  6. Did you update the CERTIFICATES variable in benchmark.py?

Updated 2 months ago