Are you having difficulty getting your GPT models to return the exact kind of outputs you're expecting? Have you already tried prompt engineering, yet the responses are still not quite what you want?

This is when you might want to try fine-tuning your GPT model. This involves performing a small amount of additional training on your GPT model, using your own data containing expected input and output pairs. With this, the GPT model will be more likely to give the expected outputs.

Quick Start

Before you start, make sure the NuPIC Inference Server and Training Module are up and running, and the Python environment is set up. To simplify things, we'll assume for this example that the Inference Server, Training Module, and Python clients are all installed on the same machine.

• Installation Guide
• Set up Python Environment
• Fine-tune using Training Module

For fine-tuning GPT models, we recommend a GPU with sufficient memory. As a general rule of thumb, you'll want a GPU with at least twice the memory requirements of your model.

Let's start by navigating to the directory containing the GPT fine-tuning example.

cd nupic.examples/examples/gpt_finetuning

In this example, suppose you want to have a GPT model that's really good at producing summaries of news articles. With this model, you want to simply provide it with some prose, and the model should return a summary, without any additional prompt engineering. Here we'll do so by fine-tuning the lightweight Gemma 2B model.

We have the following files and directories in this folder:

/gpt_finetuning/
├── datasets
│   ├── cc_news_test.csv   # For validating the model during training
│   └── cc_news_train.csv  # Training dataset containing input and output example pairs
└── finetune.sh            # Use this to start fine-tuning

Let's make finetune.sh executable, and start fine-tuning right away!

chmod +x finetune.sh
./finetune.sh

This sends a fine-tuning job to the Training Module. When fine-tuning is completed, the tuned model is saved to the indicated directory. You should see the following contents:

finetuned_models/
└── gemma-2b-it-tuned-BFB4A28B48C246DA837364D676D15C7C
    ├── 1
    │   ├── config.json
    │   ├── generation_config.json
    │   ├── model-00001-of-00003.safetensors
    │   ├── model-00002-of-00003.safetensors
    │   ├── model-00003-of-00003.safetensors
    │   ├── model.py
    │   ├── model.safetensors.index.json
    │   ├── special_tokens_map.json
    │   ├── tokenizer_config.json
    │   └── tokenizer.json
    └── config.pbtxt

To deploy this tuned model, copy the tuned model folder into the Inference Server.

cp -r finetuned_models/gemma-2b-it-tuned-BFB4A28B48C246DA837364D676D15C7C nupic/inference/models

We're almost ready to perform inference. Let's make some minor code changes to register our new model in the prompt formatter utilities. Open nupic/nupic.examples/src/nupic/client/utils.py in a text editor of your choice. We want to add the tuned model to model_naming_mapping and model_prompt_formatter_mapping. The tuned model will use the same prompt formatter as the original base model:

model_naming_mapping = {
    "llama2": "llama2.chat.7b",
    "llama2-streaming": "llama2.chat.7b.streaming",
    "llama3": "llama3.chat.8b",
    "llama3-streaming": "llama3.chat.8b.streaming",
    "zephyr": "zephyr.beta.7b",
    "zephyr-streaming": "zephyr.beta.7b.streaming",
    "gemma": "gemma.it.2b",
    "gemma-streaming": "gemma.it.2b.streaming",
    "gemma2": "gemma2.it.9b",
    "gemma2-streaming": "gemma2.it.9b.streaming",
    "nupic-corti": "nupic-gpt.7b-corti.v0",
    "nupic-corti-streaming": "nupic-gpt.7b-corti.streaming.v0",
    "nupic-dendi": "nupic-gpt.7b-dendi.v0",
    "gemma-tuned": "gemma-2b-it-tuned-BFB4A28B48C246DA837364D676D15C7C" <---------
}

model_prompt_formatter_mapping = {
        "llama2.chat.7b": Llama2PromptFormatter,
        "llama3.chat.8b": Llama3PromptFormatter,
        "zephyr.beta.7b": ZephyrPromptFormatter,
        "gemma.it.2b": GemmaPromptFormatter,
        "gemma2.it.9b": GemmaPromptFormatter,
        "nupic-gpt.7b-corti.v0": NupicGPTPromptFormatter,
        "nupic-gpt.7b-dendi.v0": NupicGPTPromptFormatter,
        "gemma-2b-it-tuned-BFB4A28B48C246DA837364D676D15C7C": GemmaPromptFormatter <-------
    }

Now let's navigate to and modify nupic/nupic.examples/examples/gpt/gpt.py to test out the summarisation capabilities of the tuned model. We'll pass an excerpt from the United States Declaration of Independence as a input. Note that we're passing just the excerpt, without any further instructions to the model.

message = (
        """We hold these truths to be self-evident, that all men are created equal, that 
  they are endowed by their Creator with certain unalienable Rights, that among these are 
  Life, Liberty and the pursuit of Happiness.--That to secure these rights, Governments are 
  instituted among Men, deriving their just powers from the consent of the governed, --That 
  whenever any Form of Government becomes destructive of these ends, it is the Right of the 
  People to alter or to abolish it, and to institute new Government, laying its foundation 
  on such principles and organizing its powers in such form, as to them shall seem most 
  \likely to effect their Safety and Happiness. Prudence, indeed, will dictate that 
Governments long established should not be changed for light and transient causes; and 
  accordingly all experience hath shewn, that mankind are more disposed to suffer, while 
  evils are sufferable, than to right themselves by abolishing the forms to which they are 
  accustomed. But when a long train of abuses and usurpations, pursuing invariably the same 
  Object evinces a design to reduce them under absolute Despotism, it is their right, it is 
  their duty, to throw off such Government, and to provide new Guards for their future 
  security.--Such has been the patient sufferance of these Colonies; and such is now the 
  necessity which constrains them to alter their former Systems of Government. The history 
  of the present King of Great Britain is a history of repeated injuries and usurpations, 
  all having in direct object the establishment of an absolute Tyranny over these States. 
  To prove this, let Facts be submitted to a candid world."""
    )

Let's run the inference! Remember to specify that you want to use the tuned model:

python gpt.py -m gemma-tuned

If all goes well, the model will know that you want a summary (because that's how we tuned it).

Expected output:

user
You are a helpful AI assistant.

We hold these truths to be self-evident, that all men are created equal, 
...
To prove this, let Facts be submitted to a candid world.

model
* All men are created equal.
* They are endowed by their Creator with certain unalienable Rights.
* These Rights include Life, Liberty and the pursuit of Happiness.
* Governments are instituted among Men, deriving their just powers from the consent of the governed.
* Whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it.

That's all to it! GPT fine-tuning isn't limited to just summarization tasks. It's generally useful if you have at least 1,000 examples of input-output pairs that you expect from the model.

Still curious? Let's dig a little deeper to see under the hood.

In More Detail

Let's start by examining the dataset. We can look at either cc_news_train.csv or cc_news_test.csv. The headers and first row should look like this:

The datasets in this example are news articles from Common Crawl, which is a repository of open web-scraped datasets.

For GPT fine-tuning, it is necessary to have the Label and Text headers to denote expected outputs and inputs, respectively. In the case of our examples, Text contains the original news article, and Label contains the summary that we'd like the model to produce.

Now let's look at how the actual fine-tuning is done. Opening finetune.sh, we see that the script is calling the regular nupic_train CLI, which in turn communicates with the inference server. The regular CLI arguments are available, but we'll highlight the ones that are important for GPT fine-tuning:

python -m nupic.client.nupic_train \
  --model google/gemma-2b-it \
  --seed $seed \
  --url http://localhost:8321 \
  --batch_size 1 \
  --epochs 5 \
  --task_type gpt_sft \
  --hf_access_token "your token here" \
  --train_path $train_dataset \
  --test_path $test_dataset \
  --use_lora \
  --input_prompt "$in_prompt" \
  --response_prompt "$rsp_prompt" \

• batch_size: use small batch sizes as GPT models already require large amounts of memory
• epochs: start with a small number of epochs, e.g., 1 to 5 epochs. If not using LORA (see below), fine-tuning for too many epochs can cause a model to forget what it was originally trained on!
• task_type: the gpt_sft string tells the Training Module that we want to fine-tune a GPT model
• use_lora: this argument indicates that instead of tuning the whole GPT model, which is computationally costly, we want to fine-tune just a small adaptor module that can modify the behavior of the model
• input_prompt and output_prompt: this prepends special tokens and/prompts to input and output examples in the dataset. This helps the model "recognise" that a given string is an input or output:

in_prompt="<start_of_turn>user\n Create a headline for the following news article:\n"
rsp_prompt="<end_of_turn>\n<start_of_turn>model\n"

Note the special tokens <start_of_turn> and <end_of_turn>. These are specific to each model family, and are typically required to help models recognise conversational turns. Please pay careful attention to the placement of these tokens for fine-tuning. For some models, the <end_of_turn> token (or its equivalent) occurs comes at the end of the user's turn/start of the model's turn, such as in the case above for Gemma. However, in other models (such as Llama 2), the same token is instead required only at the end of the model's response. For more information, please consult the official documentation for each model.