Local models

You can run model locally with tools like FastChat or LocalAI which provide a OpenAI-API compatible API.

Custom API URL

Maybe you are using a custom endpoint which is providing an OpenAI-API compatible API. For using a custom endpoint, you just need to provide the base url in the preferences. For example, if you are using FastChat, you need to put http://localhost:8000/v1 in Preferences > Local Model > API URL.

Custom Model

If you want to use a model of an OpenAI-API compatible API or from the OpenAI API which isn’t available in Bavarder (available: gpt-3.5-turbo, gpt-4, text-davinci-003), you can use the provider callled OpenAI Custom Model and set Model to the model you want to use. For example, if you want to use gpt-4-32k you just have to set Model to gpt-4-32k

Examples

FastChat

Full documentation is available here but for an easy setup of fastchat-t5-3b-v1.0 you can follow this instructions.

First, install FastChat with pip by running pip install fschat
After, you can first launch the controller

python3 -m fastchat.serve.controller

Then, launch the model worker(s), which will automatically download the weights from a HuggingFace repository.

python3 -m fastchat.serve.model_worker --model-name 'vicuna-7b-v1.1' --model-path lmsys/fastchat-t5-3b-v1.0

Finally, launch the RESTful API server

python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Now, you can use the API server by following Custom API Url and Custom Model

LocalAI

Full documentation is available here but for an easy setup of the GPT4ALL-J model you can follow this instructions. You need to have git, docker and docker-compose installed.

# Clone LocalAI
git clone https://github.com/go-skynet/LocalAI

cd LocalAI

# Download gpt4all-j to models/
wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O models/ggml-gpt4all-j

# Use a template from the examples
cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/

# (optional) Edit the .env file to set things like context size and threads
# nano .env

# start with docker-compose
docker-compose up -d --pull always
# or you can build the images with:
# docker-compose up -d --build

Now the model ggml-gpt4all-j is accessible at the API Url localhost:8080, you can use it in Bavarder by following Custom API Url and Custom Model

Go Home File an issue