Local models

You can run model locally with tools like FastChat or LocalAI which provide a OpenAI-API compatible API.

Custom API URL

Maybe you are using a custom endpoint which is providing an OpenAI-API compatible API. For using a custom endpoint, you just need to provide the base url in the preferences. For example, if you are using FastChat, you need to put http://localhost:8000/v1 in Preferences > Local Model > API URL.

Custom Model

If you want to use a model of an OpenAI-API compatible API or from the OpenAI API which isn’t available in Bavarder (available: gpt-3.5-turbo, gpt-4, text-davinci-003), you can use the provider callled OpenAI Custom Model and set Model to the model you want to use. For example, if you want to use gpt-4-32k you just have to set Model to gpt-4-32k



Full documentation is available here but for an easy setup of fastchat-t5-3b-v1.0 you can follow this instructions.

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-name 'vicuna-7b-v1.1' --model-path lmsys/fastchat-t5-3b-v1.0
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Now, you can use the API server by following Custom API Url and Custom Model


Full documentation is available here but for an easy setup of the GPT4ALL-J model you can follow this instructions. You need to have git, docker and docker-compose installed.

# Clone LocalAI
git clone https://github.com/go-skynet/LocalAI

cd LocalAI

# Download gpt4all-j to models/
wget https://gpt4all.io/models/ggml-gpt4all-j.bin -O models/ggml-gpt4all-j

# Use a template from the examples
cp -rf prompt-templates/ggml-gpt4all-j.tmpl models/

# (optional) Edit the .env file to set things like context size and threads
# nano .env

# start with docker-compose
docker-compose up -d --pull always
# or you can build the images with:
# docker-compose up -d --build

Now the model ggml-gpt4all-j is accessible at the API Url localhost:8080, you can use it in Bavarder by following Custom API Url and Custom Model

Go Home File an issue