Danswer can also make requests to an arbitrary model server via REST requests. Optionally an access token can be passed in. To customize the request format and handling of the response, it may be necessary to update/rebuild the Danswer containers.
Extending Danswer to be compatible with your custom model server
The default implementation is compatible with the blog demo shown below.
Danswer with self hosted
Llama-2-13B-chat-GGML using a custom FastAPI Server.
- See the Medium blog post.
- This demo uses Google Colab to access a free GPU but this is not suitable for long term deployments
Setting Danswer to use the custom implementation: