Skip to main content

Llama.cpp

llama.cpp python library is a simple Python bindings for @ggerganov llama.cpp.

This package provides:

  • Low-level access to C API via ctypes interface.
  • High-level Python API for text completion
    • OpenAI-like API
    • LangChain compatibility
    • LlamaIndex compatibility
  • OpenAI compatible web server
    • Local Copilot replacement
    • Function Calling support
    • Vision API support
    • Multiple Models

Installation and Setupโ€‹

  • Install the Python package
    pip install llama-cpp-python
  • Download one of the supported models and convert them to the llama.cpp format per the instructions

Chat modelsโ€‹

See a usage example.

from langchain_community.chat_models import ChatLlamaCpp
API Reference:ChatLlamaCpp

LLMsโ€‹

See a usage example.

from langchain_community.llms import LlamaCpp
API Reference:LlamaCpp

Embedding modelsโ€‹

See a usage example.

from langchain_community.embeddings import LlamaCppEmbeddings
API Reference:LlamaCppEmbeddings

Was this page helpful?