zerfoo

Zerfoo

Embed LLMs directly in Go applications. No Python. No sidecar. No HTTP calls to localhost.

model, _ := inference.Load("gemma-3-1b-q4")
reply, _ := model.Generate(ctx, "What is the capital of France?")
fmt.Println(reply) // Paris is the capital of France.

That's it. One binary, one import, one function call.

Install

go get github.com/zerfoo/zerfoo@latest

Gemma 3 1B Q4 is ~700 MB and runs on any laptop with 2 GB free RAM. No GPU required.

Examples

Chat bot in 10 lines

model, _ := inference.Load("gemma-3-1b-q4")
defer model.Close()

reply, _ := model.Chat(ctx, []inference.Message{
    {Role: "user", Content: "Write a haiku about Go"},
})
fmt.Println(reply.Content)

Streaming tokens to a terminal

model, _ := inference.Load("gemma-3-1b-q4")
defer model.Close()

model.GenerateStream(ctx, "Explain quantum computing", func(token string, done bool) error {
    fmt.Print(token)
    return nil
})

Summarize text inside a CLI tool

func summarize(text string) string {
    model, _ := inference.Load("gemma-3-1b-q4")
    defer model.Close()

    summary, _ := model.Generate(ctx,
        "Summarize this in one sentence:\n\n"+text,
        inference.WithMaxTokens(64),
        inference.WithTemperature(0.3),
    )
    return summary
}

Add an AI endpoint to an existing HTTP server

mux := http.NewServeMux()

// Your existing routes
mux.HandleFunc("GET /health", healthHandler)

// Add LLM-powered endpoint
model, _ := inference.Load("gemma-3-1b-q4")
mux.HandleFunc("POST /ask", func(w http.ResponseWriter, r *http.Request) {
    var req struct{ Question string }
    json.NewDecoder(r.Body).Decode(&req)

    answer, _ := model.Generate(r.Context(), req.Question,
        inference.WithMaxTokens(256),
    )
    json.NewEncoder(w).Encode(map[string]string{"answer": answer})
})

http.ListenAndServe(":8080", mux)

Drop-in OpenAI-compatible server

model, _ := inference.Load("gemma-3-1b-q4")
server := serve.NewServer(model)
http.ListenAndServe(":8080", server.Handler())

Works with any OpenAI client library — just point it at localhost:8080.

Classify text with structured output

model, _ := inference.Load("gemma-3-1b-q4")
defer model.Close()

prompt := `Classify this support ticket as "billing", "technical", or "general".
Reply with only the category name.

Ticket: I can't log in to my account after changing my password.

Category:`

category, _ := model.Generate(ctx, prompt,
    inference.WithMaxTokens(4),
    inference.WithTemperature(0),
)
fmt.Println(strings.TrimSpace(category)) // technical

Generate code

model, _ := inference.Load("gemma-3-2b-q4") // Larger model for code tasks
defer model.Close()

code, _ := model.Generate(ctx,
    "Write a Go function that reverses a string. Return only the code.",
    inference.WithMaxTokens(128),
    inference.WithTemperature(0.2),
)
fmt.Println(code)

Process a batch of inputs

model, _ := inference.Load("gemma-3-1b-q4")
defer model.Close()

questions := []string{
    "What is 2+2?",
    "Name the largest ocean.",
    "Who wrote Hamlet?",
}

for _, q := range questions {
    answer, _ := model.Generate(ctx, q, inference.WithMaxTokens(32))
    fmt.Printf("Q: %s\nA: %s\n\n", q, answer)
}

Recommended Models

All models run on CPU out of the box. Add inference.WithDevice("cuda") for GPU acceleration.

Supported Architectures

Gemma 3, LLaMA 3, Mistral, Qwen 2.5, DeepSeek, Phi-4 — in GGUF or ZMF format, with F32 or Q4_0 quantization.

CLI

go install github.com/zerfoo/zerfoo/cmd/zerfoo@latest

zerfoo run gemma-3-1b-q4             # Interactive chat
zerfoo serve gemma-3-1b-q4           # OpenAI-compatible API on :8080
zerfoo predict -model gemma-3-1b-q4  # Batch inference from CSV/JSON

Performance

How It Works

Zerfoo is a full ML framework written in Go — tensors, computation graphs, automatic differentiation, SIMD kernels, and CUDA support. The inference package wraps all of that into the simple API shown above.

Under the hood, inference.Load does:

1. Downloads the model (or loads from cache)
2. Memory-maps the weights (zero-copy, no heap allocation)
3. Builds a static computation graph with optimized fused kernels
4. Returns a *Model ready for generation

inference.Load("gemma-3-1b-q4")
    │
    ├── model/gguf    → Parse GGUF file, load Q4 weights
    ├── graph/        → Build computation DAG, fold transposes
    ├── compute/      → CPU engine with NEON/AVX2 SIMD, fused RMSNorm/RoPE
    └── generate/     → Autoregressive decode with PagedKV cache

Building with CUDA

# CPU only (default, no Cgo)
go build ./cmd/zerfoo

# With GPU support
CGO_CFLAGS='-I/usr/local/cuda/include' \
CGO_LDFLAGS='-L/usr/local/cuda/lib64' \
go build -tags cuda ./cmd/zerfoo

Project Structure

inference/       Load models and generate text (start here)
serve/           OpenAI-compatible HTTP server
compute/         Engine interface (34 ops), CPU and CUDA backends
graph/           Computation DAG with automatic differentiation
layers/          40+ layer types (attention, normalization, activations)
tensor/          N-dimensional arrays with Q4/Q8 quantized storage
generate/        Token sampling, speculative decoding, PagedKV cache
model/           ZMF and GGUF model format loaders
training/        SGD, Adam, AdamW optimizers and training loops
internal/cuda/   CUDA kernels (Q4 GEMM, Flash Attention)
internal/xblas/  NEON/AVX2 SIMD matrix multiply
distributed/     gRPC-based distributed training

Contributing

See docs/design.md for architecture decisions.

go test ./... -race -timeout 120s
golangci-lint run ./...

License

Apache 2.0