91 lines
No EOL
2 KiB
Markdown
91 lines
No EOL
2 KiB
Markdown
# pkg/llm
|
|
|
|
> Multi-provider abstraction for language models.
|
|
|
|
## Responsibility
|
|
|
|
Define a common interface (`LLMClient`) and adapters for the main providers.
|
|
|
|
## Public API
|
|
|
|
```go
|
|
type LLMClient interface {
|
|
Generate(ctx context.Context, req CompletionRequest) (CompletionResponse, error)
|
|
Stream(ctx context.Context, req CompletionRequest) iter.Seq2[StreamChunk, error]
|
|
Name() string
|
|
Capabilities() ProviderCapabilities
|
|
}
|
|
|
|
type CompletionRequest struct {
|
|
Messages []Message
|
|
Tools []tools.Tool
|
|
ToolChoice ToolChoice
|
|
Model string
|
|
Temperature *float32
|
|
MaxTokens *int
|
|
}
|
|
|
|
type CompletionResponse struct {
|
|
Content string
|
|
ToolCalls []tools.Call
|
|
Usage TokenUsage
|
|
StopReason string
|
|
}
|
|
|
|
type ProviderCapabilities struct {
|
|
SupportsTools bool
|
|
SupportsVision bool
|
|
MaxContextWindow int
|
|
}
|
|
```
|
|
|
|
## Included providers
|
|
|
|
| Provider | Package | Tool support |
|
|
|---|---|---|
|
|
| OpenAI | `providers/openai` | ✅ |
|
|
| Anthropic | `providers/anthropic` | ✅ |
|
|
| Ollama | `providers/ollama` | ✅ (models that support it) |
|
|
| llama.cpp | `providers/llamacpp` | ✅ (with grammar) |
|
|
|
|
## Usage
|
|
|
|
```go
|
|
import "github.com/VictorVargas/rony-llm-agent/pkg/llm/providers/anthropic"
|
|
|
|
client, err := anthropic.New(anthropic.Config{
|
|
APIKey: os.Getenv("ANTHROPIC_API_KEY"),
|
|
Model: "claude-sonnet-4.5",
|
|
})
|
|
|
|
resp, err := client.Generate(ctx, llm.CompletionRequest{
|
|
Messages: []llm.Message{
|
|
{Role: llm.RoleUser, Content: "Hello"},
|
|
},
|
|
})
|
|
```
|
|
|
|
## Streaming
|
|
|
|
```go
|
|
for chunk, err := range client.Stream(ctx, req) {
|
|
if err != nil { return err }
|
|
fmt.Print(chunk.Delta)
|
|
}
|
|
```
|
|
|
|
## Mock for tests
|
|
|
|
```go
|
|
import "github.com/VictorVargas/rony-llm-agent/pkg/llm/mock"
|
|
|
|
mockClient := mock.New(mock.Responses{
|
|
{Match: "hello", Response: "Hi! How are you?"},
|
|
{Match: "*", Response: "default"},
|
|
})
|
|
```
|
|
|
|
## See also
|
|
|
|
- [pkg/agent](../agent/README.md) — Uses `LLMClient`
|
|
- [pkg/tools](../tools/README.md) — The `Tool` definitions |