Midwest::LlmAgent

Overview

Midwest::LlmAgent is a controller concern that powers the command palette's AI chat mode. It streams an OpenAI-compatible chat completion (Server-Sent Events) to the browser while keeping the provider API key, model, and system prompt server-side. The palette's Stimulus controller consumes the standard data: {choices:[{delta:{content}}]} / [DONE] stream, so the same client works for hosted providers and local endpoints alike.

It is backed by the optional RubyLLM gem, which provides the provider matrix (OpenAI, Anthropic, Gemini, Ollama, and any OpenAI-compatible endpoint such as cocore.dev), the streaming API, and durable conversation history via acts_as_chat.

Chat everywhere, tools on capable models. Streaming chat works against any OpenAI-compatible endpoint. Native tool/function calling (see the Agentic section) requires a tool-capable provider/model — on small local models the agent degrades gracefully to plain chat.

Setup

1. Install RubyLLM

# Gemfile
gem 'ruby_llm'

bundle install
bin/rails g ruby_llm:install   # Chat / Message / ToolCall / Model tables
bin/rails db:migrate

2. Scaffold the agent

bin/rails generate midwest:llm_agent

This creates config/initializers/midwest_llm_agent.rb (provider config) and app/controllers/assistant_controller.rb.

3. Configure a provider

# config/initializers/midwest_llm_agent.rb
RubyLLM.configure do |config|
  config.openai_api_key = ENV['OPENAI_API_KEY']
  # …or any OpenAI-compatible endpoint:
  # config.openai_api_key         = ENV['COCORE_API_KEY']
  # config.openai_api_base        = 'https://cocore.dev/api/v1'
  # config.openai_use_system_role = true
end

4. Wire the controller and route

class AssistantController < ApplicationController
  include Midwest::LlmAgent
  llm_agent model:      'gpt-4o-mini',
            provider:   :openai,
            system:     'You are a helpful assistant for this application.',
            chat_model: Chat,        # acts_as_chat model → durable history
            history:    :fresh       # or :resume
end

# config/routes.rb
concern :midwest_llm_agent, Midwest::Routing::LLM_AGENT
resources :assistant, only: [], concerns: :midwest_llm_agent

5. Enable chat on the palette

<%= midwest_command_palette(
  chat: { endpoint: midwest_llm_agent_assistant_index_path }
) %>

llm_agent options

Security

• Secrets never reach the browser. The component exposes no api_key option; the key lives only in the RubyLLM initializer.
• Model and system prompt are server-owned. The client sends only the running transcript ({ role, content }); any other fields are ignored.
• The stream is relayed via ActionController::Live, so no extra job/broadcast infrastructure is required.

Durable history

When chat_model: is set, conversations are persisted through RubyLLM's acts_as_chat models (history + tool-call audit out of the box). Ownership is resolved by midwest_llm_scope (defaults to current_user, falling back to the session). The default midwest_llm_conversation assumes the chat model belongs_to :user; override it for other schemas:

private
def midwest_llm_conversation(config)
  current_account.chats.where(topic: 'assistant').order(:created_at).last ||
    current_account.chats.create!(model: config[:model], topic: 'assistant')
end

Agentic tools

The model can drive the palette. There are two kinds of tools:

Server tools — pass RubyLLM::Tool classes to tools:. RubyLLM runs the execute → result → continue loop server-side and persists each call:

class SearchInvoices < RubyLLM::Tool
  description 'Find invoices matching a query'
  param :query, desc: 'Free-text search'
  def execute(query:) = Invoice.search(query).limit(5).as_json
end
llm_agent model: 'gpt-4o-mini', tools: [SearchInvoices]

Command tools — tools: :commands turns the palette's own commands into tools. When the palette is agentic it posts its command list with each message; the agent exposes them to the model, and when the model invokes one the server streams an inline action row plus a client directive that runs the command in the browser (navigate, or dispatch midwest-command-palette:action) — the same path as manually selecting the item.

llm_agent model: 'gpt-4o-mini', tools: :commands

Security: command tools are derived only from the commands the client posted (midwest_llm_tool_allowed?), execution reuses the palette's existing selection path (so the model can't reach anything a user couldn't click), and server tools run under whatever authorization your execute enforces.

Native tool calling requires a tool-capable model. Against a small local model (or an endpoint without function-calling) the agent still answers as plain chat.

Without a server (Mode B)

For local, keyless endpoints during development you can skip this feature entirely and point the palette straight at the provider:

<%= midwest_command_palette(
  chat: { url: 'http://localhost:11434/v1/chat/completions', model: 'qwen2.5' }
) %>

No API key is ever templated into the page, and no tools are available in this mode.