Skip to content

feat: add deployment-wide template allowlist for chats#23262

Merged
johnstcn merged 18 commits intomainfrom
cian/chat-template-allowlist
Mar 25, 2026
Merged

feat: add deployment-wide template allowlist for chats#23262
johnstcn merged 18 commits intomainfrom
cian/chat-template-allowlist

Conversation

@johnstcn
Copy link
Copy Markdown
Member

@johnstcn johnstcn commented Mar 18, 2026

  • Stores a deployment-wide agents template allowlist in site_configs (agents_template_allowlist)
  • Adds GET/PUT /api/experimental/chats/config/template-allowlist endpoints
  • Filters list_templates, read_template, and create_workspace chat tools by allowlist, if defined (empty=all allowed)
  • Add "Templates" admin settings tab in Agents UI (what it looks like)

🤖 This PR was created with the help of Coder Agents, and has been reviewed by my human. 🧑‍💻

@johnstcn johnstcn self-assigned this Mar 18, 2026
@johnstcn johnstcn changed the title feat(chatd): add deployment-wide template allowlist for chat tools feat: add deployment-wide template allowlist for chat tools Mar 18, 2026
Comment thread coderd/chatd/chattool/chattool.go Outdated
Comment thread coderd/x/chatd/chattool/createworkspace.go
Comment thread coderd/chatd/chattool/listtemplates_test.go Outdated
Comment thread coderd/chatd/chattool/readtemplate.go Outdated
Comment thread coderd/chatd/chatd.go Outdated
Comment thread coderd/database/dbauthz/dbauthz.go
Comment thread coderd/chats_test.go Outdated
@johnstcn johnstcn force-pushed the cian/chat-template-allowlist branch 5 times, most recently from c2a90b6 to 72dc92d Compare March 24, 2026 12:49
@johnstcn johnstcn changed the title feat: add deployment-wide template allowlist for chat tools feat: add deployment-wide template allowlist for chats Mar 24, 2026
@johnstcn johnstcn marked this pull request as ready for review March 24, 2026 15:03
Copilot AI review requested due to automatic review settings March 24, 2026 15:03
@coder-tasks
Copy link
Copy Markdown
Contributor

coder-tasks Bot commented Mar 24, 2026

Documentation Check

Updates Needed

  • docs/ai-coder/agents/platform-controls/index.md - The "Template scoping for agents" bullet under "Where we are headed" currently reads "We intend to let administrators restrict which templates are available in the agentic interface". This PR ships exactly that feature (the Templates admin tab + allowlist API), so the future-tense description should be moved up to the "What platform teams control today" section and updated to present tense, matching the pattern used for Providers, Models, and System Prompt.

    ⚠️ Still unaddressed — no documentation changes found in this PR (re-verified after latest commits)


Automated review via Coder Tasks

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a deployment-wide “template allowlist” for Agents chats, exposing it via experimental API endpoints and enforcing it in chat workspace-provisioning tools, with an accompanying Admin UI settings tab.

Changes:

  • Store and manage a deployment-wide agents_template_allowlist value in site_configs, with GET/PUT /api/experimental/chats/config/template-allowlist.
  • Enforce the allowlist in chat tools (list_templates, read_template, create_workspace) when configured.
  • Add an Agents UI “Templates” admin settings section with Storybook coverage.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
site/src/pages/AgentsPage/components/Sidebar/AgentsSidebar.tsx Adds “Templates” admin nav item in Agents settings sidebar.
site/src/pages/AgentsPage/AgentSettingsPageView.tsx Adds Templates allowlist settings UI (multi-select + save).
site/src/pages/AgentsPage/AgentSettingsPageView.stories.tsx Adds story to exercise allowlist UI interactions.
site/src/api/typesGenerated.ts Adds ChatTemplateAllowlist TS type.
site/src/api/queries/chats.ts Adds query + mutation wrappers for template allowlist config.
site/src/api/api.ts Adds experimental API client methods for allowlist endpoints.
codersdk/chats.go Adds SDK type + ExperimentalClient methods for allowlist endpoints.
coderd/x/chatd/chattool/*.go Adds allowlist plumbing and enforcement in template/workspace tools.
coderd/x/chatd/chatd.go Loads deployment allowlist and injects into chat tools.
coderd/exp_chats.go Implements GET/PUT allowlist HTTP handlers + parsing helper.
coderd/exp_chats_test.go Adds API tests for allowlist behavior + authz + validation.
coderd/database/queries/siteconfig.sql Adds sqlc queries for get/upsert allowlist in site_configs.
coderd/database/queries.sql.go Generated sqlc output for new queries.
coderd/database/querier.go Adds new methods to sqlc querier interface.
coderd/database/dbmock/dbmock.go Adds mock methods for new store calls.
coderd/database/dbmetrics/querymetrics.go Adds metrics wrappers for new queries.
coderd/database/dbauthz/dbauthz.go Adds RBAC checks for new store methods.
coderd/database/dbauthz/dbauthz_test.go Adds RBAC test coverage for new store methods.
coderd/coderd.go Wires new config endpoints into router.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread site/src/pages/AgentsPage/AgentSettingsPageView.tsx Outdated
Comment on lines +812 to +825
beforeEach: () => {
// Track saved allowlist state across mock calls so the
// refetch after save returns the updated value.
let savedIDs: string[] = [];

spyOn(API, "getTemplates").mockResolvedValue(manyTemplates);
spyOn(API.experimental, "getChatTemplateAllowlist").mockImplementation(
async () => ({ template_ids: savedIDs }),
);
spyOn(API.experimental, "updateChatTemplateAllowlist").mockImplementation(
async (req) => {
savedIDs = [...req.template_ids];
},
);
Copy link

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This story's beforeEach calls spyOn(API.getTemplates) and spyOn(API.experimental.getChatTemplateAllowlist/updateChatTemplateAllowlist), but those same methods are already spied in the meta-level beforeEach. In Vitest/Jest, spying on an already-spied method typically throws and will break the story. Instead, reuse the existing spies by setting API.getTemplates.mockResolvedValue(...) / API.experimental.getChatTemplateAllowlist.mockImplementation(...), or remove these spies from the meta-level setup and only define them in the stories that need them.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually fine in Storybook — spyOn from storybook/test restores spies between stories, so re-spying in a story-level beforeEach is the established pattern in this file (see DefaultAutostopCustomValue, UsageDateFilterRefetchOverlay, etc.). The meta-level spy sets a default, and the story-level spyOn overrides it for that story's specific needs. No double-spy issue in practice.

🤖 This response was generated by Coder Agents.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ what he said

Copy link
Copy Markdown
Member

@mafredri mafredri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Agent on behalf of mafredri

Clean feature: allowlist enforcement at all three tool boundaries is thorough, the isTemplateAllowed helper is well-factored, and the deliberate StartWorkspace exclusion with its comment is a good design call. The UUID canonicalization and dedup on PUT, the opaque "not found" error for blocked templates, and the Storybook stateful mock are all nice touches.

Note on architecture: several panel reviewers flagged the allowlist as "loaded once per session" (hours of staleness). I verified this is incorrect — runChat is called per message turn (each processChatrunChat cycle re-reads from DB), so allowlist changes propagate on the next message. The staleness window is one LLM turn, not the chat lifetime. This is consistent with all other config loading and is fine.

A couple things to look at: 2 P2s, 2 P3s, 7 nits across 11 inline comments.

Sixteen reviewers, one allowlist, two JSON parsers. One of them even logs errors. Skål!

Comment thread coderd/x/chatd/chatd.go
Comment thread coderd/exp_chats.go
Comment thread coderd/x/chatd/chattool/listtemplates_test.go Outdated
Comment thread coderd/exp_chats.go
Comment thread coderd/database/dbauthz/dbauthz.go
Comment thread coderd/x/chatd/chattool/listtemplates.go Outdated
Comment thread site/src/pages/AgentsPage/AgentSettingsPageView.tsx
Comment thread site/src/pages/AgentsPage/AgentSettingsPageView.tsx Outdated
Comment thread coderd/database/dbauthz/dbauthz.go
Comment thread coderd/database/queries/siteconfig.sql
Copy link
Copy Markdown
Member

@mafredri mafredri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Agent on behalf of mafredri

Thanks for addressing all 11 findings — the test restructuring, corruption handling, existence validation, and doc comments are all solid. The useMemo and type predicate removals are clean.

One problem: the new template existence validation in PUT broke 3 existing API tests. 1 P1 inline.

Ten findings resolved, one JSON parser reformed, and three tests quietly waiting for templates that don't exist. Hyvää työtä otherwise!

Comment thread coderd/exp_chats.go Outdated
Comment thread coderd/x/chatd/chatd.go
@johnstcn johnstcn force-pushed the cian/chat-template-allowlist branch 2 times, most recently from c06d454 to 6029863 Compare March 24, 2026 20:51
Copy link
Copy Markdown
Member

@mafredri mafredri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Agent on behalf of mafredri

All findings from rounds 1 and 2 are resolved. Tests use real templates, the chatd asymmetry comment is clear, and the new NonexistentTemplateRejected subtest covers the validation path.

Three rounds, sixteen reviewers, zero surviving findings. The allowlist is allowed to ship. Tack!

Copy link
Copy Markdown
Member

@mafredri mafredri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🏂🏻

Comment thread coderd/exp_chats.go Outdated
Copy link
Copy Markdown
Member Author

@johnstcn johnstcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To future human Cian: READ THIS WITH YOUR EYES ONCE MORE BEFORE CLICKING THE GREEN BUTTON

Future Cian did this.

@johnstcn johnstcn force-pushed the cian/chat-template-allowlist branch from ad06568 to 0b51ef7 Compare March 25, 2026 09:55
Comment thread coderd/exp_chats.go Outdated
Comment thread coderd/x/chatd/chatd.go Outdated
Comment thread coderd/x/chatd/chattool/listtemplates.go Outdated
@johnstcn johnstcn force-pushed the cian/chat-template-allowlist branch 3 times, most recently from 0050444 to 973b2a4 Compare March 25, 2026 12:48
@johnstcn johnstcn force-pushed the cian/chat-template-allowlist branch from 973b2a4 to c239761 Compare March 25, 2026 13:03
johnstcn added 17 commits March 25, 2026 14:16
Adds the ability for deployment admins to restrict which templates the
list_templates, read_template, and create_workspace chat tools expose.
The allowlist is stored in site_configs and configured via the Agents
settings UI. An empty allowlist means all templates are available.
…ools

address review comments

- Use map[uuid.UUID]bool for allowlist type (comment 1)
- Return generic 'template not found' for blocked templates (comments 2, 4)
- Consolidate tool tests to single DB (comment 3)
- Use AsChatd ctx instead of AsSystemRestricted (comment 5)
- Require ActionRead on ResourceDeploymentConfig for GET (comment 6)
- Consolidate API tests to single coderdtest instance (comment 7)
… chat tools

address self-review findings

- Log error and fail-open when allowlist DB read fails (P1)
- Add isDirty guard to prevent spurious saves on Enter (P1)
- Add error state with retry for failed data fetch (P1)
- Deduplicate template IDs in PUT handler (P2)
- Add role=alert to save error message (P2)
- Add aria-live to selection count (P2)
- Disable checkboxes during save (P3)
- Differentiate empty state messages (P3)
- Add CreateWorkspace allowlist test (P3)
- Document StartWorkspace intentionally not gated (P4)
…ist for chat tools

fixup! fixup! fixup! feat(chatd): add deployment-wide template allowlist for chat tools

Review round 2 fixes:
- chatd: log JSON parse errors and invalid UUIDs in allowlist (P2 correctness)
- PUT handler: canonicalize UUIDs to lowercase for case-insensitive dedup (P3 correctness)
- Sidebar: fix indentation corruption, remove stray 'replace' prop (P3 UX)
- Settings: always render aria-live region for screen readers (P2 a11y)
- Settings: add role=group + aria-label to checkbox list (P2 a11y)
- Settings: rename 'Clear' to 'Deselect All' for clarity (P2 UX)
… allowlist

Swap the custom checkbox list + SearchField for the design system's
MultiSelectCombobox component. This gives us consistent styling,
built-in search/filter, badge display for selections, keyboard
navigation, and proper focus management — all for free.

Removes ~50 lines of bespoke UI code.
…d allowlist

Review round 3 fixes:
- Add aria-label to MultiSelectCombobox input for screen readers (P1 a11y)
- Log at ERROR when all UUIDs in allowlist are invalid, producing a
  silent fail-open (P2 correctness)
- Fix indentation in chatd allowlist parsing block
The component initializes its internal 'selected' state from
defaultOptions on mount, but our data loads asynchronously after
mount. Pass both defaultOptions and value, and add a key derived
from the server selection so the component remounts when server
data arrives — ensuring badges render from the start.
Combines TemplatesSection and TemplatesManySelected into one
TemplateAllowlist story with a play function that exercises the
full admin workflow:
1. Starts empty — verifies 'no templates selected' and Save disabled
2. Selects one template, verifies badge + status, saves
3. Adds remaining seven, verifies '8 templates selected', saves

Uses stateful mocks so getChatTemplateAllowlist returns the
updated value after each save.
Addresses Copilot review comment: the custom queryKey ["templates"]
creates a separate cache entry from the shared helper in
api/queries/templates.ts. Switch to templates() for consistent
caching and invalidation.
- parseChatTemplateAllowlist now returns error instead of silently
  swallowing corrupt JSON; GET handler returns 500 for corrupt data
- PUT handler validates template IDs exist in the DB (400 on unknown)
- Add doc comment on GetChatTemplateAllowlist explaining stricter
  auth policy vs peer getters
- Add descriptive SQL comment on GetChatTemplateAllowlist query
- Use slices.Collect(maps.Keys(...)) instead of manual loop
- Restructure TestListTemplates into t.Run subtests grouped by tool,
  rename to TestTemplateAllowlistEnforcement
- Add happy-path CreateWorkspace test with allowed template
- Remove redundant useMemo wrappers (React Compiler handles it)
- Remove unnecessary type predicate (TS 5.5+ infers it)
The PUT handler now validates template existence via GetTemplateByID,
so tests using uuid.NewString() get 400 instead of succeeding.

- Switch to a single shared coderdtest instance with two real
  templates created via dbgen.Template
- Use real template IDs in AdminCanSet, AdminCanClear, DeduplicatesIDs
- Keep random UUIDs for NonAdminWriteFails and UnauthenticatedFails
  (they fail on authz before reaching validation)
- Add NonexistentTemplateRejected subtest for the new 400 path
- Add chatd comment noting API-vs-runtime fail-open asymmetry
- Add doc comments to generated querier.go and queries.sql.go to
  match the SQL comment added in the previous commit
- Add nolint:tparallel,paralleltest to test functions with
  sequential subtests sharing state
- Fix CreateWorkspace/Allowed test: don't assert resp.IsError since
  the tool does additional work (asOwner, workspace lookup) beyond
  the allowlist gate
…cess

Non-admin users shouldn't learn the endpoint exists. Return
ResourceNotFound (404) instead of Forbidden (403) on both GET
and PUT handlers, consistent with the 'don't leak information'
principle.
Three improvements to the template allowlist:

1. Chat tools now accept a function (func() map[uuid.UUID]bool)
   instead of a static map. The chatd runtime builds a closure
   that re-reads the allowlist from the DB on each tool call,
   so admin changes take effect without restarting the chat.
   A nil function fails open (all templates allowed).

2. PUT handler validates templates in a single GetTemplatesWithFilter
   call that also filters out deprecated templates, wrapped in
   a transaction with the upsert. Reports which specific IDs
   are missing or deprecated in the 400 response.

3. Added DeprecatedTemplateRejected test case.
@johnstcn johnstcn force-pushed the cian/chat-template-allowlist branch from c239761 to eba6413 Compare March 25, 2026 14:20
…tion message

CreateWorkspace now returns 'template not available for chat workspaces'
instead of 'not found' when the allowlist blocks a template. The test
was still checking for the old substring.
@johnstcn johnstcn merged commit 796872f into main Mar 25, 2026
66 of 71 checks passed
@johnstcn johnstcn deleted the cian/chat-template-allowlist branch March 25, 2026 15:19
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 25, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants