AI Agents Knowledge Store
Helping clients teach Concert's AI with their own knowledge, and test it before it goes live.
Concert's AI only knew general information. Clients, especially in compliance, needed answers based on their own policies and rules.
I mapped the full life of a document, then designed the upload wizard, response testing, library, and detail panel. I worked closely with product, engineering, research, and writing through launch.
Shipped in IBM Concert. Clients can ground AI answers in their own knowledge, test a document before it goes live, and stay in control after.
Clients couldn't trust AI answers that ignored their own rules
IBM Concert is an IT automation platform that helps teams manage application risk, compliance, resilience, and cost. Out of the box, its AI agent only knew general information.
"Are my applications compliant?"
A compliance team asking this needs an answer grounded in the regulations and internal policies they actually follow, not generic advice. The obvious fix, "let users upload files," hid three harder questions:
Blind uploads
Users had no way to know whether a document would make answers better, worse, or no different.
No control after upload
Once the AI learned from a document, there was no clear way to stop it. Outdated policies would keep shaping answers.
Messy libraries
As documents piled up, admins needed to find, sort, and audit them quickly.
Make upload a test, not a leap of faith
The core idea: let users see the AI's answer change before they commit a document. Testing sits between upload and go-live, so no document reaches the AI until the user has judged its effect. After that, they can manage it anytime.
Users test a document before it goes live, then manage it anytime.
Four questions users had, four parts of the experience
I started by mapping the full life of a document, from the moment a user picks a file to the day it's retired. That map broke the experience into four parts, each answering a question users had.
The upload wizard
Testing responses before committing
The library and its actions
Details, history, and the full document
From first upload to retiring a document
1A two-step upload wizard
Uploading opens a side-by-side wizard with a progress tracker: Upload and add details, then Test response results.
- File limits (32 MB; .pdf, .json, .txt) are stated up front, so errors are rare
- The document name fills in from the file name and stays editable
- Subject (such as Compliance, Resilience, or Cost) and labels (such as Procedure, Documentation, or Framework) organize the library and give the AI context
- Next stays disabled until a file and name are added
Empty: file limits are shown up front
Filled in: name, subject, and labels added
2Test before you commitThe heart of the design
The AI generates three sample questions related to the document, such as "Are my applications compliant with industry regulations, and how can I improve?" For each one, users see two answers side by side: the AI's current answer, and its answer using the new document.
- Users pick the response they prefer, and a clear selected state confirms the choice
- Regenerate responses gives a fresh pair if neither looks right
- Users step through questions 1 to 3, and Finish upload unlocks only after all three
- Every choice also teaches the AI how to respond, so testing improves quality instead of just checking it
Sample question 1 of 3
Sample question 2 of 3
Sample question 3 of 3, then Finish upload
3A library built for control
After upload, documents land in the AI knowledge store library with a success notification.
- A table shows name, subject, labels, download, and status at a glance
- Search, filter, column settings, and refresh help admins handle large libraries
- Status (Active or Disabled) lets admins pause a document so the AI stops using it, without losing it
- Archive moves documents out of the way while keeping them recoverable
- Bulk actions let admins delete, archive, or download many documents at once
4Details, history, and the full document
Selecting a document opens a side panel with its description, size, uploader, dates, an Active toggle, and a history log of every change. View full document opens a viewer to read the content without downloading it.
The history log matters most for compliance: admins can show who added or changed a document, and when.
The end-to-end flow, in light and dark mode
These are the final high-fidelity designs I handed off to the frontend and backend developers. Each screen includes its key states, such as empty, selected, disabled, and success, so the team could build with confidence. I stayed involved after handoff to review builds and refine details until launch.
Upload a document
8 screens · scroll sideways →
Manage the library
6 screens · scroll sideways →
I led the design end to end, and we shipped it together
As the UX design lead, I owned the experience from the first concept to launch. Here's what I did, and how I worked with the team to get it live.
Mapped the experience
Defined the user flows for uploading, testing, and managing documents, based on the full life of a document.
Designed every screen
Designed the library, upload wizard, response testing step, detail panel, and document viewer, with every key state.
Created a new AI pattern
Applied and extended IBM Carbon, including a new side-by-side pattern for comparing AI responses, so it felt native to Concert.
Shaped scope with product
Partnered with the AI product manager on scope and on how users' choices feed back into the AI.
Designed within real limits
Worked with backend and frontend developers on what the system could support, and designed around file types, size limits, and how responses are generated.
Tested and refined
Worked with UX research to test with users and turn findings into design changes, and with UX writing to keep every step in plain language.
I shared flows early, ran regular design reviews, and used early prototypes so everyone saw the same thing. When a constraint changed, such as what the backend could generate, we adjusted the design together instead of handing work back and forth. I stayed involved after handoff, through design QA and iteration, until launch.
What made this hard
Making AI behavior visible
How the AI uses a document is invisible. We had to show its effect in a way non-technical admins could judge in seconds.
Trust in a high-stakes domain
Compliance answers can affect audits. Every action needed to be clear and, where possible, reversible.
Technical constraints
File types, size limits, and how responses are generated were set by the backend. The design had to work within them and explain them plainly.
A new AI pattern
Carbon had no ready-made pattern for comparing AI responses, so we designed one that still felt consistent with the product.
A generic AI assistant became one that knows each client's rules
The AI Knowledge Store shipped in IBM Concert.
Grounded answers
Clients bring in their own regulations, procedures, and frameworks, so AI answers reflect the rules they actually follow.
Confidence before commitment
Comparing responses shows a document's effect before it goes live, which lowers the risk of making answers worse.
Ongoing control
Pause, archive, and delete let admins retire outdated knowledge without losing it.
Audit readiness
Document history records who changed what, and when.
A reusable AI pattern
The side-by-side comparison gives the Concert team a model for other features where people need to judge AI output.
Show AI behavior, don't describe it
The best way to build trust in AI is to show what it does. Letting users compare answers side by side made an invisible process concrete, and gave them a real say in how the AI responds.