Feedback Loops

The loop

The loop starts with a defined question and expected answer. Without that reference, teams tend to judge outputs by brand presence or tone rather than by factual usefulness.

1. Define what good looks like

A useful evaluation begins with an expected answer, not only a prompt.

For each priority question, define:

• The user intent
• The entity or entities involved
• The facts required for a useful answer
• The source or source class that should support each fact
• The required time context
• Any material qualifications
• What would count as an inaccurate or unsafe answer

This creates a testable reference. It does not require scripting the exact wording of an external answer. It establishes the minimum conditions for accuracy and usefulness.

2. Build a question set, not a keyword export

Keywords remain useful evidence of language and demand. A feedback loop needs a more deliberate unit: the question set.

A question set groups several forms of the same user need and several adjacent needs that may require different Knowledge Objects.

For a company earnings experience, the set may include:

• When is the next earnings date?
• What did the company report last quarter?
• Did earnings beat expectations?
• What did management say on the call?
• Why did the stock move after earnings?
• What are analysts expecting next quarter?

These questions share an entity, but they do not share the same time context, source authority, or object type.

A strong question set includes direct and conversational wording, current and historical variants, comparison questions, ambiguous phrasing, and questions that should not be answered from the same source.

3. Observe several dimensions

A single visibility score hides important differences.

A source can have high visibility and poor integrity. Another can have perfect accuracy but little distribution. A third can earn citations while offering little incremental value to the user.

Separate measures make those differences visible.

4. Use several observation methods

Manual answer review

Manual review is slow but useful during framework development. It reveals ambiguity, lost context, and differences that a binary mention tracker cannot capture.

Reviewers should record the question, date, surface, answer, cited sources, factual errors, missing qualifications, and likely Retrieval Pipeline stage.

Search and referral data

Traditional search data remains important. Impressions, clicks, landing pages, query groups, crawl patterns, and conversions show whether public discovery leads to useful behavior.

Referral data from AI products can add evidence, but it is incomplete. Some experiences do not send a click, identify a referrer, or expose the query that led to the visit.

Application and support data

Internal search logs, support questions, failed searches, copied links, and repeated user corrections reveal where the knowledge model is weak.

These signals can identify a problem before external visibility changes.

Integrity monitoring

Freshness breaches, source failures, entity join errors, and disagreement across presentations should feed the same loop as search observations.

Discovery quality cannot be separated from source quality.

5. Diagnose before changing

The most important step is diagnosis.

Teams should classify the problem before deciding on a remedy.

6. Improve the lowest reusable layer

A useful product principle is to fix the lowest shared layer that can prevent the problem from recurring.

If five pages display the wrong currency because the source object lacks a currency field, editing five pages is the wrong fix. Add the field to the Knowledge Object and update the presentations.

If an answer confuses historical and current values because both use the same label, the durable fix may be separate object types and explicit period fields, not another paragraph.

If a page is inaccessible, no amount of entity modeling will solve the immediate problem. Fix the access layer first.

7. Run controlled changes

Search and AI systems are dynamic. That makes casual before and after claims unreliable.

A page change may coincide with an index refresh, a product update, a competitor change, a news event, or different query interpretation. Teams should avoid attributing every observed movement to the most recent edit.

Useful controls include:

• Changing one major structural element at a time when practical
• Recording exact implementation and observation dates
• Using stable question sets
• Comparing several similar Knowledge Objects
• Separating search traffic impact from answer quality
• Repeating tests across several observation windows
• Documenting alternative explanations

The goal is not laboratory certainty. The goal is disciplined learning.

Two speeds of feedback

Not every loop should run at the same cadence.

Operational loop

Used for failures that can harm users now, such as stale market data, expired policy information, broken pages, or incorrect calculations.

This loop can trigger alerts, suppressions, or rapid corrections.

Discovery learning loop

Used to study how search and AI systems surface, interpret, and attribute knowledge.

It should run on a stable schedule, use a controlled question set, and avoid overreacting to a single response.

Combining the two creates noise. Operational incidents need speed. Discovery learning needs patience and comparison.

Connect discoverability to business outcomes

Visibility is not the final objective.

A financial education page may help a user understand a metric before opening an account. A support object may reduce repeated contacts. A product specification may improve qualified conversion. A policy object may prevent an ineligible application.

The feedback loop should connect discovery measures to downstream outcomes where evidence is available.

• Qualified visits to the correct object page
• Completion of the intended task
• Reduction in repeated support questions
• Increase in successful internal searches
• Fewer corrections or compliance escalations
• Growth in branded follow up searches
• Assisted signups or conversions

Attribution will remain imperfect. The purpose is to keep the system focused on user and business value rather than mentions alone.

A useful review meeting

A monthly review should not become a screenshot presentation. It should make decisions.

1. Review material Knowledge Integrity failures.
2. Review changes in discovery, accuracy, freshness, context, and attribution.
3. Select the highest value recurring failure.
4. Assign an owner and a durable intervention.
5. Define the next evaluation date.
6. Record what the team learned and what remains uncertain.

This creates a product rhythm instead of a publishing treadmill.

Anti-patterns