BLOG >
What Building a Governed AI Decision System Taught Me About Knowledge, Governance & Feedback Loops
By Brian Charlonis
BLOG >
I recently completed a substantial round of updates to an AI-enabled decision system I have been building and testing through real opportunity, portfolio, and career decisions.
The system began with a practical objective: improve how complex opportunities were evaluated, how relevant evidence was retrieved, and how decisions were translated into action.
Over time, however, the work became less about building a better prompt and more about designing a governed operating system.
The most important lessons have involved structure:
The system now includes several distinct capabilities.
Together, these components form a governed learning loop rather than a single all-purpose assistant.
It would have been simpler to place all available knowledge and instructions into one system and ask it to evaluate jobs, write resumes, maintain the portfolio, and review its own performance.
That approach would also have created several problems.
A system responsible for making a decision should not automatically be trusted to validate the quality of that decision.
A system responsible for writing a resume should not independently determine whether an opportunity is strategically appropriate.
A system that maintains portfolio narratives should not become the sole authority on the factual limits of professional experience.
A system that records results should not silently inherit the assumptions of the systems it is meant to evaluate.
Separating these responsibilities created useful checks and balances.
The Decision System can recommend a positioning strategy.
The Resume Writer can execute that strategy using the available evidence.
The Application Intelligence Registry can later review whether the opportunity was well selected, whether the strongest evidence was retrieved, whether the resume followed the strategy, and whether recurring problems are appearing across applications.
That separation allows the system to challenge itself.
One of the most important architectural decisions was separating knowledge from governance.
Knowledge defines what the system knows.
It includes documented professional experience, portfolio cases, outcomes, methods, positioning, professional learning, and factual boundaries.
Governance defines how the system should use and interpret that knowledge.
It determines:
This distinction matters because adding more information does not automatically improve system performance.
A system can retrieve the correct evidence and still interpret it incorrectly.
It can find a relevant project but misunderstand the context in which the work was performed.
It can recognize a missing operating responsibility and mistakenly conclude that the underlying capability is also missing.
It can treat different terminology as different experience.
It can interpret incomplete documentation as evidence that something never occurred.
Those are not knowledge-volume problems.
They are governance and comprehension problems.
A recent review exposed a useful example.
The system had documented professional evidence of SaaS and enterprise-platform strategy through work completed for Dun & Bradstreet.
That work included product portfolio rationalization, consolidation around a more consistent data foundation, Good/Better/Best product tiering, customer journeys, self-service upgrading, platform modernization, concepts and prototypes, estimation, delivery planning, and a later standardized help-system engagement.
The evidence supported a legitimate conclusion:
There was direct professional experience involving SaaS products and enterprise software platforms.
However, the same evidence did not automatically establish:
The problem appeared when the system risked translating:
no documented internal SaaS operating ownership
into:
weak or insufficient SaaS experience
Those are not the same conclusion.
The evidence was present. The interpretation was wrong.
The correction was not to add a stronger marketing claim or rewrite the professional history.
The correction was to improve the governance.
The system now distinguishes among four types of qualification gaps.
The underlying capability has not been demonstrated by the available evidence.
The underlying capability has been demonstrated, but not within the exact organizational, industry, scale, ownership, or operating context requested.
The available evidence is insufficient to determine whether the capability exists.
Uncertainty should not automatically become a negative conclusion.
Relevant experience exists, but the terminology used by the employer differs from the terminology used in the documented experience.
These distinctions matter because they should affect evaluation differently.
A context gap is not the same as an experience gap.
A translation gap should often be resolved through accurate interpretation.
A knowledge gap should remain uncertain until more evidence is available.
The broader lesson was clear:
A governed AI system needs more than access to facts. It needs disciplined rules for understanding what those facts do and do not establish.
The Application Intelligence Registry played an important role in surfacing this problem.
Its purpose is not to make application decisions or rewrite materials.
Its purpose is to preserve, normalize, retrieve, compare, and report what happened across applications.
That separation made the review more useful.
The Registry could observe that the underlying evidence existed while the Decision System had interpreted it too narrowly.
It could also identify a related retrieval problem in the Resume Writer.
The Resume Writer was generally capable of finding obvious portfolio evidence, but it was not consistently searching the full professional knowledge base before deciding what should support a resume.
As a result, strong employment-based evidence could be underused while newer portfolio or conceptual work received greater attention.
The correction was again behavioral rather than factual.
Before selecting portfolio or independent evidence, the Resume Writer should first search the complete professional knowledge base for direct and transferable employment experience.
Professional experience should generally receive greater weight when it can support the same requirement.
Portfolio and independent work should extend the professional story, demonstrate newer capabilities, or address requirements not already established through employment history.
The Registry did not prove that every earlier resume was poor.
It showed that the system possessed stronger evidence than it consistently used.
That is the kind of distinction an independent feedback mechanism is designed to surface.
Another unexpected result was that the system began influencing the portfolio itself.
As job descriptions were evaluated over time, the system accumulated recurring enterprise problems, capability requirements, and market signals.
Patterns began to emerge.
Organizations were not simply looking for “AI experience.”
They were trying to address problems involving:
Those recurring signals exposed gaps in how my AI thinking and experience were represented publicly.
The result was the development of four new AI cases based on problems repeatedly appearing across real job descriptions.
These were not created as keyword exercises.
They were created because the market was consistently describing enterprise problems that deserved a more complete response.
That produced another learning loop:
Market signals → recurring problems → portfolio gaps → new cases → expanded knowledge → better future decisions
The portfolio was no longer a static archive of past work.
It became an evolving knowledge system informed by current enterprise demand.
The relationship also worked in the other direction.
The new cases expanded the system’s ability to reason about AI governance, monitored autonomy, decision systems, enterprise adoption, operating models, and implementation discipline.
The portfolio provided structured evidence.
The knowledge documents preserved and normalized that evidence.
The Decision System could then use it to evaluate future opportunities and explain related capabilities.
This created a reciprocal relationship:
The system shaped the portfolio, and the portfolio strengthened the system.
That relationship is especially important because the portfolio and the Decision System serve different purposes.
The portfolio remains the primary evidence source.
It demonstrates the work, methods, artifacts, and outcomes.
The Decision System helps readers interpret that evidence, connect ideas across cases, explore related capabilities, and understand how the work reflects a broader enterprise approach.
The system is an extension of the portfolio, not a replacement for it. That principle is explicitly built into the Decision System’s knowledge and operating philosophy.
After several rounds of system design, the overall architecture is becoming more stable.
The current priority is knowledge quality.
That does not simply mean adding more text.
It means validating whether the knowledge is complete, accurate, appropriately bounded, and understandable by the systems using it.
Recent work has focused on documenting parts of professional experience that traditional case studies often omit.
Examples include:
These behaviors are important because they establish seniority and operating scope.
A polished portfolio case may explain the business problem, the strategy, and the resulting artifact.
It may not explain:
Those details may not all belong on a public case page.
They do belong in the internal knowledge base.
This has led to another useful distinction.
A public portfolio case should be selective.
It should communicate the enterprise problem, Brian’s contribution, the strategic approach, the evidence created, and the transferable capability demonstrated.
An internal professional evidence base should be much richer.
It may include:
The public story creates understanding.
The internal evidence protects accuracy and supports deeper reasoning.
A governed system needs both.
The next major challenge is knowledge comprehension.
Traditional retrieval asks:
Can the system find the relevant information?
Comprehension asks:
Does the system understand what the information means, how strong it is, how it relates to other evidence, and what conclusions it supports?
For example, a system may retrieve a statement that someone presented to executives.
To use that evidence correctly, it may also need to understand:
The phrase “presented to executives” is not enough by itself.
The surrounding context determines what capability the evidence establishes.
The same is true for mentoring, supervision, RFP development, estimation, product strategy, governance, delivery, and commercial impact.
A rich knowledge base should not only contain claims.
It should contain relationships, context, strength, boundaries, and provenance.
Separating governance from knowledge provides control over how the system evolves.
If new professional evidence is discovered, it can be added to the knowledge base without silently changing the system’s evaluation philosophy.
If a decision rule proves faulty, the governance can be corrected without rewriting the underlying facts.
If one specialized runtime is failing to retrieve evidence properly, its instructions can be improved without changing every other component.
If independent review identifies a recurring error, the system can determine whether the issue originated in:
That diagnostic clarity is one of the most valuable aspects of the architecture.
Without it, every problem looks like a prompt problem.
With it, system improvement becomes more precise.
The domain I used to build and test this system is career and opportunity decision-making.
The architectural problem is much broader.
Enterprise AI systems increasingly need to reason across large amounts of organizational knowledge while operating within clear professional, regulatory, and decision-making boundaries.
They must distinguish:
They also need oversight mechanisms that do not merely reproduce the assumptions of the system being reviewed.
That requires more than a powerful model.
It requires an operating architecture.
It requires knowledge governance.
It requires role clarity.
It requires traceable evidence.
It requires feedback loops.
And it requires the humility to let real use reveal where the system is wrong.
The operating system is not finished.
There are still likely errors, incomplete knowledge, weak classifications, missing context, and system behaviors I have not yet identified.
That is not evidence that the system has failed.
It is the reason the feedback loop exists.
The goal is not to build a system that never makes a mistake.
The goal is to build a system capable of surfacing mistakes, distinguishing their causes, preserving what remains valid, and improving without destabilizing the entire architecture.
That is where the work is now focused.
The initial challenge was building the structure.
The next challenge is strengthening the quality, completeness, and comprehension of the knowledge moving through it.
The system and related cases can be explored through several connected parts of the portfolio:
The portfolio provides the evidence.
The system connects it.
The feedback loop improves it.
And the knowledge architecture determines whether the system can understand what that evidence actually means.