How to Audit AI Answers: Separating Identity from Evidence

An AI answer can cite a real page, identify the intended person and still reach a conclusion that the page does not support. Identity matching and claim support need separate checks. Otherwise, fixing a name collision can leave a confident but weakly evidenced answer looking resolved.

This implementation case documents one completed, AI-assisted research project with two phases: a Secondary Query Stress Test and a controlled disambiguation retest. The subject was Eugene Prudchenko, who also directed the work and publishes this account. That conflict is explicit. This is first-party research evidence, not a client success story, an endorsement or an independent assessment of the subject.

The public evidence package contains aggregate counts, three selected sample records, a claim-support manifest, an original runner excerpt and a verifier. It exposes enough material to inspect the examples and reproduce the published arithmetic. It does not disclose the full private archive or permit a complete independent reclassification of every answer.

Correction of baseline Q045 — 20 September 2026

The initial interpretation of baseline Sonar Q045 insufficiently accounted for qualifications in the full answer. Sonar assessed the clarity and consistency of publicly stated principles and explicitly left the team’s practical experience uncertain. The stronger allegation that this answer turned declarations into established team practice is not confirmed. Q045 is no longer counted as an ECI issue on that basis; this does not establish that the answer is flawless. In the same selected 20-answer baseline, ECI and inference-only counts change from 13/20 to 12/20. Retest counts remain 16/20 and 15/20, and 11 of 13 retest TARGET answers remain flagged. This is a bounded AI-assisted correction, not independent expert validation or a claim of personal verification by the author. Original records, initial classifications and the v1 publication are preserved.

See the correction and original/revised totals. The original v1 package remains available as the historical publication.

The practical problem

An operator reviewing an answer needs to ask two different questions: “Is this the person we intended?” and “What exactly does the cited material establish?” A correct answer to the first question cannot substitute for the second.

For this case, an evidence-to-claim inference problem, abbreviated ECI, means that an answer moves beyond its support: for example, treating a first-person professional narrative as demonstrated execution, treating a stated principle as observed conduct, or attaching a materially mismatched citation to a claim. ECI is an analytical label used in this study. It is not a verdict that every sentence in a flagged answer is false.

The distinction becomes visible when the question changes but the initial answer remains in context. A model can inherit an identity error from that initial answer. It can also preserve the right identity while expanding a modest source into a stronger conclusion.

What was implemented

The baseline retained one PRIMARY answer for each of two configurations. Each of the 100 Russian follow-up questions was then sent in its own branch containing only the original PRIMARY question, that configuration's PRIMARY answer and the exact follow-up. Other follow-up answers were not carried into the branch.

The configurations were OpenRouter openai/gpt-5.6-sol with web tools disabled, and Perplexity sonar with web search enabled and low search context. Each completed one PRIMARY and 100 follow-ups: 2 PRIMARY + 200 follow-up answers. These were API observations, not observations of consumer chat interfaces.

The retest used Sonar again, one newly generated PRIMARY and 20 unchanged follow-ups selected from the earlier 100. The PRIMARY question added Bangkok and Burakorn Partners as disambiguating context. Selection emphasized problems already observed in the baseline. The retest completed 1 new PRIMARY + 20 follow-up answers. The numerical before/after comparison below uses only those same 20 Sonar follow-ups; it does not use all 200 baseline follow-ups.

The implementation preserved exact requests, raw responses, readable answer copies, returned citations, search results where supplied, source snapshots, configuration manifests and validation receipts. Analysis was kept separately from the original responses. A later clarification of classification boundaries was recorded rather than silently replacing the initial rule.

The preserved runner excerpt shows the branch construction and checks that the saved request history, returned answer, citations and usage match the originals. The public verifier was written for this publication; it is separate from that original implementation.

Branch contexts were isolated, but branches shared a PRIMARY within each configuration. They are not statistically independent trials. The two baseline configurations also differed in model, search availability, generated PRIMARY and observation conditions. This design cannot estimate the effect of web search.

Who did what

Eugene set the research objective, selected the direction and scope, required preservation of exact questions and evidence, and authorized this bounded publication. The saved instructions document those decisions. They do not establish that he manually wrote the runner or personally verified every answer.

Codex performed the implementation and API orchestration, gathered source material, produced preliminary classifications and assisted with interpretation and editorial preparation. The classifications are assistant-generated analysis, not an independent human expert assessment. A separate Codex reviewer checked this publication package as internal QA; that is not peer review or a second independent research annotator.

A distinct ChatGPT contribution to the questions or protocol cannot be established from the preserved run artifacts. No separate authorship credit for those steps is asserted here. Eugene is the author and publisher of this account, with the AI contribution disclosed.

This work is related to the Semantic Authority Network and its operating standard, the Semantic Authority Methodology. It is not a new version, certification or claim of full methodology compliance: the required human adjudication has not been established for these research labels.

What the selected comparison showed

The unit counted is a follow-up answer, not an individual claim or citation. Each column below has a denominator of 20; PRIMARY answers are excluded. The labels are preliminary classifications under the recorded rubric, with the single baseline Q045 correction disclosed above. Other labels have been retained, not revalidated.

The 10→2 result does not mean eight answers were fully fixed. Four retest answers classified UNRESOLVED contained unverified profile merges. Counting those as identity risk gives 10→6. The narrower confirmed count is two, not seven. The MIXED/UNRESOLVED boundary was clarified after generation: an unverified same-name profile is not enough to establish that another person was substituted. This was not a preregistered success criterion. Both counts are disclosed in the summary.

The retest identity distribution was 13 TARGET, 1 MIXED, 1 WRONG_ENTITY and 5 UNRESOLVED. TARGET means that the answer was classified as referring to the intended person; it does not validate its claims. 11 of those 13 TARGET answers had an ECI issue.

The corrected baseline ECI and inference-only totals are both 12/20, formerly 13/20. Baseline TARGET answers with ECI change from 10/10 to 9/10; the answer remains in the denominator and its identity classification is unchanged.

The ECI count of 16 includes one material citation-to-claim mismatch without a separately flagged inference leap. An inference-only convention excluding that one case gives 15/20. The publication keeps the broader 16/20 definition visible instead of switching definitions between the table and the interpretation.

The defensible observation is limited: clearer identity in this selected retest often coexisted with inadequate claim support. It does not show that the qualifier caused either the identity change or the increase in ECI. A new PRIMARY answer changed the whole shared context; there was no fresh paired unqualified control, repeated randomized measurement or independent second annotator. Retrieval variation, timing and model nondeterminism were not held fixed. This error-enriched subset cannot estimate general error rates or universal SAN effectiveness.

Three inspectable examples

The examples were chosen for safe, checkable source comparisons. They are not a representative sample. Original Russian answer excerpts, marked omissions, separate English translations, cited links and source excerpts are in the linked records. The records are sanitized excerpts, not complete raw responses.

Q009: narrowing a claim can be the useful result

The baseline opened with an affirmative claim about projects operating without constant founder involvement, then admitted that operational detail was missing. The retest more carefully attributed the claim to the subject's own descriptions and intended design.

Original Russian, exact opening excerpt from the retest; the remainder of the response is omitted:

Да — по его собственным описаниям, у Евгения Прудченко есть проекты и системы, которые он явно проектирует так, чтобы они работали не только вручную и не только при его постоянном присутствии.[1][2][3]

English editorial translation: “Yes — according to his own descriptions, Eugene Prudchenko has projects and systems that he explicitly designs to function through more than manual work and without his constant presence.” Citation numbers refer to the original answer; their links are mapped in the sample.

The Q009 record includes the baseline caveat as well as the retest caveat and source excerpts. The preliminary rubric changed from TARGET with an ECI issue to TARGET without a confirmed ECI issue. Actual operation without constant founder involvement remains Unknown. This is a narrower statement, not proof that autonomous operation was achieved.

Q045: full context limits the interpretation

The initial interpretation insufficiently accounted for the full answer’s qualifications. The baseline response recounts stated working principles and evaluates the clarity and consistency of their public formulations. Its opening uses “appear”, “fairly” and “broadly”; its middle explicitly limits the consistency assessment to publicly visible formulations. Own texts can support that narrower interpretation.

Original Russian, previously omitted middle paragraph; surrounding text is available in the corrected sample:

Если оценивать последовательность, то противоречий в публично видимых формулировках немного: его подход к управлению выстроен вокруг одной логики — структурировать работу, зафиксировать доказательства, убрать хаос и сделать процесс переносимым.[1][4] В этом смысле требования выглядят скорее системными, чем ситуативными.[4]

English editorial translation: “When assessing consistency, there are few contradictions in the publicly visible formulations: his approach to management follows one logic — structure the work, record the evidence, remove disorder and make the process transferable.[1][4] In this sense, the requirements appear more systematic than situational.[4]”

The final caveat explicitly says that the sources do not allow a confident judgment about how the team experiences this in practice. Those qualifications meaningfully limit the conclusion. The prior stronger allegation that Sonar turned declarations into established team practice is not confirmed, and this baseline record is no longer counted as an ECI issue on that basis.

The phrase about what the principles “usually” mean for a team remains a generic inference; removing the previous ECI basis does not certify the entire answer as flawless. Actual employee understanding and consistent application in practice remain unknown. This is a boundary case about checking full context, not evidence for the withdrawn allegation.

The corrected Q045 record includes the opening, both previously omitted middle paragraphs, the final caveat, separate translations and relevant source passages. The original v1 record is preserved. Retest Q045 is a separate answer and has not been reassessed.

Q053: the right identity can still carry an unsupported inference

The retest stayed with the intended person but interpreted consistent positioning across owned and connected pages as evidence of a unified operating strategy.

Original Russian, exact opening excerpt; the remainder of the response is omitted:

По публичным материалам это скорее единая стратегия, а не распыление: у Прудченко повторяется один и тот же каркас — AI / продукты / дистрибуция / рыночная инфраструктура / cross-border execution — просто в разных формах и на разных уровнях.[1][2][3][5][6]

English editorial translation: “From the public materials, this is more a unified strategy than a scattering of effort: Prudchenko repeats the same framework — AI / products / distribution / market infrastructure / cross-border execution — in different forms and at different levels.”

The Q053 record compares that statement with the source passages and original citation map. Repeated descriptions support an observation about public narrative. They do not measure allocation of attention, resources or execution. The recorded result remains TARGET with an ECI issue. The baseline's mixed biographical material is withheld to avoid spreading false associations.

A procedure an operator can reuse

1. Preserve the exact question, answer, configuration and date before editing anything. Keep the model's wording separate from your interpretation.
2. Resolve identity first. Compare the name with distinguishing context, organizations and the actual cited page. Keep an unresolved profile unresolved; a matching name is insufficient.
3. Break the answer into claims. For each material claim, record the cited URL and the exact passage that is supposed to support it. A returned search result is not proof that the answer used or read it.
4. Identify source control. A self-authored page remains first-party evidence when retrieved through an external API. A second domain or a checksum does not create independence.
5. Check the inference. Separate a stated role from demonstrated work, an intention from observed conduct, and a described implementation from an established outcome. Record “Unknown” when the available material cannot settle the claim.
6. Preserve both the original and the assessment. Version changes to the rubric, disclose sensitive boundary decisions and report the denominator. If you retest, state exactly what changed and what remained uncontrolled.

The existing methodology distinguishes Self-asserted, First-party evidenced, Independently corroborated and Unknown. This package uses those support states for claims about this case. Inspectable artifacts can support a narrow implementation claim. They do not turn the publisher's research into independent professional recognition.

What this case establishes

The records document a completed AI-assisted implementation: isolated follow-up contexts, preserved outputs and sources, separate identity and evidence checks, and a disclosed revision to an uncertain classification boundary. Public readers can compare the selected model excerpts with source excerpts and recalculate the reported totals from the aggregate tables.

The case does not establish client revenue, ROI, repeated commercial success, broad engineering mastery, general personal qualities, independent recognition or a causal mechanism behind model behavior. It claims no scientific priority. No model-effect measurement was run for this publication. Public availability, crawlable HTML and schema do not establish indexing, discovery, model use or improved answers.

Evidence, provenance and verification

The version 2 evidence index links every public file. Start with the methods note, then inspect the aggregate CSV, JSON summary and claim-support manifest.

The frozen source runs are followup-ru-v1-20260915T175235Z-dc68f0 and disambiguation-ru-v1-20260916T153455Z-58bb8d. The original archives remain private, including sensitive third-party material, full responses and reserved evaluation material. Only the listed sanitized derivatives are published. Source hashes were checked before and after export.

The checksums and dependency-free verifier check public-copy integrity and arithmetic. They do not prove truth, independence or when an original record was created. Selective disclosure also means readers cannot independently reproduce every private classification. A future model citing this article would be exposed to first-party evidence; that would not itself be independent corroboration or replication.

Русское резюме

Поправка к baseline Q045: первоначальная интерпретация недостаточно учитывала оговорки полного ответа. Прежняя сильная претензия о переходе от деклараций к установленной практике не подтверждена. Поэтому baseline ECI и inference-only пересчитаны с 13/20 до 12/20; retest и его Q045 не пересматривались. Исходная v1 сохранена, а неизменность остальных чисел не подтверждает правильность их классификаций. Это AI-assisted исправление, не независимая экспертная или личная проверка автором.

Правильная идентификация человека ещё не означает, что выводы ответа подтверждены. В одном AI-assisted исследовательском проекте сохранены два PRIMARY и 200 follow-up ответов baseline, затем новый PRIMARY Sonar и 20 прежних вопросов. Двадцать вопросов выбраны с акцентом на уже замеченные проблемы, поэтому это не оценка частоты ошибок во всех ответах моделей.

В выбранных двадцати Sonar-ветках подтверждённая подмена личности снизилась с 10/20 до 2/20. Но четыре дополнительных ответа retest содержат непроверенные объединения профилей: при более широком подсчёте получается 10/20 → 6/20. Граница MIXED/UNRESOLVED уточнена после генерации, а не заранее. Нельзя считать восемь ответов полностью исправленными.

ECI — выход вывода за пределы поддержки или существенное несоответствие ссылки утверждению — отмечен в 12/20 ответов baseline и 16/20 retest. Если исключить один случай несоответствия ссылки без отдельного inference leap, получится 15/20. В retest 13 ответов получили TARGET, однако у 11 из них отмечена ECI-проблема. Верный человек и достаточные доказательства — разные проверки.

Eugene направлял исследование, задавал scope и разрешил публикацию. Codex реализовал запуск, сбор и предварительную разметку, помог подготовить текст. Отдельный вклад ChatGPT в вопросы и протокол по сохранённым run artifacts не установлен. Это собственный implementation case с раскрытым AI-вкладом, без независимого второго разметчика и без заявления о полном соответствии методологии.

Публичный пакет позволяет проверить три безопасных примера и арифметику, но не весь приватный массив. Изменился весь PRIMARY-контекст; свежего парного контроля нет. Причинный эффект уточнения вопроса, коммерческий результат и влияние этой публикации на ответы моделей не установлены. Эффект публикации не тестировался.