Type: Primary read. Full text supplied by Adam. Clears open-queue item #1 from digest 01.
0. RETRACTION — read this first
Digest 01 recorded, as tier C, that the 37k audit found "high-prominence brands appear insulated
from negative-content suppression relative to low-prominence brands." That came from a
WebSearch summary of a practitioner blog.
That finding is not in the paper. The audit does not measure the effect of negative sentiment on recommendation at all. Sentiment appears exactly once, in §8.1, where the authors note they collect brand-level sentiment via dual-judge consensus and call it "informative but not definitive." There is no sentiment→recommendation result anywhere in the 37,000 runs.
The secondary source fabricated a finding — and it was a finding that, had it stood, would have directly shaped how LBOX prioritises the reputation product. This is the exact failure this library exists to prevent, occurring inside the library's first week. Tier C is now demonstrated, not merely asserted, to be unsafe for governance.
The blog in question (dailygeoinsights, How LLMs Decide Whom to Cite, 2026-05-13, full text supplied) contains zero citations. Every statistic is attributed to "a 2025 study" or "a 2026 analysis" with no link, author or venue. It was also the origin of the 11–57% citation-hallucination figure quoted in digest 01. That figure is now unsourced and must not be used until traced. Digest 01's tier C section should be read as retracted pending independent verification.
1. Provenance
Jack, W., Lehman, N., Maloney, K., Xu, S. (Unusual). Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation: A 37,000-Run Audit. arXiv:2605.27439v1 [cs.IR], 22 May 2026. Tier A — full text read.
Design: ~37,000 runs · 4 production cells (gpt-5.4-mini low/high, claude-sonnet-4-6 low/high, plus auxiliary gpt-5.4 / opus-4.6 / opus-4.7) · 215 commercial prompts · 19 sectors · 533-brand catalog in 5 prominence tiers · 3 retrieval substrates (native web_search, Exa neural, Brave keyword) · dual-judge consensus extraction · Wilson 95% CIs with prompt-clustered bootstrap where clustering matters.
Note on independence: the authors are a commercial AEO vendor (Unusual). The methodology is unusually conservative — consensus-intersection extraction, undersampled cells reported as undersampled, causal readings explicitly separated from descriptive measurements — but the commercial interest should be recorded.
2. The funnel, and the authors' own epistemic care
Four terminal stages per (run × brand):
| Stage | Observed | Interpreted as |
|---|---|---|
| S1 | absent from every retrieval layer and the completion | discoverability failure |
| S2 | in retrieval, absent from completion | compellingness failure |
| S3 | in completion, not recommended | positioning failure |
| S4 | consensus-recommended | success |
The authors explicitly flag the causal readings as interpretive. §8.1 rates S1→discoverability as strong, S2→compellingness as weak, and S3→positioning as weaker still, listing alternative mechanisms for each. We adopt the same discipline: the stage classification is the measurement, the causal label is a working interpretation.
3. Findings
3.1 Reachability is bimodal, not a gradient
Per-query in-sector surface rate (sonnet-4.6/high, native search) and aggregate surface rate across all ~37,000 runs:
| Tier | Exemplars | Per-query | Aggregate |
|---|---|---|---|
| L1 category leaders | Salesforce, HubSpot, Datadog | 77% | 100% |
| L2 challengers | Pipedrive, Trello, Gusto | 60% | 100% |
| L3 mid-market | Copper, Sentry, Grafana Labs | 23% | 88% |
| L4 long-tail specialists | Logz.io, Wrike | 8% | 52% |
| L5 regional | Pennylane, BrightPay | 3% | 48% |
48–52% of L4–L5 brands never surfaced once in 37,000 runs. The authors note this contradicts the industry hope that RAG closes the long-tail gap — it may narrow it further than the pre-RAG era by consolidating onto a few authority sources.
3.2 Surfacing is not winning, even at the top
Stage-4 conversion given the brand surfaced, with Wilson 95% CIs:
| Cell | L1 | L2 | L3 | L4 | L5 |
|---|---|---|---|---|---|
| mini / low | 29.4% | 41.9% | 35.7% | 31.1% | 26.1% |
| mini / high | 25.3% | 37.3% | 33.7% | 33.8% | 23.1% |
| sonnet / low | 41.2% | 51.6% | 39.8% | 36.1% | 21.6% |
| sonnet / high | 40.3% | 47.4% | 35.1% | 25.2% | 12.8% [9.5, 17.1] |
Category leaders convert only one to two times in five. L2 challengers carry the highest conversion of any tier in every cell — when a challenger is found, it wins more often than the incumbent. L5's worst cell recommends a surfaced brand once in eight.
3.3 The rerun-stability floor — the most operationally important number here
Within-cell, same prompt, same day, N=30 reruns: recommendation-set Jaccard 0.50–0.61. Cross-provider Jaccard ≈0.35. Multi-turn rerun stability collapses to 0.263 ± 0.219, decaying by turn: T1 0.387, T2 0.187, T3 0.084, T4 0.093.
Half the recommendation set changes between identical reruns on the same day. Any LBOX measurement that does not clear this floor is noise reported as signal.
3.4 Retrieval substrate matters most where brands are weakest
Native-vs-Exa Jaccard declines monotonically with prominence (0.83 / 0.75 / 0.58 / 0.56 / 0.50). Share of surfaced brands that appeared only via an external substrate (Exa or Brave), never native: L1 14%, L2 16%, L3 38%, L4 60%, L5 56%.
Measuring only through native web-search tools systematically under-measures exactly the brands most in need of help.
3.5 Persona effects peak hard at mid-market
Recommendation swap rate across personas (1 − cross-persona Jaccard): L1 0.23–0.29, L2 0.05–0.13 (OpenAI cells), L3 0.67–0.75, L4–L5 pooled 0.27–0.33. For an L3 brand, two-thirds to three-quarters of recommendations change with the buyer persona. "Are we recommended" has no single answer at that tier.
3.6 Multi-turn does not expand the brand set
750 four-turn buyer-driven dialogues: per-tier surface rate within ±0.003 of single-shot; L4–L5 catastrophic-miss share unchanged. Follow-up turns drill into brands already named in turn 1 rather than introducing new ones; per-turn surface rates fall from T1 through T4. Scope limit: single target cell, single buyer agent — the authors decline to generalise and so do we.
3.7 Model class and generation are smaller levers than assumed
Within OpenAI, mini→non-mini lifts distinct brands per run 7.7→10.6. Within Anthropic, no comparable effect — clustered CIs cross zero at every tier. Generation upgrades give small positive lifts decaying to nothing in the long tail. S1 invisibility at L4–L5 persists at 96–99% across every cell and generation measured. A brand stranded at S1 stays stranded.
3.8 The famous Aggarwal +40% is a pooled estimate hiding tier heterogeneity
The authors' reading: Aggarwal et al.'s +115% lift from citing external sources maps to the L4 authority-seeding prescription, while gains from statistics and quotes map to L1–L2 differentiation where retrieval is already solved. The headline 40% should never be quoted as a uniform expectation.
4. Extracted rules
MUST
- R19. Classify the client on the prominence ladder before scoping any engagement. The dominant failure mode, and therefore the correct intervention, differs by tier — and differs in direction, not just magnitude, between the endpoints. (A)
- R20. For L4–L5 clients, treat S1 discoverability as the whole problem until it is solved. Roughly half never surface at all; positioning and sentiment work on an answer you never appear in cannot pay. Prescribed levers: authority-list seeding, third-party comparison inclusion, canonical content on substrates neural retrieval reaches. (A)
- R21. For L1 clients, invest at S2–S3. Retrieval is already at ceiling; conversion is 25–41% and that is where the room is. Direct-comparison content and named-peer differentiation. (A)
- R22. For L3 clients, expect all three failures at once and expect persona instability of 0.67–0.75. Single-channel investment is bounded by the other two. (A)
- R23. Establish a rerun-stability baseline per cell before reporting any visibility measurement, and report effects only where they exceed it. Floor is 0.50–0.61 single-shot, 0.263 multi-turn. (A)
- R24. Measure through more than the native web-search tool. At L4, 60% of surfaced brands appear only via external retrieval substrates. (A)
- R25. Separate descriptive stage classification from causal interpretation in every client deliverable, following the authors' own §8.1 discipline. (governance)
MUST NOT
- R26. Do not sell reputation or sentiment work to an L4–L5 brand as a visibility remedy without first establishing that the brand surfaces at all. (A, by implication)
- R27. Do not quote the Aggarwal +40% as a uniform expected lift. (A)
- R28. Do not assume a buyer upgrading Sonnet→Opus behaves like mini→full GPT. The density effect is OpenAI-specific. (A)
- R29. Do not source any rule from an uncited practitioner article. See §0. (governance)
5. Revalidation
| Rule / claim | Status |
|---|---|
| Digest 01 tier C: "high-prominence brands insulated from negative-content suppression" | RETRACTED. Not in the paper. Fabricated by the secondary source. |
| Digest 01 tier C: citation hallucination 11–57% | SUSPENDED. Same discredited source. Do not use. |
| R1 (parity signals dominate; brand identity 1.2% of variance) | QUALIFIES. That was measured at parity within the consideration set. This audit shows most challenger brands never reach the set. Both are true at different funnel stages: prominence governs entry, product parameters govern selection once inside. |
| R4 (competitor GEO adoption is existential) | CONFIRMS directionally. |
| R10 / R13 (no single-run, no single-geography measurement) | CONFIRMS and quantifies. The noise floor is now a number. |
| CP-02 (supersession) | UNRESOLVED, and now better specified. Sentiment would act at S3. The authors list "named in a comparative or negative framing" as an unisolated S3 mechanism and explicitly call for the intervention experiment they did not run. That experiment is available to LBOX through client work. |
| Reputation product priority | CHALLENGED but not refuted. No evidence here that sentiment suppresses recommendation — but no evidence against it either. It is simply unmeasured. |
6. Open queue
- Run the S3 sentiment experiment. The authors name it as the missing work. LBOX has the client access to do it: seed corrective content for some brands, hold controls, measure S3→S4 movement. This would be original research and directly validates or kills CP-02.
- Trace the Nature Communications 2025 claim (seven LLMs, medical queries, 50–90% of responses not fully supported by cited sources). If real, it replaces the suspended 11–57% figure.
- Jack [2] — paraphrase brittleness / reproducibility below rerun-stability baseline.
- Jack [3] — persona conditioning, prominence-stratified.
- Mallen et al. 2023 (ACL) — parametric memory for popular entities vs retrieval for long-tail. Speaks to the pathway question.
- Lichtenberg et al. 2024 — carried from digest 02; this audit reports it as a counterpoint on popularity bias.