User-Claude Sonnet 4.5 dialogue to structure the lecture on Trust in the Age of LLMs

M
Help me to plan a 1 h frontal lecture on AI TRUST. Here are the themes I will be touching, suggest 4 different structures
Sperber’s notion of reasons Mercier’s theory of the two evolved functions of reasons Highlight the connection between reasons and social functions Concept of costly signalling Concept of honest signalling Focus on language. Language used to be costly. It was a honest signal of certain qualities. Hard or impossible to fake. What happens if language is cheap. Consider human skills super high skills, middle covered by ai and centaurs Explain that there are two kinds of economics, an economics of utilitarian quality: better language is better than worse language. If language is generally better all benefit. And a market of language as signal a market of status: if language becomes generally better its value falls
Expand on these points General thread how LLM affects trust Be deep just give me structures and ideas. I will write the final lecture
Mostra meno
Here are 4 different structural approaches for your lecture:
Structure 1: “The Evolutionary Arc” (chronological-conceptual)
Act I: The Evolution of Reason as Social Tool (20 min)
• Open with Sperber’s framework: reasons as justifications, not discovery mechanisms
• Mercier’s dual functions: persuasion & evaluation in social contexts
• The social contract of reasoning: we evolved to convince and detect deception
• Bridge: Why did this system work? → Costly signaling
Act II: The Economics of Honest Signals (20 min)
• Costly signaling theory: peacock’s tail logic
• Language as historically costly: cognitive load, time, expertise required
• The signaling market: language doesn’t just communicate—it stratifies
• Two parallel economies: utilitarian (information quality) vs. positional (status/credibility)
• Tension between these markets (when everyone speaks well, no one stands out)
Act III: The LLM Rupture (20 min)
• The collapse of cost: anyone can generate sophisticated language
• Trust crisis: if signals are cheap, evaluation mechanisms break
• The centaur economy: super-high skills valued, middle hollowed out, AI handles low-end
• New equilibrium: what becomes the honest signal when language is free?
• Meta-signals, process transparency, embodied knowledge?
________________________________________
Structure 2: “The Trust Stack” (layered systems approach)
Foundation Layer: Why We Trust Reasons (15 min)
• Sperber’s epistemic vigilance mechanisms
• Mercier: reasons as tools for argumentation, not individual cognition
• The social epistemology: trust is distributed across community validation
Middle Layer: The Signal Economy (25 min)
• Costly vs. honest signaling (Zahavi’s handicap principle)
• Language as historical honest signal: requires intelligence, education, time
• The dual market structure:
o Utilitarian market: collective benefit from better communication
o Positional market: zero-sum status competition through eloquence
• Inherent tension: universal improvement devalues the signal
Surface Layer: LLM Disruption (15 min)
• Cost collapse → signal collapse → trust collapse
• The new skill distribution: bimodal (superhuman + AI, hollowed middle)
• Can’t rely on linguistic sophistication anymore
• What remains trustworthy? Embodied skills, verifiable processes, meta-cognitive honesty
Synthesis (5 min)
• The future of epistemic trust in post-linguistic-scarcity world
________________________________________
Structure 3: “Two Markets, One Crisis” (economic frame)
Part I: The Dual Economics of Language (15 min)
• Market 1 (Utilitarian): Language as pure information transfer
o Collective good: rising tide lifts all boats
o Better communication = better coordination = everyone wins
• Market 2 (Positional): Language as status signal
o Zero-sum game: relative position matters
o If everyone’s eloquent, eloquence means nothing
• These markets always coexisted in tension
Part II: How Costly Signals Mediated This Tension (20 min)
• Sperber/Mercier foundation: reasons evolved for social persuasion/evaluation
• Costly signaling maintained equilibrium: effort required proved quality
• Honest signals: hard to fake = trustworthy indicator
• Language historically balanced both markets: useful AND scarce enough to signal
Part III: LLM as Market Disruptor (20 min)
• Cost → zero, but the two markets respond differently:
o Utilitarian market: potential boom (everyone communicates better)
o Positional market: complete collapse (signals worthless)
• Trust built on positional market, so system-wide crisis
• Centaur economy emerges: super-elite, AI baseline, vanishing middle
• New question: In a world of free eloquence, what becomes the honest signal?
Coda (5 min)
• Paradox: We gain utility but lose trust infrastructure
________________________________________
Structure 4: “The Question Cascade” (problem-driven)
Q1: Why Do We Trust Reasons at All? (15 min)
• Not because they reveal truth (Sperber’s critique of intellectualist view)
• Because they serve social functions (Mercier): persuasion and evaluation
• Reasons are moves in a social game, not mental representations
• We trust them when the game has integrity—when players have skin in it
Q2: What Kept the Game Honest? (15 min)
• Costly signaling: cheap talk is cheap
• Language required: time, intelligence, education, cognitive effort
• This cost structure created honest signals—hard to fake competence
• Two simultaneous markets operated: information exchange + status signaling
Q3: What Happens When Signals Become Free? (15 min)
• LLMs eliminate cost barrier
• Utilitarian market: potential explosion of communication quality
• Positional market: catastrophic devaluation (if all peacocks have brilliant tails…)
• The trust mechanism (costly signal → honest signal → belief) breaks
Q4: What Can We Trust in Post-Cost World? (12 min)
• The centaur stratification: superhuman skills, AI-augmented baseline, hollowed middle
• Can’t trust linguistic performance anymore
• Candidates for new honest signals:
o Process transparency (showing your work)
o Embodied/executable knowledge
o Meta-cognitive awareness (knowing when you’re using AI)
o Track records, skin in game, verifiable outcomes
Closing (3 min)
• The challenge: rebuilding epistemic infrastructure for cheap-signal world
________________________________________
Cross-cutting themes to weave throughout any structure:
• The irony: LLMs make us better at producing reasons while making reasons worse at producing trust
• Distinction between competence signaling vs. commitment signaling
• Historical parallel: what happened to other costly signals when they became cheap? (e.g., printed books, photography)
• The centaur economy isn’t stable—it’s a transitional phase to something else
Which structure resonates with your teaching style?
M
Tell me about zahavi’s principe
Zahavi’s Handicap Principle
The Core Paradox
Amotz Zahavi proposed (1975) a solution to one of evolution’s most puzzling problems: Why do animals develop traits that seem actively harmful to survival? The peacock’s tail is the classic example—it’s metabolically expensive, attracts predators, and impairs movement. Darwin’s sexual selection explained that females prefer it, but not why such preferences would evolve.
Zahavi’s insight: The cost IS the point.
The Logic
A signal is reliable (honest) precisely because it’s expensive to produce. Only high-quality individuals can afford the handicap.
• Cheap signals can be faked → unreliable → ignored by evolution
• Costly signals cannot be faked → reliable indicators of underlying quality → selected for
The peacock’s tail honestly signals: “I have such good genes that I can survive despite carrying this ridiculous burden.” A weak peacock literally cannot grow a magnificent tail and survive. The cost enforces honesty.
Formal Conditions
For a handicap to work as an honest signal:
1. Differential cost: The signal must be more costly for low-quality individuals than high-quality ones
2. Correlation: The ability to bear the cost must correlate with the quality being signaled (health, genetic fitness, resource access)
3. Observability: Receivers must be able to perceive the signal and its costliness
Beyond Biology: The Principle in Human Contexts
The principle extends far beyond peacock tails:
Education as costly signal (Spence, 1973)
• Degrees may signal intelligence/conscientiousness more than specific skills
• The difficulty/cost of obtaining education is what makes it believable
• If everyone could trivially get a PhD, it would signal nothing
Luxury goods
• Expensive watches, designer clothes signal wealth precisely because they’re wasteful
• If Rolex cost $50, it would signal nothing about the wearer’s resources
Religious rituals
• Costly commitments (dietary restrictions, celibacy, time-consuming practices) signal genuine belief/group commitment
• Hard to fake devotion when it requires real sacrifice
Language and rhetoric (your lecture context)
• Historically: eloquence required years of education, cognitive ability, time
• The cost structure made sophisticated language an honest signal of:
o Intelligence
o Education/social class
o Time wealth (leisure to develop rhetorical skills)
o Cognitive capacity
The Critical Insight for Your AI Trust Lecture
When LLMs make eloquent language essentially free:
1. Differential cost collapses: Anyone can produce sophisticated text regardless of underlying capability
2. The correlation breaks: Linguistic performance no longer correlates with intelligence, education, or cognitive quality
3. The signal becomes dishonest: What previously indicated real capacity now indicates access to GPT-4
This is a structural trust crisis, not just a practical one. The mechanism that enforced honesty (cost) has been removed. We’re in a transitional period where:
• Old evaluation heuristics (judge people by how they write/speak) no longer work
• New honest signals haven’t yet stabilized
• Everyone is vulnerable to deception (including self-deception about AI augmentation)
The Zahavi Framework Predicts:
New honest signals will emerge, and they’ll be costly in new ways:
• Process transparency (showing messy work) – costly because it reveals uncertainty
• Embodied demonstration – costly because it requires physical presence/action
• Long-term track records – costly because they require time and consistent performance
• Meta-honesty about AI use – costly because it sacrifices status/impression management
The question for your lecture: In a world where language is cheap, what becomes the new handicap that enforces trust?
M
Why do you Attribute it to zahavi
You’re right to push on this – I should be more precise about intellectual credit.
The Attribution Issue:
The handicap principle is indeed Zahavi’s (1975, later formalized with Grafen 1990). But when discussing costly signaling more broadly, the credit picture is more complex:
Costly Signaling Theory – Multiple Origins:
1. Economic signaling – Michael Spence (1973, Nobel Prize 2001)
o Job market signaling, education as signal
o Developed independently in economics
2. Biological handicap principle – Amotz Zahavi (1975)
o Initially rejected, later vindicated by Grafen’s mathematical models (1990)
o Specifically about wasteful traits that work because they’re costly
3. Honest signaling – broader concept with multiple contributors
o John Maynard Smith & David Harper’s work on animal signals
o Distinction between costly signals and indices (unfakeable cues)
Why I Defaulted to Zahavi:
1. The peacock’s tail is the most vivid illustration
2. His principle is specifically about the paradox of wasteful signals – which maps beautifully to your language/LLM case
3. The “handicap” framing emphasizes that cost isn’t a bug, it’s the feature
More Accurate Attribution for Your Lecture:
You might frame it as:
• Costly signaling theory (Spence in economics, Zahavi in biology)
• Zahavi’s handicap principle specifically when emphasizing the paradox that waste enforces honesty
• Honest signaling as the broader framework
For your purposes:
• If emphasizing the economic markets of language → cite Spence
• If emphasizing the paradox that language being hard is what made it trustworthy → cite Zahavi
• If discussing the general reliability problem → honest signaling literature (Maynard Smith & Harper)
Should I have presented it as “costly signaling theory” more broadly rather than leading with Zahavi?
M
Take the last structure and expand it into a stronger structure with the most salient information from the others. Integrate a return to the effect on reason. Use the metaphor of viral reason Not vital because it expands a lot. I want to talk about reasons who do not have persons to back them
Structure 4+: “The Question Cascade” (Enhanced)
________________________________________
Q1: Why Do We Trust Reasons at All? (12 min)
Opening: The Puzzle
• Reasons don’t compel belief logically—bad arguments convince people daily
• Yet we navigate a world of claims, arguments, justifications
• What makes some reasons trustworthy and others not?
Sperber’s Framework: Reasons as Social Objects
• Not mental representations of truth-discovery process
• Reasons are justifications produced for others
• We evolved epistemic vigilance mechanisms—we’re natural lie detectors, argument evaluators
Mercier’s Dual Functions
• Persuasion: produce reasons to convince others
• Evaluation: assess reasons offered by others
• This is a social system—reasons evolved for argumentation, not individual cognition
• The game has integrity when players have accountability
The Trust Mechanism (Traditional)
• Reasons come from persons
• Persons bear costs for bad reasoning (reputation, social consequences)
• Persons have track records, observable incentives, skin in game
• We trust reasons because we can evaluate their source
Transition: But what keeps sources honest?
________________________________________
Q2: What Kept the Game Honest? (18 min)
The Economics of Costly Signals
• Not all signals are equally believable
• Costly signaling theory (Spence, Zahavi): signals are honest when they’re expensive to fake
• Differential cost structure: high-quality signaler bears cost more easily than low-quality faker
Zahavi’s Handicap Principle: The Paradox of Waste
• Peacock’s tail: costly because it’s a burden
• The cost enforces correlation between signal and underlying quality
• Only genuinely fit peacocks can afford magnificent tails
• Weakness: if cost drops, signal becomes fakeable → dishonest → ignored
Language as Historical Handicap
• Producing sophisticated reasoning/eloquence required:
o Cognitive capacity (intelligence)
o Time investment (education, practice)
o Social access (cultural capital)
o Sustained effort (composition, revision)
• These costs made language an honest signal of:
o Intellectual capability
o Educational background
o Commitment/seriousness
o Social position
The Dual Market Structure
• Utilitarian market: language as information technology
o Better communication = better collective coordination
o Non-zero-sum: everyone benefits from clarity
o Value: functional quality of information transfer
• Positional market: language as status signal
o Zero-sum competition: relative position matters
o Eloquence distinguishes you from others
o Value: scarcity and exclusivity
o If everyone is eloquent, no one is distinguished
Tension in Equilibrium
• These markets always coexisted
• Costly signaling maintained balance:
o Utilitarian market got some quality (effort required meant some competence)
o Positional market got differentiation (not everyone could afford the cost)
• Trust infrastructure depended on both: we believed reasons because quality sources were identifiable through costly signals
Transition: What happens when the cost structure collapses?
________________________________________
Q3: What Happens When Signals Become Free? (18 min)
The LLM Cost Collapse
• Anyone can generate sophisticated arguments, eloquent prose, technical reasoning
• Differential cost → zero (or: cost no longer correlates with underlying quality)
• The handicap principle’s conditions violated:
o Cost no longer filters by capability
o Signal no longer correlates with quality being signaled
o Cheap talk is now literally cheap
Asymmetric Market Response
Utilitarian Market: Potential Boom
• Everyone can communicate more clearly
• Translation, summarization, explanation all improve
• Collective informational benefit
• Best case: rising tide lifts all boats
Positional Market: Complete Collapse
• If everyone produces sophisticated language, sophistication signals nothing
• The peacock problem: when all peacocks have AI-generated tails, tails mean nothing
• Status differentiation through eloquence becomes impossible
• The scarcity that created value evaporates
The Trust Crisis
• Our trust mechanisms evolved for the positional market
• We judge argument quality partly through source quality
• Source quality was signaled through costly linguistic performance
• Now: signal breakdown → can’t identify quality sources → trust infrastructure fails
The Centaur Economy Emergence
• Skill distribution becomes bimodal:
o Super-high skills: genuinely exceptional, beyond AI capabilities (for now)
o AI baseline: competent, accessible to anyone with GPT
o Hollowed middle: skills AI can replicate become devalued
• But even this is unstable—transitional phase, not equilibrium
The Deepening Problem: Geographic vs. Viral Reasons
Traditional (Geographic) Reasons
• Attached to persons, institutions, communities
• Traceable to accountable sources
• Evaluated through: reputation, incentives, track records
• Trust through social embedding
Emerging (Viral) Reasons
• Reasons without persons to back them
• Arguments that circulate, mutate, spread independent of stable authorship
• AI-generated, remixed, anonymized, platform-amplified
• No one bears cost for being wrong
• No track record to evaluate
• No reputation at stake
The Viral Reason Problem
• Like biological viruses: optimized for spread, not truth
• Selection pressure: virality (engagement, emotional resonance) not epistemic quality
• Reasons evolve to be persuasive without being accountable
• We can’t apply traditional epistemic vigilance—no source to scrutinize
Catastrophic Combination
• LLMs make reason-production cheap
• Platforms make reason-spreading viral
• Result: ecosystem flooded with sophisticated-sounding, source-less arguments
• Our evolved evaluation mechanisms (assess source, check incentives, demand accountability) don’t work on viral reasons
Transition: So what can we trust?
________________________________________
Q4: What Can We Trust in a Post-Cost World? (10 min)
The New Honest Signals (Speculative)
What’s costly in a world where language is free?
1. Process Transparency
o Showing messy work, uncertainty, revision
o Costly because: exposes vulnerability, sacrifices polished image
o Signals: intellectual honesty, genuine engagement
2. Embodied/Executable Knowledge
o Physical demonstrations, buildable artifacts, reproducible results
o Costly because: requires actual capability, not linguistic performance
o Signals: practical competence, not just rhetorical skill
3. Long-term Track Records
o Consistent performance over time, verifiable outcomes
o Costly because: requires sustained commitment, can’t be faked quickly
o Signals: reliability, genuine expertise
4. Meta-cognitive Honesty
o Disclosing AI use, admitting uncertainty, showing reasoning boundaries
o Costly because: sacrifices status, reveals limitations
o Signals: epistemic integrity
5. Institutional/Reputational Binding
o Putting institutional credibility behind claims
o Costly because: wrong claims damage long-term reputation capital
o Signals: seriousness, accountability
The Return to Reason: Mercier’s Functions Under Stress
• Persuasion function: hyperstimulated
o Easier than ever to produce persuasive arguments
o But persuasion decoupled from truth/quality
o Viral reasons optimized purely for persuasion
• Evaluation function: overwhelmed
o Can’t rely on traditional source-assessment
o Flooded with sophisticated-seeming arguments
o No persons to hold accountable for viral reasons
o Our epistemic vigilance mechanisms evolved for person-backed reasons
The Fundamental Asymmetry
• LLMs made producing reasons cheap
• But didn’t make evaluating reasons easier
• In fact, made evaluation harder (more volume, harder to trace sources)
• Mercier’s two functions now wildly imbalanced
The Viral Reason Ecosystem
• Reasons reproduce and spread like organisms
• Selection pressure: engagement, not truth
• No evolutionary pressure for accuracy when reasons have no authors
• We’re in an informational pandemic: sophisticated-sounding claims spreading faster than our ability to evaluate them
________________________________________
Closing Synthesis (2 min)
The Core Irony
• LLMs make us better at producing reasons (persuasion function)
• While making reasons worse at producing trust (evaluation function breaks)
• We’ve optimized for one half of Mercier’s system while destroying the other
The Challenge
• Can’t go back to costly language (genie’s out)
• Can’t trust linguistic sophistication anymore
• Must rebuild epistemic infrastructure for a world where:
o Reasons are cheap
o Reasons are viral (unattached to accountable persons)
o Traditional evaluation mechanisms don’t work
The Open Question
• In an ecosystem of viral reasons competing for attention, not accuracy…
• Where linguistic handicaps no longer enforce honesty…
• What becomes the new basis for epistemic trust?
________________________________________
Key Thematic Threads Throughout:
1. Cost as enforcer: from Zahavi through language to LLMs
2. Dual markets tension: utilitarian vs. positional, and which one trust depends on
3. Person vs. virus: geographic reasons (accountable) vs. viral reasons (free-floating)
4. Mercier’s imbalance: persuasion hyperstimulated, evaluation broken
5. The handicap loss: what happens when the wasteful thing that guaranteed honesty disappears
Would you like me to suggest specific examples, case studies, or visual metaphors for any of these sections?
M
Let’s talk about the puzzle. I think that this is not sperber’s view. I think that is that even without relying on formal logic there is an evolved capacity to detect if arguments make sense as such, am I wrong
You’re right to call me out – I was being sloppy about Sperber’s actual position.
Sperber’s View (More Accurately):
Sperber does think we have evolved cognitive mechanisms for evaluating arguments as arguments – not just for evaluating sources. This is his epistemic vigilance framework.
Key Components:
1. We can assess argument quality directly
o Not through formal logic (most people can’t do that)
o But through intuitive coherence-checking, plausibility assessment
o “Does this make sense?” is a genuine cognitive capacity
2. Vigilance mechanisms are calibrated
o We check: Does this cohere with what I know?
o Does this follow sensibly from the premises?
o Are there obvious gaps or contradictions?
o We’re actually quite good at this intuitively
3. Dual track evaluation
o We assess both SOURCE (who’s saying this? what are their incentives?)
o AND CONTENT (does the argument itself make sense?)
o These work together but are separable
Where I Went Wrong:
I collapsed Sperber’s position into “we only trust reasons by trusting sources” – but that’s not his view at all. He thinks we have both:
• Content vigilance (argument makes sense internally)
• Source vigilance (speaker is trustworthy)
So the Puzzle Should Be Reframed:
Opening: The Dual Puzzle
• We’re surprisingly good at detecting bad arguments even when we can’t articulate why
• “That doesn’t follow” is an intuition we have before we can explain the logical gap
• We also track sources: who benefits? who has expertise? who’s been reliable?
Sperber’s Answer:
• We evolved specialized cognitive mechanisms for epistemic vigilance
• These operate on both:
o Content level: does this argument cohere? make sense? fit with what I know?
o Source level: is this person reliable? honest? competent?
• Not formal logic, but evolved intuitive inference-checking
Mercier Adds:
• These mechanisms evolved because reasoning is social
• We produce reasons to persuade (so we got good at making arguments)
• We evaluate reasons from others (so we got good at detecting weak ones)
• It’s an arms race: better persuaders → better evaluators → better persuaders
The Trust Mechanism (Revised):
• We trust reasons through BOTH channels:
o Direct argument assessment (does this cohere?)
o Source assessment (who’s making this claim?)
• Costly signaling enters primarily on the SOURCE side
• But also affects content: producing coherent, sophisticated arguments was costly
Does this better capture the puzzle you want to set up?
The question becomes: If we have evolved content-vigilance mechanisms, why does the LLM shift still create a trust crisis?
Possible answer: Because our content-vigilance evolved in an environment where coherent-sounding arguments correlated with genuine understanding. When that correlation breaks (AI can produce coherent arguments without understanding), our intuitive mechanisms give false positives.
Am I closer now?
M
Yes, you should rethink 4+
Structure 4+: “The Question Cascade” (Revised)
________________________________________
Q1: Why Do We Trust Reasons at All? (12 min)
Opening: The Dual Puzzle
• We’re not logicians, yet we navigate a world saturated with arguments
• We can often detect that “something’s wrong” with an argument before we can articulate why
• We also care about who is arguing: expertise, incentives, track record matter
• How do these two work together?
Sperber’s Epistemic Vigilance
• We evolved specialized cognitive mechanisms for evaluating arguments
• Dual-track evaluation:
o Content vigilance: Does this argument cohere? Does it make sense given what I know? Are there obvious gaps?
o Source vigilance: Who’s saying this? What are their incentives? Are they competent/reliable?
• Not formal logic, but intuitive inference-checking
• These mechanisms are calibrated – we’re actually quite good at this
Mercier’s Evolutionary Story
• The argumentative theory of reasoning
• Reasoning didn’t evolve for individual truth-finding (we’re bad at that)
• It evolved for social persuasion and evaluation
• Two functions in arms race:
o Producing reasons (persuasion): make arguments to convince others
o Evaluating reasons (vigilance): assess arguments from others
• Better persuaders → better evaluators → better persuaders
• This creates pressure for both sophisticated argument production and sophisticated detection
The Historical Equilibrium
• Content vigilance worked because coherent arguments were hard to produce
• Source vigilance worked because people bore reputational costs for bad reasoning
• These reinforced each other:
o Good arguments suggested competent sources
o Competent sources suggested arguments worth engaging with
• Critical correlation: The ability to produce coherent, sophisticated arguments correlated with genuine understanding/expertise
Transition: But this correlation depended on something specific about how arguments were produced…
________________________________________
Q2: What Made Arguments Costly (And Why Did That Matter)? (18 min)
The Hidden Cost Structure
• Producing sophisticated arguments historically required:
o Cognitive capacity (intelligence, working memory)
o Deep knowledge (education, experience)
o Time investment (composition, revision, practice)
o Sustained attention (coherence across complex chains of reasoning)
• These costs created a filter
Costly Signaling Theory (Spence, Zahavi)
• Signals are reliable when they’re expensive to fake
• Zahavi’s Handicap Principle: The peacock’s tail works because it’s wasteful
o Only genuinely fit peacocks can afford the burden
o Cost enforces correlation between signal (tail) and quality (fitness)
o If cost drops, anyone can fake the signal → becomes unreliable
Language/Arguments as Handicap
• Sophisticated reasoning was a costly signal of:
o Intellectual capacity
o Domain knowledge
o Educational investment
o Cognitive resources
• The key correlation: Producing coherent arguments required actually understanding the domain
• You couldn’t fake deep understanding without having it (or paying someone who did)
This Supported Both Vigilance Mechanisms
For Content Vigilance:
• Coherent-sounding arguments generally came from people who understood the domain
• Our intuitive “does this make sense?” detector was calibrated to this reality
• False positives were rare (hard to produce sophisticated nonsense)
For Source Vigilance:
• Demonstrated argumentative skill revealed genuine capability
• Track record of good arguments = track record of understanding
• Reputational costs for being wrong were real (you’d invested effort, revealed limitations)
The Dual Market Structure (More Nuanced)
• Utilitarian market: Arguments as information/coordination tools
o Better reasoning = better collective decision-making
o Non-zero-sum benefit
o Value: functional quality for truth-finding
• Positional market: Arguments as status signals
o Argumentative sophistication as competitive advantage
o Zero-sum: relative eloquence matters
o Value: distinction/credibility through demonstrated skill
The Markets Interacted
• Positional competition drove people to develop genuine skills
• These genuine skills produced actually useful arguments (utilitarian benefit)
• The costly investment meant people cared about being right (reputation at stake)
• Trust depended on this entire ecology
Transition: What happens when the cost structure that maintained this equilibrium vanishes?
________________________________________
Q3: What Happens When Argument Production Becomes Free? (18 min)
The LLM Transformation
• Anyone can now generate:
o Coherent-sounding arguments
o Sophisticated prose
o Technical reasoning
o Multi-step logical structures
• Cost → essentially zero
• Time investment → minutes instead of hours/years
The Correlation Breaks
• Historical: Coherent argument ← requires → genuine understanding
• Post-LLM: Coherent argument ← requires → access to GPT
• The coupling between argumentative sophistication and actual comprehension is severed
Mercier’s Functions Become Unbalanced
Persuasion Function: Hyperstimulated
• Producing compelling arguments is now trivial
• Volume of sophisticated-seeming reasons explodes
• Quality variance collapses (everything sounds pretty good)
• No effort barrier means no commitment signal
Evaluation Function: Overwhelmed
• Content vigilance gives false positives:
o “Does this cohere?” → Yes (AI is good at local coherence)
o “Does this make sense?” → Seems to (sophisticated surface form)
o But: coherence no longer implies understanding
o Our intuitions were calibrated for a world where incoherent = incompetent
• Source vigilance breaks down:
o Can’t tell if person or AI produced the argument
o No way to assess genuine expertise from text alone
o Reputational costs weaken (everyone uses AI, plausible deniability)
The Asymmetry
• LLMs made producing arguments cheap
• But didn’t make evaluating arguments easier
• In fact, made evaluation harder:
o More volume to process
o Better surface quality makes bad arguments harder to spot
o Source signals unreliable
• Arms race tips toward persuaders, away from evaluators
Geographic vs. Viral Reasons
Traditional (Geographic) Reasons
• Attached to identifiable persons/institutions
• Embedded in communities with norms
• Traceable authorship
• Accountability through reputation
• Both vigilance mechanisms can engage:
o Check the argument (content)
o Check the source (person/institution)
Emerging (Viral Reasons)
• Arguments without stable authorship
• AI-generated, remixed, anonymized, recycled
• Spread through platforms optimized for engagement
• Mutate as they spread (telephone game with LLMs)
• No one ultimately accountable
The Viral Reason Problem
• Like biological viruses: selected for replication, not truth
• Optimized through engagement metrics:
o Emotional resonance
o Confirmation of priors
o Shareability
o Sophisticated surface form (thanks to AI polish)
• Our evolved evaluation mechanisms fail:
o Content vigilance: Viral reasons are often coherent-sounding (AI-polished)
o Source vigilance: No stable source to evaluate (who do you hold accountable?)
• Selection pressure inverts:
o Historical: good arguments survived because they convinced and their sources stayed credible
o Viral: arguments survive if they spread, regardless of truth
o Truth becomes orthogonal to fitness
Market Response
Utilitarian Market:
• Potential: Everyone can articulate complex ideas
• Reality: Flood of plausible-sounding arguments of uncertain quality
• Navigation becomes harder, not easier
• Information overload without quality signals
Positional Market:
• Complete collapse of linguistic sophistication as signal
• When everyone has AI-written eloquence, eloquence means nothing
• Status differentiation through argument quality becomes impossible
• The scarcity that created value evaporates
The Compounding Crisis
1. Our content vigilance is fooled (coherence without understanding)
2. Our source vigilance has nothing to grab onto (viral reasons, unclear authorship)
3. The volume overwhelms our processing capacity
4. Reputational costs for bad arguments decline (everyone uses AI, everyone has deniability)
5. Arguments optimize for spread rather than truth
We’re in an epistemic environment our cognitive systems weren’t designed for.
Transition: So what can we actually trust?
________________________________________
Q4: Rebuilding Trust in a Post-Cost World (10 min)
The Challenge
• Can’t rely on argumentative sophistication (too cheap to produce)
• Can’t rely on coherence alone (AI is good at local coherence)
• Can’t always identify sources (viral reasons)
• Our evolved vigilance mechanisms are miscalibrated
What Becomes Costly (And Therefore Honest)?
1. Process Transparency
o Showing messy reasoning, dead ends, uncertainty
o Costly because: exposes intellectual vulnerability
o Signals: genuine engagement, intellectual honesty
o Problem: Also fakeable with AI (“generate a draft with uncertainties visible”)
2. Embodied/Executable Knowledge
o Physical demonstrations, working code, reproducible experiments
o Costly because: requires actual capability beyond language
o Signals: practical competence, not just rhetorical skill
o Harder to fake: “show me it working” vs. “tell me how it works”
3. Long-term Track Records
o Consistent performance over time, verifiable predictions/outcomes
o Costly because: requires sustained commitment, can’t be instantly generated
o Signals: reliable expertise, skin in game
o Problem: Slow to establish, harder for newcomers
4. Meta-cognitive Honesty
o Explicitly disclosing AI use, boundaries of knowledge, reasoning limitations
o Costly because: sacrifices status, reveals dependence
o Signals: epistemic integrity
o Paradox: Currently signals weakness, but might become strength
5. Institutional Commitment
o Organizations staking reputation on claims
o Costly because: wrong claims damage accumulated credibility
o Signals: seriousness, accountability structures
o Problem: Institutions also use AI, face same pressures
Recalibrating Vigilance
Content Vigilance Must Evolve:
• From “does this cohere?” to “does this connect to verifiable reality?”
• From admiring sophistication to demanding evidence
• From evaluating arguments to evaluating argument sources (data, methods, transparency)
• We need new intuitions for AI-saturated environment
Source Vigilance Must Adapt:
• From “who said this?” to “what’s the chain of accountability?”
• From individual reputation to institutional/process credibility
• From evaluating polish to evaluating vulnerability/honesty
• Track meta-signals: disclosure, process, track record over time
The Return to Reason: Mercier’s Functions in Crisis
• Persuasion function: Operating in overdrive
o Easier than ever to produce compelling arguments
o But decoupled from understanding
o Viral reasons optimize purely for persuasion (engagement)
• Evaluation function: Under siege
o Volume overwhelming
o Quality signals degraded
o Source signals unreliable
o Content signals (coherence) give false positives
The Fundamental Imbalance
• We evolved these in arms race equilibrium
• LLMs broke the equilibrium:
o Made persuasion trivial
o Made evaluation exponentially harder
o The “better persuaders → better evaluators” cycle is breaking
The Viral Reason Pandemic
• Arguments reproducing and spreading like pathogens
• Selection for virality, not veracity
• No persons bearing reputational cost
• Our immune system (epistemic vigilance) encounters novel pathogens it can’t recognize
• Sophisticated-sounding claims spreading faster than our ability to evaluate them
________________________________________
Closing Synthesis (2 min)
The Core Transformation
• Sperber and Mercier described an evolved system that worked
• It worked because of a hidden assumption: producing good arguments was hard
• That difficulty created correlation between sophistication and understanding
• That correlation calibrated both our content and source vigilance
• LLMs severed the correlation
The Irony
• We have better tools for producing reasons (persuasion hyperstimulated)
• We have worse ability to trust reasons (evaluation mechanisms broken)
• Arguments are simultaneously more sophisticated and less reliable
• The very thing that made arguments trustworthy (costliness) is what we’ve eliminated
The Open Question
• In an ecosystem where:
o Coherent arguments are cheap to produce
o Viral reasons spread without accountable authors
o Our evolved vigilance mechanisms are miscalibrated
o Volume overwhelms our processing capacity
What becomes the new basis for epistemic trust?
The answer isn’t clear yet. We’re living through the transition. The old handicaps that enforced honesty are gone. The new ones haven’t stabilized.
We’re in an informational interregnum – between the epistemic order that was and whatever comes next.
________________________________________
Does this better capture the dynamic you want to explore? The key shift is that both content and source vigilance are under pressure, but for different reasons – and viral reasons specifically exploit the gap where neither mechanism can get purchase.
M
Can you add the concept of meme, the original one from Dawkins to the viral reason account. There’s nothing original in the Idea, it’s Dawkins+marcier
You’re absolutely right – I was reinventing the wheel. Let me revise that section:
________________________________________
Q3: What Happens When Argument Production Becomes Free? (18 min)
The LLM Transformation
• Anyone can now generate:
o Coherent-sounding arguments
o Sophisticated prose
o Technical reasoning
o Multi-step logical structures
• Cost → essentially zero
• Time investment → minutes instead of hours/years
The Correlation Breaks
• Historical: Coherent argument ← requires → genuine understanding
• Post-LLM: Coherent argument ← requires → access to GPT
• The coupling between argumentative sophistication and actual comprehension is severed
Mercier’s Functions Become Unbalanced
Persuasion Function: Hyperstimulated
• Producing compelling arguments is now trivial
• Volume of sophisticated-seeming reasons explodes
• Quality variance collapses (everything sounds pretty good)
• No effort barrier means no commitment signal
Evaluation Function: Overwhelmed
• Content vigilance gives false positives:
o “Does this cohere?” → Yes (AI is good at local coherence)
o “Does this make sense?” → Seems to (sophisticated surface form)
o But: coherence no longer implies understanding
o Our intuitions were calibrated for a world where incoherent = incompetent
• Source vigilance breaks down:
o Can’t tell if person or AI produced the argument
o No way to assess genuine expertise from text alone
o Reputational costs weaken (everyone uses AI, plausible deniability)
The Asymmetry
• LLMs made producing arguments cheap
• But didn’t make evaluating arguments easier
• In fact, made evaluation harder:
o More volume to process
o Better surface quality makes bad arguments harder to spot
o Source signals unreliable
• Arms race tips toward persuaders, away from evaluators
Dawkins’ Memes: The Original Framework
What Memes Are (Dawkins, 1976)
• Cultural replicators, analogous to genes
• Units of cultural transmission: ideas, tunes, catchphrases, arguments
• Replicate by jumping from brain to brain
• Subject to variation, selection, retention
• Key insight: Memes compete for replication, not necessarily for truth
Memetic Selection Pressures
• Fidelity: how accurately does it copy?
• Fecundity: how many copies does it make?
• Longevity: how long does each copy last?
• Critical point: These don’t correlate with truth value
• A false but memorable idea can outcompete a true but boring one
Traditional Memetic Constraints
• Horizontal constraint: Peer evaluation in communities
o Other people check your arguments
o Reputational costs for spreading bad memes
o Social accountability acts as quality filter
• Vertical constraint: Cost of meme production
o Sophisticated arguments required effort
o Investment meant people cared about quality
o Hard to mass-produce compelling nonsense
Mercier + Dawkins: Reasons as Memes
The Synthesis
• Reasons are a special class of memes
• They evolved to serve Mercier’s dual functions (persuasion/evaluation)
• But they replicate memetically (Dawkins)
• Historically balanced:
o Memetic selection pressure (spread)
o Epistemic vigilance pressure (truth/quality)
o Reputational accountability (source)
Why the Balance Held
• Producing sophisticated reason-memes was costly
• This cost meant:
o Limited volume (evaluation could keep up)
o Quality correlation (good arguments = capable sources)
o Accountability (sources attached to reasons)
• Our vigilance mechanisms evolved to operate in this ecology
The LLM Disruption: Unleashing Memetic Selection
Constraint Removal
• Horizontal constraint weakens:
o Can’t identify sources reliably (AI vs. human unclear)
o Reputational costs diluted (everyone uses AI)
o Community evaluation overwhelmed by volume
• Vertical constraint collapses:
o Cost of producing sophisticated reason-memes → zero
o Anyone can mass-produce compelling arguments
o Investment no longer signals commitment
Pure Memetic Competition
• Reason-memes now selected primarily for:
o Fecundity: How shareable? How engaging?
o Fidelity: How memorable? How quotable?
o Longevity: How emotionally resonant?
• NOT selected for:
o Truth value
o Epistemic quality
o Source credibility
The Pandemic Metaphor (More Precise Now)
• Not just a metaphor – literal memetic evolution
• Reason-memes reproducing with variation
• Selection environment has changed:
o Old environment: filtered by costly production + epistemic vigilance
o New environment: optimized for engagement metrics
• Our “immune system” (vigilance) evolved for old environment
• Encountering reason-memes evolved in new environment
The Specific Failure Modes
1. Content vigilance fooled
o AI produces locally coherent reason-memes
o Our “does this make sense?” evolved for human-produced memes
o Sophisticated surface form triggers false confidence
2. Source vigilance has no target
o Reason-memes circulate detached from stable authors
o Mutate as they spread (LLMs can rephrase/remix)
o No person to hold accountable
o Can’t apply “who benefits?” heuristic
3. Volume overwhelms processing
o Mass production of reason-memes
o Our evaluation capacity evolved for much lower density
o Triage replaces careful evaluation
4. Selection inverts
o Historically: good reason-memes survived because they convinced AND maintained source credibility
o Now: reason-memes survive if they spread, period
o Truth becomes orthogonal to fitness
Market Response
Utilitarian Market:
• Potential: Everyone can articulate complex ideas
• Reality: Flood of reason-memes competing for attention
• Navigation becomes harder – signal-to-noise ratio collapses
• Can’t distinguish high-quality from high-virality
Positional Market:
• Complete collapse of linguistic sophistication as signal
• When everyone can produce sophisticated reason-memes, sophistication signals nothing
• Status differentiation through argument quality impossible
• The scarcity that created value evaporates
The Compounding Crisis
We’re experiencing runaway memetic evolution of reasons:
• Production costs removed (LLMs)
• Selection pressure shifted (engagement platforms)
• Quality filters degraded (vigilance overwhelmed)
• Accountability weakened (source opacity)
Result: An explosion of sophisticated-sounding reason-memes optimized for replication, not truth.
This isn’t a new phenomenon – it’s Dawkins’ memetics operating without the constraints that previously kept reason-memes somewhat tethered to reality.
Transition: So what can we actually trust?
________________________________________
The key addition: Framing “viral reasons” explicitly as memetic evolution makes clear this isn’t some novel AI phenomenon, but rather the removal of constraints on a process Dawkins identified 50 years ago. LLMs didn’t create memetic competition – they just removed the cost barrier that kept it somewhat in check for the specific domain of sophisticated argumentation.
Does this integration work better?
M
The disruption part is apocalyptic. Mentions that some people talk about epistemia (quattrociocchi). But we need a twist in the story. It’s unlikely that humans will stay in epistemia forever? Why? The answer is in Mercier’s arm race account. Just the new equilibrium will be different.
Yes! Let me revise that section with the turn:
________________________________________
The LLM Disruption: Unleashing Memetic Selection
Constraint Removal
• Horizontal constraint weakens:
o Can’t identify sources reliably (AI vs. human unclear)
o Reputational costs diluted (everyone uses AI)
o Community evaluation overwhelmed by volume
• Vertical constraint collapses:
o Cost of producing sophisticated reason-memes → zero
o Anyone can mass-produce compelling arguments
o Investment no longer signals commitment
Pure Memetic Competition
• Reason-memes now selected primarily for:
o Fecundity: How shareable? How engaging?
o Fidelity: How memorable? How quotable?
o Longevity: How emotionally resonant?
• NOT selected for:
o Truth value
o Epistemic quality
o Source credibility
The Epistemic Crisis
The Apocalyptic View
• Some researchers (Quattrociocchi and others) warn of “epistemia”
• Permanent degradation of our ability to know things collectively
• Information ecosystem too polluted to navigate
• Trust infrastructure irreparably broken
• Reason-memes optimized purely for virality, detached from truth
The Failure Modes
1. Content vigilance gives false positives (AI coherence fools us)
2. Source vigilance has no target (authorship unclear)
3. Volume overwhelms processing capacity
4. Selection pressure inverts (spread matters, truth doesn’t)
Market Response
Utilitarian Market:
• Flood of reason-memes competing for attention
• Signal-to-noise ratio collapses
• Can’t distinguish high-quality from high-virality
Positional Market:
• Linguistic sophistication as signal completely collapses
• When everyone produces sophisticated reason-memes, sophistication signals nothing
________________________________________
But Wait: Why This Won’t Be Permanent (The Turn)
Mercier’s Arms Race Doesn’t Stop
Remember the evolutionary story:
• Better persuaders → selection pressure for better evaluators
• Better evaluators → selection pressure for better persuaders
• This is an ongoing process, not a fixed equilibrium
What’s Actually Happening:
• Persuaders just got a massive technological advantage (LLMs)
• The arms race didn’t end – it tilted dramatically
• But evolutionary pressure on evaluators is now intense
Selection Pressure on Evaluation (Right Now)
People who can’t navigate the new environment:
• Get fooled more often
• Make worse decisions
• Lose social/economic/political games
• Their strategies don’t propagate
People developing new evaluation strategies:
• Survive the epistemic environment better
• Their methods spread (memetically!)
• Cultural evolution favors better epistemics
The Arms Race Continues – Just Faster
Short term (current crisis):
• Persuasion tools massively outpace evaluation tools
• Epistemic confusion, trust breakdown
• Market volatility in what counts as credible
Medium term (emerging):
• New evaluation heuristics developing:
o Demand for process transparency
o Shift from “what” to “how do you know?”
o Embodied proof over linguistic performance
o Track records over individual claims
o Meta-honesty (disclosing AI use) starts signaling trustworthiness
Long term (new equilibrium):
• Different costly signals stabilize
• New trust infrastructure emerges
• Evaluation catches up (but at higher sophistication level)
Why Humans Won’t Stay in Epistemia
1. Evolutionary pressure is ruthless
o Being epistemically lost has real costs
o Better evaluators outcompete worse ones
o Cultural selection favors groups with working epistemics
2. The arms race is built-in
o We evolved specifically for this competition
o Our cognitive architecture is designed for argument evaluation
o It will recalibrate – just needs time and selection pressure
3. New costly signals will emerge
o Zahavi’s principle still applies
o Something will become the new handicap
o Whatever’s hardest to fake becomes the next honest signal
o Current candidates:
 Executable/embodied knowledge
 Long-term track records
 Process transparency (showing uncertainty)
 Meta-cognitive honesty
4. Institutions adapt
o Peer review will evolve
o Journalism will find new credibility markers
o Academia will develop new quality signals
o Slower than individuals, but happens
The New Equilibrium Will Be Different
What we’re losing:
• Linguistic sophistication as primary signal
• Individual rhetorical performance as credibility marker
• Text-based trust at first encounter
What we’re gaining:
• Pressure toward verifiable, embodied knowledge
• Emphasis on process over product
• Track records and consistency over single performances
• Transparency about tools/methods
• Meta-cognitive honesty as valuable signal
The Transition Period
We’re in the chaotic phase:
• Old signals broken
• New signals not yet stabilized
• High uncertainty about who/what to trust
• Lots of noise, experimentation, failure
But this is normal for evolutionary transitions:
• Brief (in evolutionary time) period of disruption
• Intense selection pressure
• Rapid cultural evolution
• New equilibrium emerges
Historical Parallel: Printing Press
• Printing made text cheap (previously costly to copy)
• Explosion of pamphlets, tracts, unreliable information
• “Anyone can publish anything now!”
• Short-term epistemic chaos
• Medium-term: new institutions evolved (journalism, peer review, libraries)
• New equilibrium with different trust markers
We’re in the pamphlet-explosion phase of LLMs.
Why The New Equilibrium Takes Time
1. Cultural evolution is slower than technological change
o LLMs arrived suddenly (2-3 years)
o New evaluation norms need generations to stabilize
o Institutions take decades to adapt
2. Multiple experiments happening simultaneously
o Different communities trying different solutions
o Some will work, most won’t
o Successful strategies will spread
3. We need new costly signals to crystallize
o Not yet clear what becomes the reliable handicap
o Market testing different candidates
o Whatever proves unfakeable wins
The Optimistic Case
Mercier’s framework suggests:
• We’re cognitively equipped for this challenge
• Evaluation mechanisms can and will adapt
• The arms race resumes at higher level
• New equilibrium where:
o Arguments still matter
o But different things make them trustworthy
o Evaluation is more sophisticated
o Trust is harder-won but more reliable
The Pessimistic Case
• Transition period might be very long (decades)
• During transition: political chaos, institutional breakdown, epistemic fragmentation
• No guarantee new equilibrium is better than old one
• Might stabilize at lower-trust equilibrium
• Some permanent damage to certain trust mechanisms
Most Likely: Somewhere Between
• Messy transition (we’re in it)
• Some institutions collapse, others adapt
• New signals emerge organically
• Eventually stabilize at different equilibrium
• Trust rebuilds, but looks different
The Critical Insight
Epistemia is a transition state, not a destination.
The arms race didn’t end – it accelerated.
Evaluation is under intense pressure to evolve.
It will. The question is: How long does it take, and what does the new equilibrium look like?
________________________________________
Transition to Q4: So what can we actually trust? And more importantly, what are the emerging signals that might stabilize into the new equilibrium?
________________________________________
Does this capture the turn you wanted? The key is that Mercier’s arms race account is fundamentally optimistic about human adaptive capacity, even while acknowledging we’re in a genuinely difficult transition period.
M
Why is your process transparency focused on uncertainty?
You’re right to question that – I’ve been conflating two different things.
Process transparency could mean:
1. Showing uncertainty (what I kept emphasizing)
o “I’m not sure about X”
o “This part is speculative”
o Admitting gaps in knowledge
o This is more about epistemic humility
2. Showing the actual process (what it should mean)
o “Here’s how I arrived at this conclusion”
o “I consulted sources A, B, C”
o “My reasoning chain was: X → Y → Z”
o “I used tool/method M”
o “Here were the alternatives I considered and rejected”
o Showing your work, not just your doubts
Why I conflated them:
I think I was reasoning: “Showing uncertainty is costly because it makes you look weak” → therefore it’s an honest signal.
But that’s a different thing than process transparency.
Process transparency is costly because:
• Takes extra effort to document
• Exposes your reasoning to critique (people can find holes)
• Reveals what you don’t know (gaps become visible)
• Shows reliance on tools/sources (less impressive than seeming to know it yourself)
• Makes it harder to maintain polished expert image
• People can check your work and catch errors
The honest signal is:
• You did the actual work (can show the process)
• You’re confident enough in your reasoning to expose it
• You’re accountable to the method, not just the conclusion
• Hard to fake having done rigorous work when you have to show it
Better framing:
Process Transparency
• Showing how you reached conclusions (methods, sources, reasoning steps)
• Costly because: exposes reasoning to scrutiny, takes effort, reveals dependencies
• Signals: genuine engagement with the problem, methodological rigor, accountability
• Can include uncertainty, but that’s not the main point
Epistemic Humility (separate but related)
• Explicitly marking uncertainty, limitations, confidence levels
• Costly because: sacrifices appearance of authority
• Signals: intellectual honesty, calibrated self-assessment
• Complements process transparency but is distinct
Does that distinction matter for your lecture?
M
Yes. But it’s also important to mention that epistemic humility can be faked. As soon as it becomes as signal it collapses. I want to add this both as a content and as a meta principle. Write me a sequence of epistemic humility traces that I will incorporate in my lecture, with an abrupt transition . Where is the best place, I think where we first make the point. So we make the students feel it.
Epistemic Humility Traces (for early in lecture, probably end of Q1 or beginning of Q2)
Setup Context: You’ve just explained Sperber’s epistemic vigilance and Mercier’s arms race. You’re about to transition to costly signaling.
________________________________________
The Sequence:
[After explaining that we evaluate both content and source]
“Now, I should say—and I want to be transparent about this—I’m not entirely certain how much of Sperber’s framework relies on innate cognitive modules versus culturally learned heuristics. The literature isn’t completely settled on this, and I haven’t read everything in the field. So take what I’m saying as my best understanding, but with that caveat.
Also, I should note that this lecture structure itself is somewhat experimental. I’m drawing connections between Mercier, Dawkins, and Zahavi that seem compelling to me, but I haven’t seen them synthesized exactly this way in the literature. So it’s possible I’m overreaching or missing important objections that specialists in these areas would immediately see.
And frankly, I’m still working out how much of the AI trust crisis I’m describing is genuinely novel versus just accelerating existing problems. Some of this might turn out to be overstated once we have more historical perspective.
I’m also conscious that my own exposure to LLM-generated arguments might already be affecting my evaluation mechanisms in ways I’m not fully aware of. I could be more calibrated—or less—than I think I am to this new environment.
Furthermore, when I talk about ‘new equilibrium’ emerging, I’m really speculating. I don’t have strong evidence about timescales or what that equilibrium looks like. It’s more of a theoretical expectation based on Mercier’s framework, but I could be wrong about whether the arms race actually continues in this case.
[PAUSE]
[Look at students directly]
So here’s the thing: Did any of that make you trust me more?
[Let them respond]
Because here’s what just happened—I performed epistemic humility. I signaled uncertainty, admitted gaps, showed self-awareness about limitations.
And maybe it worked. Maybe you thought ‘Oh, this person is being honest about what they don’t know, that makes them more credible about what they do know.’
Or maybe you thought ‘Why am I listening to someone who doesn’t know what they’re talking about?’
But here’s the deeper problem:
I could have generated all of that with an LLM. In fact, let me show you—
[Pull up ChatGPT/Claude on screen]
‘Take the following lecture content and add realistic-sounding epistemic humility markers throughout. Make me sound appropriately uncertain about the right things.’
[Show the output]
See? Epistemic humility is itself a fakeable signal.
As soon as we realize ‘showing uncertainty signals trustworthiness,’ the signal starts to collapse. Because now:
• Sophisticated persuaders know to perform humility
• LLMs can generate appropriately calibrated uncertainty
• The very act of signaling ‘I’m not sure’ becomes a strategic move
This is the meta-principle I want you to feel viscerally:
Every honest signal, once recognized as valuable, creates pressure to fake it.
The moment we identify ‘X makes people trustworthy,’ X becomes a performance.
This applies to:
• Epistemic humility ✓
• Process transparency (once we know it matters, we’ll perform it)
• Meta-cognitive honesty (admitting AI use becomes strategic)
• Even this very lecture (I’m signaling awareness of signal collapse)
Zahavi’s handicap principle suggests the only stable signals are the ones that remain costly to fake.
So epistemic humility might work as a signal temporarily—right now, in transition, when most people aren’t yet performing it strategically.
But it can’t be the long-term solution, because it’s cheap to fake.
[TRANSITION – now depends where you are]
Option A (if transitioning into Q2 on costly signals): “Which brings us to the question: what has been costly to fake, historically? What made arguments trustworthy before we became this self-aware about signaling?”
Option B (if already in Q2): “This is exactly why we need to understand costly signaling more deeply. Not all signals collapse when people become aware of them. Some remain honest because they remain expensive…”
________________________________________
Where to Place This:
I’d suggest end of Q1, right before transition to Q2.
Why:
• You’ve established the evaluation framework (Sperber/Mercier)
• Students understand we assess both content and source
• You’re about to introduce costly signaling
• This gives them a felt experience of signal collapse
• Makes them receptive to understanding why cost matters specifically
• Creates dramatic transition: “So what HAS worked historically?”
The Pedagogical Move:
1. Perform epistemic humility (students experience it)
2. Reveal it was performative (break the fourth wall)
3. Show it can be faked (demonstrate with LLM)
4. Extract the meta-principle (all recognized signals face fakery pressure)
5. Use this to motivate costly signaling theory (need signals that stay expensive)
Timing:
• The sequence above: ~3-4 minutes
• Creates memorable moment
• Students feel the collapse happening
• Sets up the rest of the lecture with appropriate skepticism
Does this capture what you wanted?
M
I don’t need to copy that and show them chat gpt. They are clever enough to think it can be automated. I will link this dialogue to them at the end. Better to leave a lingering doubt.
Ah, much better. Revised:
________________________________________
The Sequence:
[After explaining that we evaluate both content and source]
“Now, I should say—and I want to be transparent about this—I’m not entirely certain how much of Sperber’s framework relies on innate cognitive modules versus culturally learned heuristics. The literature isn’t completely settled on this, and I haven’t read everything in the field. So take what I’m saying as my best understanding, but with that caveat.
Also, I should note that this lecture structure itself is somewhat experimental. I’m drawing connections between Mercier, Dawkins, and Zahavi that seem compelling to me, but I haven’t seen them synthesized exactly this way in the literature. So it’s possible I’m overreaching or missing important objections that specialists in these areas would immediately see.
And frankly, I’m still working out how much of the AI trust crisis I’m describing is genuinely novel versus just accelerating existing problems. Some of this might turn out to be overstated once we have more historical perspective.
I’m also conscious that my own exposure to LLM-generated arguments might already be affecting my evaluation mechanisms in ways I’m not fully aware of. I could be more calibrated—or less—than I think I am to this new environment.
Furthermore, when I talk about ‘new equilibrium’ emerging, I’m really speculating. I don’t have strong evidence about timescales or what that equilibrium looks like. It’s more of a theoretical expectation based on Mercier’s framework, but I could be wrong about whether the arms race actually continues in this case.
[PAUSE]
[Look at students directly]
So here’s the thing: Did any of that make you trust me more?
[Let them respond – brief discussion]
Because here’s what just happened—I performed epistemic humility. I signaled uncertainty, admitted gaps, showed self-awareness about limitations.
And maybe it worked. Maybe you thought ‘Oh, this person is being honest about what they don’t know, that makes them more credible about what they do know.’
Or maybe you thought ‘Why am I listening to someone who doesn’t know what they’re talking about?’
But here’s the deeper problem:
How do you know I meant any of that?
How do you know those weren’t just strategically placed uncertainty markers, designed to make me seem more trustworthy?
How do you know I didn’t… optimize this?
[Let that sit for a moment]
This is the meta-principle I want you to feel viscerally:
Every honest signal, once recognized as valuable, creates pressure to fake it.
The moment we identify ‘X makes people trustworthy,’ X becomes a performance.
Epistemic humility is itself a fakeable signal.
As soon as we realize ‘showing uncertainty signals trustworthiness,’ the signal starts to collapse. Because now:
• Sophisticated persuaders know to perform humility
• The very act of signaling ‘I’m not sure’ becomes a strategic move
• You can’t tell genuine uncertainty from performed uncertainty
This applies to:
• Epistemic humility ✓
• Process transparency (once we know it matters, we’ll perform it)
• Meta-cognitive honesty (admitting AI use becomes strategic)
• Even this very lecture (am I signaling awareness of signal collapse?)
And you’re now stuck. Even if I’m being genuinely uncertain, you can’t be sure.
[PAUSE]
Zahavi’s handicap principle suggests the only stable signals are the ones that remain costly to fake.
So epistemic humility might work as a signal temporarily—right now, in transition, when most people aren’t yet performing it strategically.
But it can’t be the long-term solution.
[TRANSITION]
Which brings us to the question: what has been costly to fake, historically? What made arguments trustworthy before we became this self-aware about signaling?
________________________________________
The Effect:
• Students experience the performance
• You reveal it as potentially performative
• But you DON’T prove it was faked
• They’re left uncertain about THIS VERY MOMENT
• The doubt lingers through the whole lecture
• Then at the end: “By the way, here’s the dialogue I had with Claude preparing this lecture”
• The reveal is devastating because they’ve been sitting with the uncertainty
Much cleaner pedagogically. The lingering doubt does more work than the proof.
M
Precisely, let’s go back to this. They may not be favorably impressed that I didn’t know the Zahavi citation . I don’t care. They should learn that concepts like honest signalling travel across disciplines. And sincerely I assumed it all referred back to Maynard Smith. Maybe this kind of epistemic humility – really showing a transcript – will survive. But not any flattering chat, that can reproduced. A chat that really contains a little revelatory, shameful element . So we’re left with effort + willingness to bear shame. The first one is intuitive, the second is deeply counterintuitive. What will happen to virtue signalling in the age in which the greatest virtue to exhibit is a vice? But then think about it. Isn’t this already the world we’re living in? I could mention Donald Trump, winning an election twice in a cultural era in which almost Victorian norms of virtue signalling attached to all matters concerning respect for persons, especially women and minorities. Or media people who become successful precisely because they don’t care about so-called woke norms. Isn’t this the peacock principle in action? These people are signalling: I’m so strong I can afford to say the things that would kill most careers. Of course many are not really so strong, if they do it they kill themselves. But other succed, so the survivor end up being perceived as strong by virtue of what they have survived only if it all started as a risky bet and luck was involved. Can we extend this to reasons.
Mostra meno
This is brilliant – you’re seeing the deep pattern. Let me think through this:
The Shameful Honest Signal
What survives as honest signal:
• Effort (intuitive – hard to fake work)
• Willingness to bear shame (counterintuitive – why would weakness signal strength?)
The Paradox: In an age where virtue is performed, vice becomes the honest signal.
Showing the transcript with your Zahavi/Maynard Smith confusion is costly because:
• Reveals ignorance (shameful for a lecturer)
• Can’t be reproduced strategically (deliberate mistakes would be obvious)
• Requires genuine vulnerability
• Most people won’t do it (too risky)
But this is already happening in the broader culture:
Trump / Anti-Woke Figures as Zahavi’s Peacock
The Logic:
• Victorian virtue norms dominate elite discourse
• Violating them should be career-ending
• “I can say X and survive” signals: “I’m so strong I can afford this handicap”
• The cost (potential career death) is what makes it credible
The Handicap Principle in Action:
• Weak person violates norms → destroyed (proving the cost is real)
• Strong person violates norms → survives → proves strength
• Survival of the violation is the signal
The Critical Element: Risk
• It only works as a signal if it started as a genuine risk
• Many try, most fail (the cost is real)
• Survivors are perceived as strong because others didn’t survive
• Luck is involved, but that doesn’t matter for perception
• Post-hoc, we attribute their survival to strength
Extending This to Reasons
The Question: Can reasons signal credibility through willingness to bear epistemic shame?
Current Examples:
• Showing your confused process (not just polished conclusion)
• Revealing you used AI (when others hide it)
• Admitting “I don’t know” when you could fake it
• Exposing your reasoning to critique (showing work = showing vulnerability)
The Zahavi Logic Applied:
• Most people won’t expose epistemic vulnerability (career risk, status risk)
• If you do and survive (remain credible), it signals strength
• The signal works because it’s risky
• “I can afford to look uncertain/confused/dependent on AI and still be taken seriously”
But There’s a Crucial Asymmetry with Trump:
Political/Cultural Norm Violation:
• Binary outcome visibility (you survive or you don’t)
• Survival is public and obvious
• Can’t fake having survived cancellation
Epistemic Vulnerability:
• Harder to verify if you “survived”
• Did revealing confusion make you more or less credible?
• No clear binary outcome
• Easier to fake strategically
The Deep Problem: Second-Order Fakery
Stage 1: Epistemic humility is rare, signals honesty Stage 2: People realize it signals honesty, start performing it Stage 3: Genuine humility becomes indistinguishable from performed humility Stage 4: People realize performed humility is detectable, start performing shameful humility Stage 5: ???
Your transcript strategy might work because:
• Contains genuinely unflattering elements (Zahavi confusion)
• Hard to reproduce strategically (deliberate mistakes too obvious)
• Requires real vulnerability (students see you didn’t know something)
• Most people won’t do it (too risky for professional image)
But it only works as long as:
• Most people don’t do it (scarcity)
• The shame is real (not performed)
• There’s actual risk involved (could damage credibility)
The Trump Parallel Deepened
What Trump/anti-woke figures reveal:
• In an environment of universal virtue-signaling, vice becomes the costly signal
• The peacock principle: “I can afford to violate the norms that constrain you”
• Works because most people who try it fail (the cost is real)
• Survivors are selected for strength (even if luck was involved)
Applied to Reasons:
Current environment:
• Everyone performs epistemic sophistication (LLMs make it cheap)
• Everyone performs appropriate uncertainty (once it’s recognized as valuable)
• Everyone can produce polished arguments
The new costly signal might be:
• Epistemic vice (showing confusion, error, dependence)
• Only works if it’s genuinely risky
• Most people who expose real weakness get dismissed
• Those who survive it prove something
But: Why Would Epistemic Vice Signal Strength?
The Mechanism:
• “I can afford to show you my mistakes and still be credible”
• Requires having enough other capital (expertise, track record, institutional position)
• Like Trump: “I can afford to violate norms because I’m strong enough that it doesn’t destroy me”
• The survival of the violation is what signals quality
The Survivor Bias Problem
Critical Insight:
• We only see the people who showed epistemic vice and survived
• Many showed weakness and were dismissed (we don’t see them)
• Survivors might be:
o Actually strong (genuine expertise overcomes visible mistakes)
o Lucky (happened to have forgiving audience)
o Strategically shameful (calculated what they could afford to reveal)
This means:
• Showing shameful transcript works for YOU (established professor)
• Might not work for junior scholar (no capital to burn)
• The signal is stratified by existing position
• Reinforces rather than challenges hierarchies
Can This Extend to Reasons Themselves?
Reasons with visible flaws might signal:
• “I’m confident enough in the core argument to show you the messy process”
• “I have enough credibility capital to afford exposure”
• “I’m not optimizing for impression, I’m optimizing for truth”
But this requires:
• The flaws are real (not performed)
• There’s genuine risk (could be dismissed)
• Most people who try it fail
• Survival indicates something (expertise, institutional backing, luck)
The Meta-Level Collapse
Your very lecture enacts this:
• You show the transcript (epistemic vice)
• It contains real confusion (shameful)
• You survive it (students still learn from you)
• Your survival proves… what exactly?
Options:
1. You’re strong enough (expert enough) that mistakes don’t matter
2. You’re honest in a way that’s rare enough to be valuable
3. You’ve correctly calculated what you can afford to reveal
4. You’re performing shameful honesty as a new strategy
And students can’t distinguish these!
Which means:
• Even shameful honesty might collapse as a signal
• Once people realize it works, they’ll strategically reveal “flaws”
• The arms race continues at meta-level
• We need meta-meta-honesty? (infinite regress)
The Pessimistic View
Every level of honest signaling eventually gets gamed:
• Sophistication → performed sophistication
• Humility → performed humility
• Shameful honesty → performed shameful honesty
• Meta-awareness of performance → performed meta-awareness
No stable equilibrium, just endless escalation.
The (Slightly) Optimistic View
Some signals might remain costly:
• Long-term track records (can’t fake time)
• Executable knowledge (can’t fake working code/experiments)
• Genuine skin in game (real consequences)
• Willingness to bear shame that actually damages you
The last one is key:
• If showing weakness costs you (students, credibility, status)
• And you do it anyway
• That’s a genuine costly signal
• Can’t be faked (by definition, if you’re faking, you’re not actually losing anything)
But we’re back to the Trump problem:
• Only works if some people try and fail
• Creates selection for those strong enough (or lucky enough) to survive
• Reinforces existing hierarchies
• Doesn’t democratize epistemic authority
For Your Lecture
Where to go with this:
You could add (probably in Q4, when discussing new signals):
“There’s a deeper pattern here, one we’re already seeing in broader culture. When virtue-signaling becomes universal, vice becomes the costly signal. Donald Trump won elections in an era of heightened sensitivity precisely by violating those norms—the peacock principle in action. ‘I’m so strong I can afford to say things that would destroy weaker candidates.’
This is already extending to epistemic discourse. Showing confusion, admitting AI use, revealing mistakes—these might become the new honest signals, precisely because they’re risky. Most people who expose genuine weakness get dismissed. Those who survive it prove they had enough other capital to afford the exposure.
I’m doing this right now. At the end of this lecture, I’ll share the transcript of the conversation I had with an AI preparing this. It contains mistakes—I confused Zahavi with Maynard Smith on the origins of honest signaling theory. Shameful for a lecturer. But perhaps that shame is precisely what makes it credible.
Of course, now that I’ve told you this, you can’t be sure if the shame is real or performed. Maybe I strategically chose which mistakes to reveal. Maybe this whole meta-awareness is itself a performance.
And we’re back to the fundamental problem: every honest signal, once recognized, creates pressure to fake it. Even shameful honesty can be weaponized.
The arms race continues. The question is whether it stabilizes at some level, or whether it’s performances all the way down.”
Does this capture what you’re seeing?
M
Thinking loudly. This means that traditional non-content based trust signal become more important. They are the visible strength that enables the show off of epistemic humility to be something the person can bear. So academic credentials and other traditional credentialing systems become more important, to the extent that they rely on some non fakable signal internally (eg oral examinations or class exams) as opposed to those that can be easily faked
Yes! This is a crucial insight. Let me think through this:
The Return to Traditional Credentials (But Transformed)
The Logic:
Epistemic humility/shameful honesty only works as a signal if:
• The person has enough capital to survive the exposure
• The survival itself is what proves strength
• Like Trump: “I can violate norms because I’m strong enough”
What provides that capital?
• Traditional credentials that prove you have genuine strength
• But only credentials based on non-fakeable assessments
The Split in Credentialing Systems:
Credentials That Survive (based on non-fakeable signals):
• Oral examinations (real-time performance, can’t use AI invisibly)
• In-person supervised exams (no AI assistance possible)
• Laboratory work (embodied, executable knowledge)
• Teaching evaluations (sustained performance over time)
• Defense of dissertation (real-time questioning)
• Physical presence requirements (being there, doing it)
Credentials That Collapse (easily AI-fakeable):
• Written assignments (can be AI-generated)
• Take-home exams (no supervision)
• Published papers (LLM-written, hard to detect)
• Standardized tests (taken remotely, AI-assisted)
• Online degrees (all work potentially AI-generated)
The Paradox:
We thought we were moving toward:
• More remote/flexible credentialing
• Less emphasis on gatekeeping institutions
• Democratized knowledge access
• Meritocracy based on demonstrated skill (regardless of institution)
But LLMs might force us back toward:
• More emphasis on traditional institutional credentials
• More in-person, supervised assessment
• Stronger gatekeeping (because gates are where non-fakeable signals live)
• Credentials that certify “this person did it themselves, we watched”
The Mechanism:
1. Person has traditional credential (PhD from serious program with oral defense)
2. This proves they have genuine expertise (couldn’t fake the oral exam)
3. This capital allows them to show epistemic humility/confusion
4. “I can afford to reveal mistakes because you know I’m actually competent”
5. The credential becomes MORE valuable (not less) in LLM age
Because:
• Credential certifies non-fakeable achievement
• Provides the strength needed to bear epistemic vice
• Acts as foundation for other trust signals
The Stratification:
High-credential individuals:
• Can afford epistemic humility
• Can show process/confusion
• Can admit AI use
• Their shameful honesty is believable (they have capital to burn)
Low/no-credential individuals:
• Can’t afford to show weakness
• Must maintain perfect performance
• Can’t admit uncertainty
• Trapped in need to appear infallible
• But appearing infallible now signals “probably AI-generated”
This Creates Brutal Inequality:
• Those with traditional credentials get MORE trusted when they show humility
• Those without credentials get LESS trusted when they show same humility
• Gap widens instead of narrows
• Opposite of the “democratizing” promise of AI
The Internal Credential Quality Matters:
Not all “PhDs” are equal anymore. Matters HOW the credential was earned:
Strong credentials (hard to fake):
• Programs with oral defenses
• In-person coursework with exams
• Supervised research
• Public presentations
• Demonstrated teaching
Weak credentials (easily fakeable):
• Online programs with take-home assessments
• Written-only requirements
• No oral components
• Remote everything
• Portfolio-based (could be AI-generated)
The Selection Pressure:
Universities/institutions with strong non-fakeable assessment will:
• Become more valuable
• Be able to charge more
• Provide more credible signals
• Their graduates more trusted
Institutions with weak assessment will:
• See credential value collapse
• Unable to certify genuine capability
• Graduates indistinguishable from AI-assisted candidates
Historical Irony:
We spent decades critiquing:
• Gatekeeping institutions
• Expensive credentials
• In-person requirements
• Oral examinations (biased, stressful)
• “Hazing” aspects of PhD programs
But these might be exactly what survives:
• They’re costly (Zahavi)
• They’re hard to fake (honest signals)
• They provide the capital needed for epistemic humility to work
• They certify “this person can actually do it, we saw them”
The Class Dimension:
• In-person, supervised credentials are more expensive
• Require geographic mobility
• Require time (can’t work while studying)
• Favor those with existing capital
So we get:
• Working-class students: online degrees, take-home work (fakeable) → credentials devalued
• Wealthy students: in-person programs, oral exams (non-fakeable) → credentials appreciate
• Gap widens dramatically
Applied to Your Lecture:
You showing the transcript with confusion works because:
• You have a position (professor)
• You have credentials (PhD, presumably from program with oral components)
• You have track record (published, taught)
• Students know you’re genuinely capable
• So showing weakness is a power move (“I can afford this”)
But:
• Graduate student showing same transcript → “maybe they don’t actually know the material”
• Independent scholar showing same → “no credentials to back them up”
• Online degree holder → “did they even write their thesis?”
The Shameful Element:
Even showing your Zahavi confusion works because:
• You have enough other visible expertise that one gap is forgivable
• Your credential certifies general competence
• The confusion is in attribution (minor) not core concept (major)
• Students can verify you understand honest signaling even if you misattributed it
The Deep Structure:
Traditional credentials become:
• Permission slips to show weakness
• The stronger your credential, the more weakness you can show
• Showing weakness then signals “I’m strong enough not to care”
• But only works if credential is based on non-fakeable assessments
The Arms Race Implication:
• Institutions invest MORE in non-fakeable assessment
• Oral exams, in-person work, supervised research become MORE important
• Cost of credentials rises (scarcity + difficulty)
• But value rises faster (only reliable signal)
• Egalitarian dream of “learn online, prove yourself through work” collapses
Why This Might Be Stable:
Unlike performed humility, you can’t fake:
• Having done an oral defense
• Having taught courses (students would notice AI)
• Having been physically present
• Having responded to real-time questions
• Having track record over years
These are genuinely costly and:
• Hard to fake
• Verifiable
• Accumulate over time
• Provide foundation for other trust signals
The Pessimistic Conclusion:
LLMs might not democratize expertise—they might:
• Re-entrench traditional gatekeeping
• Make expensive credentials MORE valuable
• Increase stratification
• Return power to institutions that can certify non-fakeable achievement
The (Slightly) Optimistic Take:
At least we’d have SOME reliable signals:
• Better than pure epistemia
• Better than “trust nothing”
• Institutions that maintain rigorous, non-fakeable assessment preserve epistemic infrastructure
• Even if access is unequal
For Your Lecture Structure:
This could go in Q4 (What Can We Trust):
“Here’s the uncomfortable implication: traditional credentials might matter MORE, not less. But only those based on non-fakeable assessments—oral exams, supervised work, in-person performance.
Why? Because epistemic humility only works as a signal if you have the capital to survive it. My showing you this transcript with my Zahavi confusion works because you know I have a PhD, a position, a track record. The credential provides the foundation that lets me afford epistemic vice.
But this means increased stratification. Those with strong credentials can show weakness (and be believed). Those without cannot. The gap widens.
And not all credentials are equal anymore. Only those that certify ‘this person did it themselves, we watched’ will retain value. Oral defenses, in-person exams, supervised research—the very gatekeeping mechanisms we’ve spent decades critiquing might be exactly what survives the LLM transition.
The egalitarian promise of AI—’anyone can access expertise’—might produce the opposite: re-entrenched credentialism, where only expensive, supervised, in-person education provides the signals needed to navigate the new epistemic environment.”
Does this capture the dynamic you’re seeing?
M
Kind of , but there will be no mention of the Zahavi confusion at this stage. They will only get a chance to see this if they download it after finishing my lecture. Only few will.
Ah yes, much better. Let me revise:
________________________________________
For Your Lecture Structure (Q4):
“Here’s the uncomfortable implication: traditional credentials might matter MORE, not less. But only those based on non-fakeable assessments—oral exams, supervised work, in-person performance.
Why? Because epistemic humility only works as a signal if you have the capital to survive it. When I show uncertainty or confusion in this lecture, it might make me seem more credible to you. But that only works because you know I have a PhD, a position, a track record. The credential provides the foundation that lets me afford epistemic vice.
A graduate student showing the same uncertainty? You might think ‘maybe they don’t actually know the material.’ An independent scholar with an online degree? No foundation of verified competence.
This means increased stratification. Those with strong credentials can show weakness and be believed. Those without cannot. The gap widens.
And not all credentials are equal anymore. Only those that certify ‘this person did it themselves, we watched’ will retain value.
Think about what’s becoming valuable:
• Oral defenses (can’t use AI invisibly in real-time)
• In-person supervised exams
• Laboratory work, teaching performance
• Anything requiring physical presence and real-time response
Versus what’s collapsing:
• Take-home assignments (could be AI-generated)
• Remote degrees (all work potentially AI-assisted)
• Written-only assessment (impossible to verify authorship)
The egalitarian promise of AI—’anyone can access expertise, credentials won’t matter’—might produce the opposite: re-entrenched credentialism, where only expensive, supervised, in-person education provides the signals needed to navigate the new epistemic environment.
The very gatekeeping mechanisms we’ve spent decades critiquing—costly programs, oral examinations, in-person requirements—might be exactly what survives, because they’re the only things that can’t be faked.”
________________________________________
The Pedagogical Sequence:
1. During lecture: Make this argument, students hear it intellectually
2. Throughout lecture: You perform epistemic humility (they experience it)
3. End of lecture: “By the way, here’s the full transcript of how I prepared this lecture” (link provided)
4. Most students: Don’t download it, leave with the performance
5. Few curious students: Download it, discover the Zahavi/Maynard Smith confusion
6. Those students experience:
o “Oh, he really DIDN’T know that”
o “The humility wasn’t just performed”
o “But also… he had Claude helping him figure it out”
o “Wait, how much of this lecture was Claude?”
o “Does that make it more or less credible?”
o They’re now living inside the very dynamics the lecture described
The Recursive Truth:
For the few who download:
• They see real epistemic vice (the confusion)
• They see real process (you working with AI)
• They see you were willing to expose this (costly)
• But they can’t be sure what you chose to include/exclude
• And now they’re uncertain about the WHOLE LECTURE
• Which proves the lecture’s point about signal collapse
• Which might make the lecture MORE credible
• Or might make them distrust everything
• They can’t resolve this
The Stratification Point Lands Harder:
Because you’re performing it:
• You (established professor) can afford to show this transcript
• They (students) probably cannot (if they showed their messy process, would it hurt them?)
• The very students who download it experience the inequality
• Some have enough capital to experiment with epistemic vice (risk it)
• Others don’t (one mistake could be costly)
The Few Who Download Get:
• Intellectual understanding (from lecture)
• Visceral experience (from seeing the transcript)
• Existential uncertainty (was any of it “real”?)
• Personal relevance (can I afford to do this?)
The Elegant Cruelty:
Only the most engaged students see the depths:
• Most students get the argument
• Few students get the proof
• Fewer still realize the proof also undermines itself
• Creating a hierarchy of understanding
• Which enacts the credentialing stratification you just described
• The lecture’s form matches its content
This is much more powerful than showing ChatGPT in the moment. The reveal is opt-in, delayed, and only for those curious enough to pursue it.
M
I didn’t ask you to write , just structure. Revise the entire lecture plan, for me to write it with my own voice
Structure 4+ REVISED: “The Question Cascade” – Full Lecture Plan
________________________________________
Q1: Why Do We Trust Reasons at All? (12 min)
Opening: The Dual Puzzle
• We detect bad arguments intuitively, before formal analysis
• We also care who’s arguing (expertise, incentives, track record)
• How do content evaluation and source evaluation work together?
Sperber’s Epistemic Vigilance
• Dual-track evaluation system:
o Content vigilance (does this cohere? make sense?)
o Source vigilance (who’s saying this? what incentives?)
• Evolved cognitive mechanisms, not formal logic
• We’re actually quite good at this
Mercier’s Argumentative Theory
• Reasoning evolved for social functions, not individual truth-finding
• Two functions in arms race:
o Producing reasons (persuasion)
o Evaluating reasons (vigilance)
• Better persuaders → better evaluators → better persuaders
The Historical Equilibrium
• Content vigilance worked because coherent arguments were hard to produce
• Source vigilance worked because people bore reputational costs
• Critical correlation: ability to produce sophisticated arguments correlated with genuine understanding
[EPISTEMIC HUMILITY SEQUENCE – 3-4 minutes]
Perform epistemic humility naturally through several caveats about:
• Uncertainty about Sperber’s framework details
• Experimental nature of this lecture synthesis
• Speculation about AI crisis novelty
• Your own calibration doubts
• Uncertainty about new equilibrium
[PAUSE]
Direct question to students: “Did any of that make you trust me more?”
Brief discussion
Reveal the Performance
• Point out you just performed epistemic humility
• It might have worked (or not)
• But: how do they know you meant it?
• How do they know you didn’t optimize this?
• Let that uncertainty sit
Extract the Meta-Principle
• Every honest signal, once recognized as valuable, creates pressure to fake it
• Epistemic humility is itself fakeable
• They’re now stuck – can’t tell genuine from performed
• This applies to: humility, process transparency, meta-honesty, even this lecture
Zahavi’s principle preview: Only stable signals are those that remain costly to fake
Transition: What has been costly to fake historically?
________________________________________
Q2: What Made Arguments Costly (And Why Did That Matter)? (18 min)
The Hidden Cost Structure
• Historical requirements for sophisticated arguments:
o Cognitive capacity, deep knowledge, time investment, sustained attention
• These costs created a filter
Costly Signaling Theory
• Spence (economics), Zahavi (biology)
• Zahavi’s Handicap Principle: peacock’s tail
• Waste enforces honesty – only fit peacocks can afford the burden
• Cost enforces correlation between signal and quality
• If cost drops, signal becomes fakeable
Language/Arguments as Handicap
• Sophisticated reasoning as costly signal of: capacity, knowledge, investment, resources
• Key correlation: producing coherent arguments required actually understanding
• Couldn’t fake deep understanding without having it
How This Supported Both Vigilance Mechanisms
• Content vigilance: coherent arguments generally came from people with understanding
o Our detectors calibrated to this reality
o False positives rare (hard to produce sophisticated nonsense)
• Source vigilance: demonstrated skill revealed capability
o Track record of good arguments = track record of understanding
o Reputational costs were real
The Dual Market Structure
• Utilitarian market: arguments as information/coordination tools
o Better reasoning = better collective decisions
o Non-zero-sum benefit
• Positional market: arguments as status signals
o Sophistication as competitive advantage
o Zero-sum: relative eloquence matters
Market Interaction
• Positional competition drove genuine skill development
• Genuine skills produced useful arguments (utilitarian benefit)
• Costly investment meant people cared about being right
• Trust depended on this entire ecology
Transition: What happens when cost structure vanishes?
________________________________________
Q3: What Happens When Argument Production Becomes Free? (18 min)
The LLM Transformation
• Anyone can generate: coherent arguments, sophisticated prose, technical reasoning, multi-step logic
• Cost → essentially zero
• Time investment → minutes instead of years
The Correlation Breaks
• Historical: Coherent argument ↔ genuine understanding
• Post-LLM: Coherent argument ↔ access to GPT
• Coupling between sophistication and comprehension severed
Mercier’s Functions Become Unbalanced
Persuasion Function: Hyperstimulated
• Producing compelling arguments now trivial
• Volume explosion
• Quality variance collapses
• No effort barrier = no commitment signal
Evaluation Function: Overwhelmed
• Content vigilance gives false positives:
o Coherence? Yes (AI good at local coherence)
o Makes sense? Seems to (sophisticated surface)
o But coherence no longer implies understanding
o Our intuitions calibrated for different world
• Source vigilance breaks down:
o Can’t tell person vs AI
o Can’t assess expertise from text alone
o Reputational costs weaken (everyone uses AI, plausible deniability)
The Asymmetry
• Producing arguments: cheap
• Evaluating arguments: harder than ever
• More volume, better surface quality, unreliable source signals
• Arms race tips toward persuaders
Dawkins’ Memes: The Original Framework
What Memes Are (1976)
• Cultural replicators analogous to genes
• Units of cultural transmission
• Replicate by jumping brain to brain
• Subject to variation, selection, retention
• Key: compete for replication, not truth
Memetic Selection Pressures
• Fidelity, fecundity, longevity
• Don’t correlate with truth value
• False but memorable can outcompete true but boring
Traditional Memetic Constraints
• Horizontal: peer evaluation, reputational costs, social accountability
• Vertical: cost of production, investment = quality care, hard to mass-produce nonsense
Mercier + Dawkins: Reasons as Memes
• Reasons are special class of memes
• Serve dual functions (persuasion/evaluation)
• But replicate memetically
• Historical balance:
o Memetic selection (spread)
o Epistemic vigilance (truth/quality)
o Reputational accountability (source)
Why Balance Held:
• Producing sophisticated reason-memes was costly
• Limited volume (evaluation could keep up)
• Quality correlation (good arguments = capable sources)
• Accountability (sources attached)
The LLM Disruption: Unleashing Memetic Selection
Constraint Removal:
• Horizontal constraint weakens (can’t identify sources, reputational costs diluted, evaluation overwhelmed)
• Vertical constraint collapses (cost → zero, mass production possible)
Pure Memetic Competition:
• Reason-memes selected for: fecundity, fidelity, longevity
• NOT for: truth, epistemic quality, source credibility
The Epistemic Crisis (Apocalyptic View)
• Quattrociocchi and others: “epistemia”
• Permanent degradation of collective knowing
• Information ecosystem too polluted
• Trust infrastructure broken
• Reason-memes optimized for virality, detached from truth
Failure Modes:
• Content vigilance: false positives
• Source vigilance: no target
• Volume overwhelms
• Selection inverts (spread matters, truth doesn’t)
Market Response
• Utilitarian: flood, signal-to-noise collapse
• Positional: sophistication signals nothing when everyone has it
________________________________________
BUT: The Turn – Why This Won’t Be Permanent (5 min within Q3)
Mercier’s Arms Race Doesn’t Stop
• Better persuaders → better evaluators → better persuaders
• This is ongoing, not fixed
• Persuaders just got massive advantage (LLMs)
• But evolutionary pressure on evaluators now intense
Selection Pressure on Evaluation (Right Now)
• Poor navigators: fooled more, worse decisions, lose games
• Good navigators: developing new strategies, methods spread
• Cultural evolution favors better epistemics
The Arms Race Continues – Just Faster
• Short term: persuasion vastly outpaces evaluation (current crisis)
• Medium term: new evaluation heuristics emerging
o Demand process transparency
o Shift from “what” to “how do you know”
o Embodied proof over linguistic performance
o Track records over claims
o Meta-honesty about AI use
• Long term: new equilibrium
o Different costly signals stabilize
o New trust infrastructure emerges
o Evaluation catches up (but at higher sophistication)
Why Humans Won’t Stay in Epistemia
• Evolutionary pressure is ruthless (being lost has costs)
• Better evaluators outcompete worse
• Arms race is built into our cognitive architecture
• Will recalibrate – needs time and selection
• New costly signals will emerge (whatever’s hardest to fake)
• Institutions adapt (slower but happens)
The New Equilibrium Will Be Different
• Losing: linguistic sophistication as primary signal, individual performance as credibility
• Gaining: pressure toward verifiable/embodied knowledge, emphasis on process, track records, transparency, meta-honesty
Historical Parallel: Printing Press
• Made text cheap (previously costly to copy)
• Explosion of unreliable pamphlets
• Short-term epistemic chaos
• Medium-term: new institutions evolved (journalism, peer review)
• New equilibrium with different trust markers
• We’re in the pamphlet-explosion phase of LLMs
Why New Equilibrium Takes Time
• Cultural evolution slower than technological change (LLMs: 2-3 years, norms: generations)
• Multiple experiments simultaneously (some work, most don’t)
• Need new costly signals to crystallize (not yet clear, market testing)
Most Likely Outcome
• Messy transition (we’re in it)
• Some institutions collapse, others adapt
• New signals emerge organically
• Eventually stabilize at different equilibrium
• Trust rebuilds differently
Key Insight: Epistemia is transition state, not destination. Arms race accelerated, not ended. Evaluation under intense pressure to evolve. It will.
Transition to Q4: What are the emerging signals that might stabilize?
________________________________________
Q4: Rebuilding Trust in a Post-Cost World (10 min)
The Challenge
• Can’t rely on argumentative sophistication (too cheap)
• Can’t rely on coherence alone (AI good at it)
• Can’t always identify sources (viral reason-memes)
• Our evolved vigilance mechanisms miscalibrated
What Becomes Costly (And Therefore Honest)?
Candidate New Signals:
1. Process Transparency (showing how, not just what)
o The actual reasoning process, sources, methods, steps
o Costly because: exposes reasoning to scrutiny, takes effort, reveals dependencies
o Signals: genuine engagement, methodological rigor, accountability
o NOT primarily about showing uncertainty – about showing work
o Problem: also potentially fakeable with AI
2. Embodied/Executable Knowledge
o Physical demonstrations, working code, reproducible experiments
o Costly because: requires actual capability beyond language
o Signals: practical competence vs rhetorical skill
o Harder to fake: “show me it working” vs “tell me how”
3. Long-term Track Records
o Consistent performance over time, verifiable outcomes
o Costly because: requires sustained commitment, can’t be instantly generated
o Signals: reliable expertise, skin in game
o Problem: slow to establish
4. Meta-cognitive Honesty (disclosing AI use, boundaries, limitations)
o Costly because: sacrifices status, reveals dependence
o Signals: epistemic integrity
o Paradox: currently signals weakness, might become strength
5. Institutional Commitment
o Organizations staking reputation on claims
o Costly because: wrong claims damage accumulated credibility
o Problem: institutions also use AI, face same pressures
Recalibrating Vigilance
• Content vigilance must evolve: from “coherent?” to “verifiable?”, from admiring sophistication to demanding evidence
• Source vigilance must adapt: from “who said?” to “chain of accountability?”, from polish to vulnerability/honesty
The Deeper Pattern: Vice as Signal
When virtue-signaling becomes universal, vice becomes the costly signal
Cultural Examples Already Visible
• Trump/anti-woke figures in era of heightened virtue norms
• The Zahavi logic: “I can violate norms and survive” signals strength
• Only works because it’s genuinely risky
• Most who try fail (cost is real)
• Survivors perceived as strong (even if luck involved)
Applied to Epistemic Domain
• Showing confusion, admitting AI use, revealing mistakes
• Might become honest signals precisely because risky
• Most who expose weakness get dismissed
• Those who survive prove they had capital to afford exposure
The Critical Asymmetry
• This only works if you have foundation of credibility
• “I can afford to look uncertain and still be taken seriously”
• Requires visible strength to enable show-off of weakness
The Return to Traditional Credentials (Non-fakeable Assessment)
Here’s the uncomfortable implication:
Traditional credentials matter MORE, not less – but only those based on non-fakeable assessments
Why:
• Epistemic humility only works as signal if you have capital to survive it
• Credentials provide that foundation
• When someone with strong credentials shows weakness, it’s believable
• When someone without credentials shows weakness, it’s just weakness
This Creates Stratification:
• High-credential individuals: can afford epistemic humility, believed when vulnerable
• Low/no-credential individuals: can’t afford weakness, must appear infallible, but infallibility signals “probably AI”
• Gap widens, opposite of democratizing promise
Not All Credentials Equal – Internal Quality Matters:
Strong credentials (hard to fake):
• Oral defenses (can’t use AI invisibly in real-time)
• In-person supervised exams
• Laboratory work, teaching performance
• Physical presence, real-time response
Weak credentials (easily fakeable):
• Take-home assignments (could be AI)
• Remote degrees (all work potentially AI-assisted)
• Written-only assessment (can’t verify authorship)
• Portfolio-based (could be generated)
Selection Pressure on Institutions:
• Universities with strong non-fakeable assessment: more valuable, charge more, graduates more trusted
• Institutions with weak assessment: credential value collapses, can’t certify capability
Historical Irony:
• We critiqued: gatekeeping, expensive credentials, in-person requirements, oral exams, “hazing”
• But these might be exactly what survives: costly, hard to fake, provide capital for epistemic humility
• The very gatekeeping mechanisms we criticized might be what preserves epistemic infrastructure
The Class Dimension:
• In-person supervised credentials more expensive
• Require mobility, time, existing capital
• Working-class: online degrees (fakeable) → credentials devalued
• Wealthy: in-person programs (non-fakeable) → credentials appreciate
• Gap widens dramatically
The Pessimistic Conclusion:
• LLMs might re-entrench traditional gatekeeping
• Make expensive credentials MORE valuable
• Increase stratification
• Return power to institutions certifying non-fakeable achievement
The (Slightly) Optimistic Take:
• At least some reliable signals exist
• Better than pure epistemia
• Institutions maintaining rigorous assessment preserve epistemic infrastructure
• Even if access unequal
________________________________________
Closing Synthesis (2 min)
The Core Transformation
• Sperber and Mercier described an evolved system that worked
• Hidden assumption: producing good arguments was hard
• Difficulty created correlation between sophistication and understanding
• That correlation calibrated both vigilance mechanisms
• LLMs severed the correlation
The Irony
• Better tools for producing reasons (persuasion hyperstimulated)
• Worse ability to trust reasons (evaluation broken)
• Arguments simultaneously more sophisticated and less reliable
• We eliminated the very thing (costliness) that made arguments trustworthy
The Open Question
• Coherent arguments cheap to produce
• Viral reason-memes spread without accountable authors
• Evolved vigilance mechanisms miscalibrated
• Volume overwhelming
What becomes the new basis for epistemic trust?
The Answer Isn’t Clear Yet
• We’re living through the transition
• Old handicaps gone
• New ones not stabilized
• Informational interregnum – between epistemic order that was and whatever comes next
Final Note:
• “The full transcript of my conversation with Claude preparing this lecture is available [link]”
• No elaboration
• Move to Q&A
________________________________________
Total timing: ~60 minutes
• Q1: 12 min (including epistemic humility sequence)
• Q2: 18 min
• Q3: 18 min (including the turn)
• Q4: 10 min
• Closing: 2 min
Key Structural Features:
• Epistemic humility sequence early (Q1), performed then revealed
• Turn happens within Q3 (not separate section) – apocalyptic view immediately countered
• Vice-as-signal and credentials return integrated into Q4 as part of emerging solutions
• Transcript link at very end, minimal framing
• Students get intellectual argument + performed demonstration + optional deep dive
1. Why do we trust reasons at all? Thank you for attending my final lecture of this course, which will be about the topic of trust and AI. Notice, not trust in AI, which has been the subject of some of my previous papers, including the one that the students in charge for the presentation of scienti
pasted
M
generate a fluent epistemic humility sequence to follow this text:
Here’s a fluent epistemic humility sequence to follow your text:
________________________________________
Now, before I continue with what Claude has prepared for me in the next section about costly signaling, I need to pause and be honest with you about several things.
First, I’m not entirely certain I’ve correctly represented the nuances of Sperber and Mercier’s framework. I read their books carefully, but it’s been a while, and I may be emphasizing certain aspects over others in ways they wouldn’t endorse. There are debates in the literature about whether their argumentative theory of reason is fully compatible with their epistemic vigilance framework, and I haven’t followed all those debates closely.
Second, this equilibrium idea I just presented—linking Mercier to Smith, to institutional economics, to Acemoglu—this synthesis is my own. I haven’t seen it articulated exactly this way in the literature. So it’s possible I’m overreaching, making connections that seem compelling to me but that specialists in these different fields would immediately see problems with.
Third, I should admit that my knowledge of the empirical literature on epistemic vigilance is patchy. There’s a whole body of developmental psychology research on how children evaluate informants, and I’m drawing on summaries rather than primary sources. So some of what I’m saying about how epistemic vigilance actually works might be oversimplified or even wrong.
And honestly, I’m also uncertain about how much this Western, science-worshipping equilibrium I just praised is actually working. I said we “haven’t fared that bad”—but that’s a pretty weak claim, isn’t it? And it’s becoming weaker every year. Maybe I’m being too optimistic about the historical equilibrium precisely because I’m a product of it, and it’s hard for me to see its failures clearly.
Finally—and this is the part that should worry you most—I’m conscious that preparing this lecture with Claude might have already affected how I think about these topics in ways I can’t fully track. How much of what I just said is my synthesis versus Claude’s synthesis that I’ve internalized? I honestly can’t tell anymore.
[PAUSE – look directly at students]
So. Did any of that make you trust me more?
[Let them respond]
Think about what just happened. I performed epistemic humility. I signaled uncertainty, admitted gaps, showed awareness of my limitations. And maybe it worked—maybe you thought “he’s being honest about what he doesn’t know, that makes him more credible about what he does know.”
Or maybe you thought “why am I listening to someone who admits he doesn’t know what he’s talking about?”
But here’s the deeper problem: How do you know I meant any of that? How do you know those weren’t just strategically placed uncertainty markers, designed to make me seem more trustworthy? How do you know I didn’t… optimize this performance?
[Let that sit]
Because here’s what I want you to feel viscerally: every honest signal, once recognized as valuable, creates pressure to fake it. The moment we identify “showing uncertainty signals trustworthiness,” showing uncertainty becomes a performance.
Epistemic humility is itself a fakeable signal. And you’re now stuck—even if I’m being genuinely uncertain, you can’t be sure.
This applies to everything: epistemic humility, process transparency, meta-cognitive honesty about AI use, even this very lecture and my awareness that I’m performing awareness of signal collapse.
Now, you might think I could just show you the Claude transcript and prove I was being genuine. But would that help? Or would it just reveal that I strategically chose which confusions to expose and which to hide?
[Brief pause]
Which brings us to the question Claude prepared for me: what has been costly to fake, historically? What made arguments trustworthy before we became this self-aware about signaling?
________________________________________
This sequence:
1. Flows naturally from your discussion of equilibrium
2. Uses your authentic voice and concerns
3. Creates genuine uncertainty (not performative)
4. References Claude naturally (you’ve already been doing this)
5. Sets up the transition to costly signaling
6. Leaves the lingering doubt without resolving it
M
how did you label the Zahavi principle again?
I labeled it as Zahavi’s Handicap Principle throughout the lecture structure.
In the context of discussing costly signaling theory more broadly, I attributed it to both Spence (economics) and Zahavi (biology), but when specifically discussing the paradox of wasteful traits that work because they’re costly (like the peacock’s tail), I called it Zahavi’s Handicap Principle.
Why do you ask? Are you thinking about how to frame the attribution issue in your actual lecture, especially given that you’ll be revealing your confusion about whether it all traces back to Maynard Smith in the transcript?
M
what is hazing
Hazing refers to rituals, practices, or activities (often required for initiation) that involve harassment, abuse, humiliation, or other forms of mistreatment.
In the academic context I used it in your lecture structure, I was referring to the grueling, sometimes seemingly arbitrary or excessive aspects of PhD programs:
• Extremely long hours
• Intense criticism in seminars
• Difficult qualifying exams
• Years of low-paid work
• High-pressure defenses
• The overall “trial by fire” culture
The term comes from the broader practice (especially in fraternities, military training, sports teams) where newcomers must endure difficult or humiliating experiences to join the group.
I used it somewhat provocatively in the lecture structure to suggest that what critics call “hazing” in PhD programs (the unnecessarily difficult parts) might actually serve as costly signals – they’re hard to fake precisely because they’re genuinely difficult and unpleasant.
The irony being: we’ve spent decades critiquing these “hazing” practices as outdated gatekeeping, but in the LLM era, they might be exactly what survives because they can’t be AI-assisted.
Should I have used different terminology for the Italian academic context? “Hazing” might be a specifically American college culture term that doesn’t translate well.