[Picture generated by Chat GPT 5.1 thinking based on the contents of the talk. Transparency declaration on the use of AI: Claude Sonnet 4.5 for planning (and some headings/bulletpoint text that survived in the final text, visible here), ML for first draft, Chat GPT 5.1 for polishing (typos, reference format, etc). Bold=linking and summary paragraphs suggested and written entirely by Chat GPT. Personality? Mine.]

1. Why do we trust reason at all?

1. Thank you for attending my final lecture of this course, which will be about the topic of trust and AI. Notice, not trust in AI, which has been the subject of some of my previous papers, including the one that the students in charge of the presentation today will present, but trust and AI. Why trust and AI? Because I believe that the strongest impact of LLM-driven conversational AI will be on human trust and trust infrastructure, so that’s really the most important question to discuss about trust right now.

2. Here is the whole story in one line: for most of human history, producing good arguments was hard, and that hardness made arguments trustworthy. Large language models make argument production cheap, which blows up that old equilibrium. Today I want to trace how human reason evolved as a social device, producing convergence around truth as a beneficial side effect of less noble aspirations — and then ask what happens when the institutional conditions that favored that convergence are eroded.

3. I have already mentioned to you the two books that have shaped most of my reasoning about this topic. They are The Enigma of Reason by Hugo Mercier and Dan Sperber and Not Born Yesterday by Hugo Mercier. I confess that although I spent three years, around the time of my PhD, studying the literature on reasons in analytic philosophy, I was kind of shocked to realize how the “reason” discussion had evolved outside of the strict confines of analytic philosophy — how what had started as a very abstract discussion in the hands of philosophers of science or action, like Jonathan Dancy, evolved in the hands of more empirically grounded cognitive scientists like Mercier and Sperber.

4. And indeed, as I read Mercier and Sperber, I also noticed a connection with another article that I remembered reading many years ago, and that I had to ask ChatGPT to help me find.

5. Let me tell you about this, since it’s quite telling of the advantages and disadvantages of the tool. I was too lazy to even check the plausibility of my memories, so my question was full of stupid mistakes. This forced it to search more broadly, and that’s how I got to one more relevant source: a paper by Paul R. Smart, a philosopher, published in the very high-quality philosophy journal Synthese, that — based on both the abstract and GPT’s synopsis — expresses a very similar idea. Smart’s Synthese paper has a fantastic title: “Mandevillian Intelligence” (Synthese 195(9): 4169–4200, 2018). Even mistakes are helpful sometimes! I will explain shortly why this title is so apt.

6. So now we have: Mercier and Sperber (my main references); Fabio Paglieri’s paper (where I first encountered this idea), “A plea for ecological argument technologies” (Philosophy & Technology, 30(2), 209–238, 2017); and now this paper by Smart, with the apt title “Mandevillian Intelligence”, roughly converging on the same idea (that I will shortly present). My lecture will be based primarily on Mercier and Sperber, whom I’ve read more recently and who arguably presented the most comprehensive conceptual framework. But I will begin with a reflection on the title of Smart’s essay, that capture’s Mercier and Sperber’s view too.

7. I will not get into too much details about Paglieri’s paper. I recommend you read that too, if you are interested in social media disinformation more broadly, but this is not directly relevant to my focus on today’s lecture on trust and AI. Still, as an introduction to the theme of Mandevillian Intelligence, I want to briefly introduce a contrast between Paglieri’s research on disinformation and that of another famous Italian researcher, Walter Quattrociocchi. In this lecture, I will refer a few times to Quattrociocchi’s idea of epistemia. Quattrociocchi, like Paglieri, wrote (more precisely, figures as co-author of) some influential papers on web disinformation. Quattrociocchi’s (group) studies are really important to assess the empirical evidence: how do people react when they encounter disinformation? Paglieri’s paper is, in my view, more interesting from a philosophical point of view. It deals with the same topic (disinformation and how hard it is for people to change their mind), but it represents a very different philosophical, political and ethical attitude towards it. Using Umberto Eco’s famous phrase, we may say that the difference between Quattrociocchi and Paglieri is the one between “Apocalittici e Integrati” — the apocalyptic and the integrated. Quattrociocchi focuses on what goes wrong with disinformation and studies people’s resistance to correction. Paglieri reflects on how resistance to correction is a regular, and perhaps not so problematic, feature of the human cognitive apparatus, and suggests a way to work with this, rather than fight it.

8. This contrast between “apocalittici and integrati” is also salient because Quattrociocchi has a very outspoken public posture on AI — you will notice if you follow him on LinkedIn. With regards to the impact of LLMs on the public sphere, he uses the expression “epistemia” to refer to yet another apocalyptic idea: the risk that most other people (not himself, I guess) will be fooled by AI (we’ll get into more details in part 2 of this lecture). I’m vastly closer to Paglieri’s position: we are both “integrati”. You should not be surprised about that; remember the title of Mercier’s book that inspires me: Not Born Yesterday. This is the quintessential contrarian book about the idea – common among most intellectuals I know – that individuals are easily fooled (of course, except intellectuals). I side with Mercier: I don’t believe that’s true. Average people are not easily fooled, when it matters to them (and intellectuals are as easily fooled as others, in things that don’t matter to them). And I side with Paglieri: human reason isn’t perfect, in some Platonic sense, but it still serves us quite well given the reality of our human needs. This is, in today’s AI debate, the stance of “integrati” in Eco’s vocabulary.

9. So I think it’s important to be very explicit about one’s priors. These inner “feelings” or beliefs shape the way we do science. As one student observed last week, they make us prey to confirmation bias. But, if you are convinced by the sort of arguments that I, Paglieri, Mercier, Sperber, and Smart find plausible, you’ll agree with me eventually that this issue (of confirmation bias) is not such a big problem in our overall information ecology.

10. So, let’s return to the primary thread of part 1: confirmation bias and how to view it from the perspective of Mandevillian Intelligence. Why “Mandevillian”, then? First of all, who’s heard this name before? And which expression does this name attach to? It’s actually a very famous expression, a viral one, a cultural meme before the internet era. (We will also speak about what memes are in this lecture, and why they existed before social media.)

11. Mandeville wrote The Fable of the Bees: or, Private Vices, Publick Benefits, a poem, in the early eighteenth century. The main idea of this poem bears some resemblance to Adam Smith’s idea that in complex, large societies, public benefits are not produced mainly by individuals in the pursuit of moral goals. But while Adam Smith’s idea is to trace the production of public benefits back to individuals’ pursuit of private interest, Mandeville went further. The expression in the title — “Private Vices, Public Benefits” — just stuck in our shared Western cultural canon. Even those who — like me — haven’t read his poem remember this expression. Mandeville’s idea was — probably not really convincing as he put it — that a lot of public vices are good, for example because they generate jobs for people in charge of correcting them. It’s only thanks to thieves that policemen have a job. I’m not even sure whether Mandeville meant this as a serious argument. If I remember well, prof. Francesca Pongiglione, who is an expert on the topic, told me he actually was ironic. If you read her books on the subject, you can check.

12. Now, whether Mandeville really meant the private-vices-public-virtues argument is not really important. The point is that you remember the concept: what is a vice at the individual level can be a virtue at the social level. We can apply this to epistemology: confirmation bias, which lowers the epistemic value of individual belief formation and deliberation, can be beneficial for the group containing many individuals with this bias.

13. And why would that be the case? Because we are social beings, and social beings are not isolated minds in a social vacuum. Social beings exchange reasons. And this changes the entire picture of how human reason works. Why is that? Again, think about Adam Smith’s argument in economics: every individual pursuing their selfish interest leads to a thriving economy where people do work that other people find useful, generating social wealth. The main point is that there are emergent properties of discourse, of reason in a community, that are not reducible to individual intentions. They are collateral effects: positive externalities in the parlance of current economics. So, for confirmation bias, what we have is a vice at the individual level: every individual is made a worse knower, individually, because of it. But in the social exchange of reasons, this is a virtue: each individual specializes in finding the reasons for one argument, and that means that as a collective we have a clash of viewpoints — the opposite arguments being built by individuals who really believe in them — where every viewpoint is expressed with the strongest support it can get. The public gets to hear the steel-manned version of each side — the strongest case that can be built for it. The model here is the law court, when a judge gets to hear the arguments from both the defense lawyer and the prosecutor, neither of whom has an interest in pursuing the objective truth.

14. And now back to the lecture script — that I got Claude to write to put some order in my hyperconnected mind. The point of all this was to introduce this concept of reasons seen as social devices, not individual ones that we only evaluate for their validity with those few billion brain cells in our little skulls. This is really key to how I’m framing this issue. I have provided you with a few excellent references; now it’s up to you to further explore by googling these researchers or just having further conversations with AI. So back to the script.

15. In the main part of this chapter, I want to give you three ideas: (1) reason as a social, argumentative device, (2) epistemic vigilance, and (3) how these two can still produce truth in equilibrium.

16. Claude suggested starting with a key question, that I write in bold in my notes, since, as you know, AI thinks in a bold way: Why Do We Trust Reasons at All?

17. And now, according to the plan, I’m supposed to start with Mercier and Sperber’s view of reason cognition.

18. According to them: first, we detect bad arguments intuitively, not formally; and second, we also care about who is arguing (we check their expertise, incentives, and track record). In terms of traditional concepts in cognitive science, the first is Reason (with a big “R”) and the second heuristic (with a small “h”), but as we shall see, talking about Reason with a big “R” isn’t really coherent with the philosophy underlying Mercier and Sperber’s view.

19. (I use the big R to poke fun at Kant, Aristotle, and others who have always identified human dignity with reason. I side with David Hume in rejecting this – I don’t think it’s so important for us to distinguish ourselves from other animals, and I would never put reason on a pedestal without mentioning emotions.)

20. The key concept, for both Mercier and Sperber, is again socio-ecological, not individual. You now hopefully understand what I mean by “socio-ecological”, since we talked about the Mandevillian view of confirmation bias: bad for individuals, good for communities. We now think more carefully about how reason, not just confirmation bias, evolved. To do this, think about humans like earthy, evolved animals, not like Platonic souls, please! (I call this “the Humean attitude”, from David Hume, a philosopher who got most ideas wrong, but also a few fundamentally correct ones, I believe.) All social animals, and humans among them, must balance two competing forces if they want to survive and pass their genes to the next generation: First, position themselves to exploit the benefits of social cooperation; Second, position themselves to win, or at least avoid being the loser, in social competition. This endows humans with cognitive capacities that culture builds upon.

21. (Most ideologies, by the way, derive from reducing social animals to either one of these poles. Some ideologies stress only the cooperative side. Other ideologies stress only the competitive side. Obviously, that’s a partial view of who we are. Social animals, humans among them, can only be fully understood if we attach to cooperation and competition roughly the same amount of importance.)

22. So, how do we translate this in relation to reason? The social side of humans is reflected in the idea that we have evolved reason for the sake of arguing. The goal of reason is convincing others, not achieving the truth. Now pause a little bit – this is a powerful thought. And those of you who – like me – positively hate (and that’s a euphemism) post-modern postures may start to feel quite uncomfortable. But you shouldn’t be. We are stressing persuasion as the social pole of this: humans need to exercise persuasion because they need to convince others to cooperate. And cooperation is necessary for all the great things that homo sapiens does. What may make you uncomfortable is the place of truth in this equation. If so, again, that’s only because you’re thinking solipsistically, not ecologically.

23. Because the other side, the competitive and anti-social side, is actually what rescues the role for truth in this picture. The competitive side of humans makes it the case that not all persuasion will serve mutually beneficial purposes. In few words, if you always believe what others tell you, you may get screwed. Humans therefore need to also evolve epistemic vigilance. As Sperber and Marcier explain, this has two dimensions: (1) Content vigilance. This goes back to our “reason assessment” organ, our intuitive ability to check whether an argument makes sense. (2) Source vigilance. This is the heuristic side: who’s saying this? What incentives does this person have? (As Taleb Nassim, the ADHD polymath whose style I am naturally inclined to imitate would put it: “what’s the relevance to reason cognition of having skin in the game?”)

24. Humans activate these two forms of epistemic vigilance all the time. Now, let’s go back to truth. The way Sperber and Marcier frame the purpose of reason (to convince others, not to reach truth) may appear to sacrifice truth. But now epistemic vigilance enters the picture. If people are equipped with epistemic vigilance, it’s not so easy to fool them. Remember the title of Marcier’s solo book: not born yesterday. In a context of equilibrium between people’s argumenting ability and epistemic vigilance, speaking the truth can actually become a winning strategy. Or at least, you know, approximating truth. Or at least, not departing from it in a way that is too easy to spot. Or at least, let’s say, avoiding.complete.bullshit.

25. Now, I find this Humean vision of reason to be compelling. For it can account for the wide diversity of our epistemic practices, including the differences between fields, like mathematics and physics, where there are strict, firm, established tests for screening bullshit out, and those, like the humanities (where I proudly include myself) where those tests are always flexible, malleable, and permanently, essentially contested (Gallie 1956). Let me now go back on the script Claude wrote for me. We’ve covered: (1) The social nature of reason (2) Epistemic vigilance (3) Argumentative theory of reason (4) Historical equilibrium.

26. So, so far, here is the picture in plain terms: reason is social; we’re not idiots, we have epistemic vigilance; reason’s job is persuasion, not abstract Truth; and yet, in the right social set-up, truth can still win as a by-product of people trying to persuade each other with appropriate vigilance.

27. You should immediately smell the analogy between this way of thinking and the Smithian way of thinking about markets, let me read it aloud for you:

“It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner,but from their regard to their own interest. We address ourselves, not to their humanity but to their self-love, and never talk to them of our necessities but of their advantages.” (from The Wealth of Nations.)

28. Now, unless you’re one of the few radical economic libertarians in this country, you’ll react to this by objecting that the Smithian picture of markets is a bit too rosy. We can’t take for granted the public benefit Smithian thesis. On this question I side with institutional economist, like Daron Acemoglu from MIT. This brand of economics stresses that institutions shape long-run economic development. My view of social epistemology is very similar. It is possible for social institutions to favor an equilibrium in which truth-seeking becomes a winning strategy at least in certain important social functions, in the reason exchange game. But we can’t and shouldn’t take this for granted.

 [This bit LLM generated, in order to teach a lesson:  “Now, before I continue with what Claude has prepared for me in the next section about costly signaling, I need to pause and be honest with you about several things.

First, I’m not entirely certain I’ve correctly represented the nuances of Sperber and Mercier’s framework. I read their books carefully, but it’s been a while, and I may be emphasizing certain aspects over others in ways they wouldn’t endorse. There are debates in the literature about whether their argumentative theory of reason is fully compatible with their epistemic vigilance framework, and I haven’t followed all those debates closely.

Second, this equilibrium idea I just presented—linking Mercier to Smith, to institutional economics, to Acemoglu—this synthesis is my own. I haven’t seen it articulated exactly this way in the literature. So it’s possible I’m overreaching, making connections that seem compelling to me but that specialists in these different fields would immediately see problems with.

Third, I should admit that my knowledge of the empirical literature on epistemic vigilance is patchy. There’s a whole body of developmental psychology research on how children evaluate informants, and I’m drawing on summaries rather than primary sources. So some of what I’m saying about how epistemic vigilance actually works might be oversimplified or even wrong.

And honestly, I’m also uncertain about how much this Western, science-worshipping equilibrium I just praised is actually working.

[PAUSE – look directly at students]

So. Did any of that make you trust me more?

[Let them respond]

Think about what just happened. I performed epistemic humility. I signaled uncertainty, admitted gaps, showed awareness of my limitations. And maybe it worked—maybe you thought “he’s being honest about what he doesn’t know, that makes him more credible about what he does know.”

Or maybe you thought “why am I listening to someone who admits he doesn’t know what he’s talking about?”

But here’s the deeper problem: How do you know I meant any of that? How do you know those weren’t just strategically placed uncertainty markers, designed to make me seem more trustworthy? How do you know I didn’t… optimize this performance?

[Let that sit]

Because here’s what I want you to feel viscerally: every honest signal, once recognized as valuable, creates pressure to fake it. The moment we identify “showing uncertainty signals trustworthiness,” showing uncertainty becomes a performance.

Epistemic humility is itself a fakeable signal. And you’re now stuck—even if I’m being genuinely uncertain, you can’t be sure.

This applies to everything: epistemic humility, process transparency, meta-cognitive honesty about AI use, even this very lecture and my awareness that I’m performing awareness of signal collapse.

Now, you might think I could just show you the Claude transcript and prove I was being genuine. But would that help? Or would it just reveal that I strategically chose which confusions to expose and which to hide?

[Brief pause]

Which brings us to the question Claude prepared for me: what has been costly to fake, historically? What made arguments trustworthy before we became this self-aware about signaling?”]

2. What Made Arguments Costly (And Why Did That Matter)?

1. Let’s take stock. Reason does not aim for truth, but conviction. Bullshit is kept in check by the second fundamental function of reason: epistemic vigilance. The equilibrium between these two creates a filter. We now move to a second main theme: costly signals. Let me just read from Marcier’s book, that’s a memorable spot:

2. “With a sleek shape, long horns, elegant black streaks on their flanks, and a bright white rump, Thomson’s gazelles are gorgeous animals—gorgeous, but maybe a bit daft. Packs of wild dogs roam the savanna, ready to chase and devour them, yet when a gazelle spots a pack of dogs, it often fails to flee at full speed. Instead, it stots, jumping on the same spot while keeping its legs straight. It stots high, sometimes as high as six feet. It stots in the absence of any obstacles. It stots even though stotting slows it down. Why does the stupid gazelle not stop stotting?”

3. The answer is in costly signaling theory, developed in parallel by Space in economics and Zahavi in biology. Consider the tail of the peacock. Beautiful, but completely useless. It even slows the animal down. Darwin, who was a great observer and knew which questions to ask, answered this apparent contradiction to survival of the fittest by invoking sexual selection. Peacocks have long tails, because females like mails* who have them. But sexual selection only shifts the question: how could the trait of liking the useless tail, in turn, be selected for?

4. Biology answers this with the handicap principle, saying that the disadvantage of the tail is precisely the reason it gets selected. It’s a waste of calories and slows down the animal. Only the fittest peacocks can afford it. Waste enforces honesty. Cost enforces correlation between signal and quality.

5. Now think about the gazelle. What is the gazelle communicating to potential predators by stotting? Think about this from the perspective of the predator. You have a choice between chasing a healthy, strong gazelle and a less fit one. What do you do? The mechanism is the same: the signal is honest because dishonesty would be costly. In the peacock case, the cost is being slowed down, the signal is directed to the female: I have better genes, can afford this! In the gazelle case, it is directed to the enemy, the potential predator. They are enemies, normally with an interest to fool each other. Instead, honesty gets selected as the equilibrium: it makes both prey and predator better off. Honest signals, again, work only in so far an unfit gazelle wouldn’t be able to fake them (without paying a significant, and ultimately self-defeating, cost).

6. Now back to humans again. Sophisticated reasoning used to be a costly signal of capacity, knowledge, investment, resources. Producing coherent arguments required actual understanding of a topic. It was hard to fake deep understanding without having it.

7. The costly signaling aspect of sophisticated reasoning was compatible with equilibrium between highly developed argumentative skills and highly developed bullshit detectors. On the content side: sophisticated nonsense was costly to produce, so few would produce it. Most nonsense, as a result, was not sophisticated. Because it wasn’t sophisticated, it was easy to spot.

8. On the source side: the linguistic skill of the source demonstrated an underlying capability. A good track record of arguments signaled a track record of authentic understanding. The reputational cost for saying something stupid was real. But what happens if the cost of producing good arguments drops?

9. We need to reflect about the fact that the cheap availability of reasons for persuasion affects two markets, not a single one.

10. The first market is utilitarian. Think again about the cooperative side of human sociality. Arguments are tools for information and coordination. Better reasons – reasons that persuade us because they “sound right” – lead to better collective decisions. This benefits everyone.

11. But there is also a positional market of sophisticated arguments: in this market, good is never enough, it’s always important to be better than someone else’s. Benefits in a positional market are zero sum: sophistication is a competitive advantage as long as not everyone has it.

12. Because of LLMs, anyone can generate coherent argument, sophisticated prose, technical reasoning, sophisticated logical arguments. Cost and time investments are close to zero. For many people, fluent prose immediately sounds “convincing”. The appearance of coherence used to signal genuine understanding of a topic. Now it means access to GPT.

13. In this phase, Mercier’s functions become unbalanced. Our persuasion organ is enhanced and producing compelling arguments has become easier. The volume of such arguments explodes, with less obvious variance in the external signals of their real quality. We can’t observe the efforts behind this production, nor accurately estimate it based on the output. Our evaluation function gets overwhelmed.

3. What Happens When Argument Production Becomes (almost) Free? Epistemia.

1. Prof. Walter Quattrociocchi calls this epistemia: superficial fluency mistaken for epistemic value. In a condition of epistemia, we can’t distinguish people vs. AI and true expertise from faked fluency. Producing arguments has become cheap, while evaluating them is harder than it has ever been.

2. Another useful way to look at the problem is through Dawkin’s theory of memes. Meme is an ordinary word to describe the social media environment, but how many among you know where the term comes from? It comes from Dawkins’ extension of his own gene-centered view of natural selection. Dawkins is most famous for the theory of selfish genes: the real competition is not between organism, but between genes that inhabit them.

3. Memes are cultural replicators analogous to genes. They are units of cultural transmission. They don’t need sex to be transmitted; instead, they jump from brain to brain via communication. Like genes, they compete for selection and retention. Replication, not truth: false but memorable memes can outcompete true but boring ones.

4. If we combine Sperber, Mercier, and Dawkins, we get the idea of reasons – and reasonings – as memes. Arguments – complex chains of reasons – can be spread like memes. They don’t need to exist in the mind of people, agents, etc. They can be produced in large quantities from LLMs, when they serve a person’s argumentative need; and then, spread.

5. (According to Umberto Eco writing in the sixties, intellectuals had moral panic about mass media and mass culture. How could something be for the mass and yet be good? With the arrival of mass – AI we enter a new era, where the problem is mass-reason.)

6. In response, thinkers like Quattrociocchi adopt an apocalyptic view: because of LLMs, we risk a permanent degradation of collective knowing, we end up with polluted information ecosystems, broken trust information, and veer towards a world where logically bad but persuasive argument (reason-memes optimized for virality, not truth), abound.

7. But now we ought to pause for a moment and ask ourselves: is this realistic in the long run? What about Mercier’s point – are we born yesterday?

8. Another way of saying this is to ask: is epistemia a possible equilibrium? Remember Mercier’s concept of communication as an arm race, where generating better persuaders leads to better evaluators, that in turn generates better persuaders, in a sort of infinite progress. With LLMs, and the short pace of social media communication, persuaders just got massive advantage. But evolutionary pressure on evaluators hasn’t vanished. Right now, it’s foreseeable we are in a state of epistemia: persuasion vastly outpaces evaluation. But in the medium term, new evaluation heuristic will emerge. Evolutionary pressure is ruthless (being confused has a cost). Better evaluators outcompete worse. Arms race is built into our cognitive architecture, presumably.

9. So, we should expect new costly signals to emerge – what will they turn out to be? Whatever is hardest to fake. Institutions will adapt – they will be redesigned around the ability to generate, isolate and evaluate the new costly signals. A new equilibrium will be achieved, that will be vastly different from the old one.

10. Filippo Strata, addressing the Doge of Venice, c. 1473-1474, wrote that: “The pen is a virgin; the printing press is a whore.” (Scribe Filippo de Strata’s Polemic Against Printing : History of Information). The early days of the press were the days of pamphlets. This epistemic chaos was short termed. Freedom of the press took long to establish as a legal principle. Now, we’re in the pamphlet-explosion phase of LLMs. I hear a lot of people arguing in favor of tighter regulation: LLMs should not be allow to talk about this or that – perhaps we’re going to enter a new prohibitionist phase.

4. Rebuilding trust in a post-cost world

1. Clearly, reaching a new equilibrium will take time. LLMs have achieved these incredible capabilities in 2-3 years. I don’t think that growth will be as fast in the coming years, but it will be steady (I predict). Norms evolve more slowly: conservativism, in all its forms, is actually a healthy feature of social institutions, for norms must exhibit a certain permanence if they are to function as norms, at all.

2. I believe that we have already overcome the hype phase of LLMs and we’re already in the reckoning phase. I see this phase as one of market testing. A phase in which creative innovators will push for new solutions to the emerging problems, many will be tested, and a few will work. My hope is for some institutions to adapt, while others will collapse. Those that survive will have found a way to let signals emerge organically, for trust to be rebuilt, and for new stable equilibria to be reached.

3. We have already explored hypotheses about the social features that hold promise for rebuilding the social world last week. We discovered that the following signals are no longer sufficient, possibly not even necessary, for trust, especially at a stage of LLM writing more advanced than the ones we have now (more advanced=needing less human-in-the-loop refinement and supervision): 1) Argumentative sophistication (too cheap); 2) Coherence (AI good at it); 3) Sources (can’t always identify them)

4. What will become the new signals? Here are some candidates – again, those of you who were here last week found them out independently: (1) Process transparency: costly because it exposes the LLM or human reason using agent to scrutiny and takes considerable efforts; (2) genuine engagement and accountability: It is costly because it requires permanence; (3) Long-term track record: consistent performance over time with verifiable outcomes: costly and it can’t be instantly generated (especially if proven before the age of LLMs). But this is also slow to establish; (4) Meta-cognitive honesty: this would be about disclosing AI use and showing epistemic humility. For epistemic humility, I already shown at the beginning of this lecture that it can be faked easily too. These are, if you want, the positive news.

5. I now want to explore the darker side of the future of our trust institutions, one that I suspect has already planted seeds in the world we live in.

6. Dark patterns of costly signals

7. First dark pattern: vice as signal. Claude captured it in a nice slogan: when virtue-signaling becomes universal, vice becomes the costly signal. Think again about the logic of Zahavi’s Handicap Principle. It implies a paradox: wasteful traits work because they are costly. The handicap becomes virtue. What would be analogous to it in relation to reasoning, credentials, trustworthiness? One fascinating possibility is that we might be entering – have entered, really – an era in which showing one’s vices sends a signal of strength. I can’t resist thinking about Trump and other anti-woke figures that have emerged, and started to win, precisely in an era of heightened virtue norms (and rampant virtue-signalling on social media). The logic here would be similar to the peacock or the gazelle: “I can violate norms and survive” signals strength. This only works because it’s genuinely risky. Most people who try simply fail; they lose their job or status. But survivors are perceived as strong (even if they have no special merits: they were simply the lucky draw in a fundamentally stochastic machine).

8. What would be the parallel mechanism in the epistemic domain? Perhaps showing confusion, admitting AI use, revealing mistakes (have you notice how often I do it) become honest signals precisely because it’s risky. And again, most who expose those weaknesses will get dismissed. But those who survive prove that they had capital (cultural capital, reputation to risk or burn) to afford such exposure.

9. But notice one thing – and this isn’t good news for the new generation. Saying “I can afford to look uncertain – or to show you my usage of AI to do this better or faster – and still be taken seriously” only works if you have a foundation of credibility. You need visible strength to afford a show-off of weakness.

10. So, as my LLMs love to put this: here’s the uncomfortable implication: Traditional credentials matter MORE, not less, if they are based on non-fakable assessments. Epistemic humility only works if you have capital to survive it; credentials provide that foundation. An intentional exhibit of weakness without credentials is not strength, it’s just… weakness.

11. The problem is that this creates more stratification, not less. Some AI hypers have been telling a fable of AI acting as the great leveler. An opportunity equalizer. But now we see that high-credential individuals can best afford to use AI, because they can do it transparently, paying a small price. They can afford to benefit from AI while resulting more trustworthy by virtue of this honesty signal. But for low/no-credential individuals, the opposite of this is true. They can’t afford weakness, must appear infallible, but infallibility signals “probably AI”. It’s a vicious circle they can’t break from. The initial gap widens, which is the opposite of the initial, democratizing promise.

12. The result has a profound historical irony: those how heralded new technology often criticized gatekeeping, expensive credentials, in-person requirements (at work or in school), oral exams. Or your boss putting an extraordinary amount of stress on you for no real reason, but as a manly, toxic way to test you, to see whether you can survive the pressure. These may be exactly what survives: costly, hard to fake, credentials. Reputational capital you need to have to be able to burn it, in a show off of authenticity or epistemic humility in AI usage. In-person, supervised credentials, moreover, are more expensive, so already advantaged families can afford them more easily, while online learning and other cheaper opportunities lose value, further.

13. This is the dark side of trust preservation. But before I leave you to the rest of your courses, let me provide a summary and then ask you with three questions.

14. Summary: Sperber and Mercier described reasons – in a social way – as constituting an evolved system that worked. This was based on the hidden assumption that producing good arguments was hard. This difficulty created correlation between sophistication and understanding, which calibrated with vigilance mechanisms. All those systems are now no longer in sync, because of LLMs.

15. Now we inhabit an era in which (persuasive, even objectively good) arguments (like this one!) are simultaneously more sophisticated and less reliable. What LLM reduced (their costliness) was paradoxically the very thing that made arguments trustworthy.

16. Now the three questions I believe you want to ask yourself.
1) The first question is: who survives better in this world, specialists or generalists? People who are very good at doing one very specialized things or those who combine different skills from different fields creatively or in a unique way?
2) The second question is: what kind of skills are you acquiring in this degree? Specialist or generalist?
3) And the third question is: what kind of institutional rebuilding is necessary for generalist to be benefited, for centaurs – human-machine hybrid thinkers – to have their costly honest signals, too?

Thank you for you attention and best of luck with your exam!

References:

Acemoglu, Daron, and James A. Robinson. 2008. The role of institutions in growth and development (English). Commission on growth and development working paper ; no. 10 Washington, DC: World Bank. http://documents.worldbank.org/curated/en/232971468326415075

Dancy, Jonathan. 2000. Practical Reality. Oxford: Oxford University Press.

Dawkins, Richard. 1976. The Selfish Gene. Oxford: Oxford University Press.

Darwin, Charles. 1871. The Descent of Man, and Selection in Relation to Sex. London: John Murray.

Eco, Umberto. 1964. Apocalittici e integrati: Comunicazioni di massa e teorie della cultura di massa. Milan: Bompiani.

Gallie, W. B. 1956. “Essentially Contested Concepts.” Proceedings of the Aristotelian Society 56: 167–198.

Hume, David. 1978. A Treatise of Human Nature. Edited by L. A. Selby-Bigge and P. H. Nidditch. Oxford: Clarendon Press. Original work published 1739–40.

Mandeville, Bernard. 1988. The Fable of the Bees: Or, Private Vices, Publick Benefits. Edited by F. B. Kaye. Indianapolis: Liberty Fund. Original work published 1714.

Mercier, Hugo. 2020. Not Born Yesterday: The Science of Who We Trust and What We Believe. Princeton, NJ: Princeton University Press.

Mercier, Hugo, and Dan Sperber. 2017. The Enigma of Reason. Cambridge, MA: Harvard University Press.

Paglieri, Fabio. 2017. “A Plea for Ecological Argument Technologies.” Philosophy & Technology 30 (2): 209–238.

Pongiglione, Francesca. 2013. Bernard Mandeville. Tra ragione e passioni. Rome: Studium.

Smart, Paul R. 2018. “Mandevillian Intelligence.” Synthese 195 (9): 4169–4200.

Smith, Adam. 1976. An Inquiry into the Nature and Causes of the Wealth of Nations. Edited by Edwin Cannan. Chicago: University of Chicago Press. Original work published 1776. An Inquiry Into the Nature and Causes of the Wealth of Nations (Cannan ed.), vol. 1 | Online Library of Liberty

Spence, Michael. 1973. “Job Market Signaling.” The Quarterly Journal of Economics 87 (3): 355–374.

Strata, Filippo de. 1986. Scribe Filippo de Strata’s Polemic Against Printing. Translated by S. Grier. [Pamphlet].

Taleb, Nassim Nicholas. 2018. Skin in the Game: Hidden Asymmetries in Daily Life. New York: Random House.

Zahavi, Amotz. 1975. “Mate Selection—A Selection for a Handicap.” Journal of Theoretical Biology 53 (1): 205–214.

*This is the typical mistake a Large Language Model would never do (unless you ask it to), because it is due to phonetic similarity. So I’ll leave it here as a “vicious signal” of the human behind the pen.