An epistemic difference principle between humans and AI

In a recent preprint, Hoda Heidari and Aydin Mohseni defend the importance of preserving human understanding in AI-driven science. But their argument seems to rest on a questionable assumption: that understanding a result requires understanding the grounds of its justification.

Why?

An AI could prove a theorem through a demonstration no human can comprehend. We might nevertheless be justified in accepting it, and fully understand some of its implications.

Knowledge we cannot understand can produce understanding we otherwise could not attain.

This creates a trade-off within understanding itself.* Insisting on intelligible foundations might reduce our overall understanding.

Imagine AI knowledge growing by 90% and human understanding by 10%. We lose epistemic ground relatively, but gain absolutely.

Two models, borrowed from economics:

1. Epistemic trickle-down. Let AI knowledge grow. Some understanding will eventually trickle down to humans.

But why assume this? The Substance (great movie and huge metaphor, hence the picture above) offers a disturbing metaphor: the superior successor flourishes by consuming the body from which it emerged.

2. An epistemic difference principle. Adapt Rawls: epistemic inequalities between AI and humans are justified insofar as they maximize the understanding attainable by the epistemically weaker class—humans.

Why is this attractive?

Because it establishes a particular relationship between humans and AI: one in which the superior epistemic capacities of AI become a source of human epistemic advancement, rather than human epistemic obsolescence.

It neither requires epistemic equality nor assumes that any trickle-down benefit is sufficient.

The point is not to prevent AI from surpassing us. It is to make being surpassed the best possible way of expanding our understanding.

*To be clear, Heidari and Mohseni explicitly concede that AI-generated knowledge beyond human understanding can have pragmatic value. My objection is that they seem to overlook its potential epistemic value: knowledge we cannot understand may nevertheless produce further understanding. The relevant trade-off is therefore not merely between human understanding and practical benefits, but within human understanding itself.