Automated theorem proving has moved from a research curiosity toward a practical capability. Systems built with large language models and formal proof assistants can now produce mathematics that are verifiable by machines, with every inference checked down to the final step.
The next problem is different.
Generating a correct proof is only part of the process. As machines produce more formal mathematics, researchers will need ways to determine which proofs introduce genuinely new ideas and which ones simply combine techniques that are already well established.
PriorProof, a research program by Neel Somani, explores that question: can a machine measure whether a proof represents a meaningful mathematical contribution?
The Coming Challenge: Too Many Proofs, Not Enough Human Attention
Consider a proof assistant library a few years from now that receives thousands of machine-generated contributions every week.
Each submission may be formally correct. A verifier can confirm that the proof works, but it cannot determine which results deserve attention.
Some proofs may introduce new techniques or open new directions in mathematics. Others may be valid but largely reproduce existing approaches in a different form.
That creates a problem for the people responsible for evaluating mathematical work. Researchers, journal editors, and maintainers of formal libraries will need ways to prioritize attention across a growing collection of certified results.
Human review alone will not scale indefinitely. Asking language models to judge originality creates a different problem: subjective evaluations that are difficult to verify.
PriorProof is an attempt to build a more systematic approach to measuring mathematical novelty.
Measuring Novelty Without Hindsight
Somani’s approach defines novelty in terms of surprise.
The question is not simply whether a proof looks unusual today. The question is whether the techniques used in the proof would have been unexpected given what was known at the time it was created.
For a theorem proved at time t in Lean’s Mathlib library, PriorProof examines the mathematical tools used in the proof and compares them against a historical prior built only from the library as it existed before t.
The prior is conditioned on the theorem statement itself. The system identifies similar earlier theorems and looks at the proof techniques that were typically used for those problems.
A proof that follows the path previous mathematics would suggest receives a lower novelty score. A proof that relies on techniques the historical record would not have predicted receives a higher score.
The process relies on Lean’s elaborated proof terms and commit history. It does not require a manually created taxonomy of mathematical techniques, human labels, or a language model deciding what a proof “means.”
The point-in-time design is also important. A proof is evaluated only against the mathematical knowledge available when it was created. Later discoveries cannot change how surprising the original contribution was at the time.
Building a Metric That Can Explain Its Own Limits
One of the more notable aspects of PriorProof is the attention given to where the metric could fail.
The project treats potential weaknesses as part of the research problem rather than as issues to ignore.
One example is leakage, where future information accidentally influences a historical evaluation. PriorProof addresses much of this risk by constructing its prior from a snapshot of the library frozen before the proof was created.
That prevents later developments from affecting the measurement.
Other sources of leakage require different tests. Retrieval models used during evaluation may contain knowledge from later mathematics, so the project includes counterfactual experiments that replace historical context with unrelated information and test whether results change. If they do not, that suggests the model may be relying on something other than the intended historical signal.
The statement encoder creates a similar concern. If it has learned later proof patterns during training, measurements could be influenced by information unavailable at the time of the original proof. The project tests whether results remain consistent across different time periods.
The framework also makes a directness assumption: it measures the proof that was written, not whether that proof was the shortest or most elegant route to the result.
That creates a potential edge case where an unnecessarily unusual proof of a familiar theorem could appear more novel than it deserves. Rather than ignoring that possibility, the project defines the assumption and includes redundancy detection where mechanical identification is possible.
Where existing methods cannot fully solve a problem, the documentation acknowledges the limitation rather than treating the absence of detection as evidence that the issue does not exist.
The remaining biases are also described carefully. Mathematical ideas can spread through teaching, collaboration, and informal knowledge sharing without appearing in formal dependency graphs. As a result, some genuinely novel proofs may appear less surprising than they actually were.
In that sense, the metric is more likely to underestimate novelty than exaggerate it.
That attention to validation reflects habits from Somani’s earlier work as a quantitative researcher at a major hedge fund, where avoiding look-ahead bias and testing assumptions are central parts of building reliable models.
Testing the Idea Without Overstating the Results
The validation strategy follows the same cautious approach.
Six mechanical tests examine whether the historical prior contains useful predictive information, whether retrieval and smoothing improve results, whether leakage controls work, and whether redundancy detection performs as expected.
The human evaluation component is intentionally limited. Two reviewers compare pairs of proofs and judge which one appears to use the less standard approach.
Those evaluations provide a reference point for the metric and are compared against a language-model baseline using the same prompt.
The design avoids making broad claims about measuring originality from limited evidence. Instead, it treats the current work as a framework that can be tested, refined, and expanded.
Why Measuring Mathematical Novelty Could Matter for AI
PriorProof is designed as a starting point rather than a final solution.
Future directions include evaluating a wider range of machine-generated proofs, measuring surprising connections between different areas of mathematics, and eventually extending similar ideas to informal mathematics, where proof structure is much harder to extract.
The broader question is what happens when machines produce more mathematical work than humans can realistically review.
Novelty has always played an important role in mathematics. It influences recognition, publication decisions, and which ideas receive further attention. But historically, it has been judged through expert intuition rather than a measurable framework.
Somani’s work turns that intuition into a set of more precise questions: novel relative to what body of knowledge? At what point in time? Measured using which signals? With what limitations?
Regardless of whether PriorProof becomes the standard approach, those questions will become increasingly relevant as AI systems take on a larger role in mathematical discovery.
The difference between a proof that is merely correct and one that genuinely advances mathematics may become one of the most important distinctions AI researchers need to measure.
Lynn Martelli is an editor at Readability. She received her MFA in Creative Writing from Antioch University and has worked as an editor for over 10 years. Lynn has edited a wide variety of books, including fiction, non-fiction, memoirs, and more. In her free time, Lynn enjoys reading, writing, and spending time with her family and friends.
