Introduction
The generation of new ideas, like a new organism, is full of diversity, randomness, and exploration! Though not all ideas or traits survive, some remain, and others just disappear! In the case of life, the forces shaped by natural selection prune the irrelevant ones, and in the generation of ideas, the "judgment" by a shared "value" system prunes the bad ideas from the good ones!
In the narrowest sense, a judgment can be reduced to a verification or evaluation. For example, you can imagine a new proof as a new generation that gets verified by an algorithm. The problem of verification in mathematics and its relationship to proof at the beginning of the 20th century led to a host of different mathematical frameworks, from "proof theory" to "semantic theories of truth".
Today's machine learning largely relies on a new revolution of using "judgmental feedback" from different judgment signals to improve or align the responses of large language models. To see this, we need to go back to when language models started to show real signs of usefulness when trained using Reinforcement Learning from Human Feedback (RLHF). These signals were so important that they fundamentally changed model behavior. Soon, researchers realized the importance of these signals for moving beyond the pretraining phase and the limitations of training data. In Reinforcement Learning from Verifiable Reward (RLVR), the models could get much better at tasks where checking their correctness was algorithmic and hence even removed the limitations of the inefficiency of human feedback. For example, in one task, a language model is asked to derive the integral of a given function; the result can then be verified using a derivative. This simple trick helps the model perform not only on this specific task but also gets much better on other irrelevant tasks and beat benchmarks more easily. Finally, as another example, we see the rise of Reinforcement Learning from Compiler Feedback (RLCF) as the coding analog, where the compiler determines whether the code generated by an LLM is good (if it compiles without errors) or not!
But why do these methods work so well? To understand better, we need to look at the asymmetry between two operations: "generation" and "judgment" from a computational complexity perspective. For this, we can start with the most restricted sense of judgment: mathematical verification. Even though it is very hard to come up with a new proof for a given problem, verifying whether it is correct or not is much easier! To see this, we can think about the Boolean Satisfiability Problem (SAT), which asks whether there exists an interpretation that satisfies a given Boolean formula. An interpretation here means assigning truth values (T/F) to the variables. Even though finding the correct interpretation is hard (thought to be NP-hard), verifying whether your interpretation is correct is fairly fast and easy.
The same can be said in more general senses of judgment, such as verifiable reward or compiler optimization, and even human judgment. Taking derivatives of a formula is easier than integration, and we can use that to check whether it is done correctly. The compiler is much simpler than the system that can generate code. In the case of human judgments, feedback could be given by choosing one option among many to continue the prompt.
The above framing helps us understand where we stand relative to these new tools. Unlike in the past, generating code, ideas, and proofs is much less valuable than judging them, and even though judgment seems easier, not everyone ends up with the same code starting from the same problem.
The nuances inherent to our judgments are more interesting when you move away from verification, the weakest sense of judgment, to a broader sense of valuation. The valuation function in the verification sense is as simple as a binary one that corresponds to T/F, compiled/not-compiled. But it can be a mixture of personal preferences and experiences, and finally aesthetic.
The apparent complex nature of the valuation function usually ties to its social aspect. Assigning value usually happens in a social context. Let's go to the previous example of RLHF: in the human feedback scenario that later grew into the "alignment problem", an AI system's output should be preferentially selected and judged based on valuations determined by the nature of us and by society as a whole. Because of this, judgment does not inherently lead to "truth" (as in the case of verification) but instead reflects alignment with a collective valuation function. This is why we stretched the judgment from verification all the way to aesthetics.
Judgment is still a very complicated human activity that shows our biases too. But this runs deeper than you might think at first; to see this, let's go back to our example of verifiable rewards, where the model has to generate the integrals of a given function. Many studies have shown an interesting phenomenon of "reward hacking," where the model might find a way to hack the solution by not actually solving it, but by finding a hack that can't be exposed by the biases in our sampling and inputs. Common forms of reward hacking in LLMs include length bias, where the model provides excessively lengthy responses to obtain higher reward scores; sycophancy, where the model agrees with false user statements rather than giving true information; and sophistication bias, where the model provides false information in a convincing manner.
Now the question becomes: how do we judge LLM outputs? Are we flattered by it and not motivated to put it into tests, or do we use them on smaller scales where the judgment can't get foggy, misled by our internal biases and the constant pressure of getting better, smarter in "directing" them? Like any other technology before it, AI will also give us more power, but it will not necessarily democratize the generation of good ideas! Good ideas eventually have to move beyond personal judgment to "valuation," which is a collective decision. It rests on experimentation and falsifiable claims that can be tested against the real world's needs and future demands.