Abstract: |
Research on judging decisions made by fallible (human) agents is not as much advanced as research on finding optimal decisions, and on supervision of AI agents´ decisions by humans. Human decisions are often influenced by various factors, such as risk, uncertainty, time pressure, and depth
of cognitive capability, whereas decisions by an AI agent can be effectively optimal without these limitations. The concept of `depth´, a well-defined term in game theory (including chess), does not have a clear formulation in decision theory. To quantify ´depth´ in decision theory, we can configure an AI agent of supreme competence to `think´
at depths beyond the capability of any human, and in the process collect evaluations of decisions at various depths. One research goal is to create an intrinsic measure of the depth of thinking required to answer certain test questions, toward a reliable means of assessing their difficulty apart from item-response statistics. We relate the depth
of cognition by humans to depths of search, and use this information to infer the quality of decisions made, so as to judge the decision-maker from his decisions. Our research extends the model of Regan and Haworth to quantify depth, plus related measures of complexity and difficulty, in the context of chess. We use large data from real chess tournaments
and evaluations from chess programs (AI agents) of strength beyond all human players. We then seek to transfer the results to other decision-making fields in which effectively optimal judgements can be obtained from either hindsight, answer banks, or powerful AI agents. In some applications, such as multiple-choice tests, we establish an isomorphism of the
underlying mathematical quantities, which induces a correspondence between various measurement theories and the chess model. We provide results toward the objective of applying the correspondence in reverse to obtain and quantify measure of depth and difficulty for multiple-choice tests, stock market trading, and other real-world applications. |