BeelineBench is a ready-to-integrate decision model benchmark with seven problem domains and six models, including Jev. The strongest models beat the classic heuristics-based solvers in 4 out of 7 problem domains using best-first search, indicating planning ability.
Our benchmark divides the number of nodes on the shortest possible path to a solution (provided by an oracle) by the number of nodes explored by the decider. A perfect score is 1, indicating the model ‘beelined’ to the goal state. Other diagnostic measures are provided as well.
In this version of the benchmark (v1.0.0), Mercury Decide and Perplexity’s Decider were often the best-performing and on the cost-score efficient frontier.
Our benchmark divides the number of nodes on the shortest possible path to a solution (provided by an oracle) by the number of nodes explored by the decider. A perfect score is 1, indicating the model ‘beelined’ to the goal state. Other diagnostic measures are provided as well.
In this version of the benchmark (v1.0.0), Mercury Decide and Perplexity’s Decider were often the best-performing and on the cost-score efficient frontier.