🚀[2025-06-09]: We establish the ChemCoTBench and evaluate scores for open-source LLMs, commercial LLMs, and Chemical LLMs on the Leaderboard!🌟

Overview of the ChemCoTBench and ChemCoT-Dataset. ChemCoTBench presents four key strengths: 1) Novel Setup: Beyond ChemQA, it’s the first step-wise chemical reasoning benchmark; 2) Broad Coverage: Includes foundational tasks (molecule understanding, editing) and two critical applications: molecular optimization (for drug design) and reaction equations (for organic synthesis).; 3) Large Scale: Provides 1,495 samples across 22 subtasks for benchmarking, plus 17K high-quality Chain-of-Thought samples for model training; 4) High Quality: Curated by 13 chemistry Ph.D.s from Tsinghua and Peking University, with strict prompt constraints to ensure CoT correctness.

Despite recent advances in LLM reasoning capabilities, chemistry, a discipline fundamental to areas like drug discovery and materials science, still lacks a benchmark that assesses whether these improvements extend to its complex, domain-specific problem-solving needs. While several benchmarks have been proposed for LLMs in chemistry, they primarily focus on domain-specific question answering, which suffers from several key limitations:

1. Lack of Structured, Stepwise Reasoning and Real-World Relevance: Current evaluations often reduce chemistry assessment to factual recall (e.g., naming compounds or reactions), neglecting the need for operational reasoning akin to arithmetic or coding. Unlike mathematical problems, where solutions demand explicit, verifiable steps, chemistry QA tasks fail to simulate how experts decompose challenges. For instance, they don't capture the process of iteratively refining a molecule’s substructure to optimize properties, considering crucial real-world factors like synthesizability or toxicity, or deducing reaction mechanisms through intermediate transformations. This gap means we're not fully evaluating the analytical depth required in real-world chemistry. Therefore, evaluations must shift from these textbook-like problems to challenges that better reflect practical applications.

2. Ambiguous Skill Attribution in Hybrid Evaluations: Existing benchmarks often conflate reasoning, knowledge recall, and numerical computation into single "exam-style" metrics—for instance, asking LLMs to calculate reaction yields while simultaneously recalling reagent properties. This obscures whether strong performance stems from structured reasoning (e.g., analyzing reaction pathways) or memorized facts (e.g., solvent boiling points). Such ambiguity hinders targeted model improvement and misaligns evaluations with downstream tasks like drug discovery, where success depends on modular reasoning (e.g., decoupling molecular design from synthesizability checks) rather than monolithic problem-solving.

To address these limitations, we introduce ChemCoTBench, a step-by-step, application-oriented, and high-quality benchmark for evaluating LLM reasoning in chemical applications. A core innovation of ChemCoTBench is its formulation of complex chemical tasks, specifically targeting molecular modeling and design, into explicit sequences of verifiable modular chemical operations on SMILES structures (e.g., substructure addition, deletion, or substitution). This approach allows for a granular assessment of an LLM's ability to execute and chain together fundamental chemical transformations. The benchmark features progressively challenging tasks, spanning from basic molecular understanding and editing to property-guided structure optimization and complex multi-molecule chemical reactions. High-quality evaluation is ensured through a dual validation process combining LLM judgment with expert review from 13 chemists.