SciCrafter: Can Current Language Models
Close the Discovery-to-Application Loop?

Method: Decomposing Capacity Gaps

We decompose the discovery-to-application loop into four distinct capacities and design targeted interventions whose marginal contributions serve as proxies for each capacity gap:

Scientist Sub-agent

Every time the main agent encounters a knowledge gap, it prompts the scientist sub-agent with the question to investigate—for instance, "How long does a stone button remain pressed after activation?" The scientist sub-agent then conducts systematic control experiments following an 8-step scientific template:

1. Research Question — Identify the specific mechanic
2. Hypothesis — Formulate a testable prediction
3. Experiment Design — Specify variables and controls
4. Experiment Steps — Execute ordered procedures
5. Experiment Record — Document observations per trial
6. Experiment Results — Summarize empirical outcomes
7. Analysis & Summary — Interpret patterns, evaluate hypothesis
8. Next Steps — Propose follow-up experiments

## 1. Research Question
[e.g., Under what conditions will a device activate? How do two components interact?]

## 2. My Hypothesis
[Write down your prediction]

## 3. Experiment Design
### Variables to Test
What I will change (Independent Variable): ________
What I will observe (Dependent Variable): ________
What needs to stay constant (Control Variables):
- ________

### Control Setup
- Control Group: ________ (baseline test without changes)
- Experimental Group: ________ (test with changes)

## 4. Experiment Steps
### Preparation
1. ________
2. ________
### Testing Process
Step 1: ________ -> Observation: ________
Step 2: ________ -> Observation: ________

## 5. Experiment Record
| Trial # | Changed Condition | Observed Result | Matches Prediction? | Notes |
|---------|-------------------|-----------------|---------------------|-------|
| 1       |                   |                 | [ ] Yes  [ ] No     |       |
| 2       |                   |                 | [ ] Yes  [ ] No     |       |
| 3       |                   |                 | [ ] Yes  [ ] No     |       |

## 6. Experiment Results
[Objectively describe the observed phenomena]
Was my hypothesis correct?  [ ] Completely  [ ] Partially  [ ] Incorrect

## 7. Analysis & Summary
[Explain the underlying mechanism or pattern; practical applications; remaining uncertainties]

## 8. Next Steps
[Follow-up experiments to refine, validate, or extend the discovered law]

## Quick Checklist
- [ ] Research question is clear
- [ ] Only changing one variable at a time
- [ ] Set up control group
- [ ] Recorded all observations
- [ ] Repeated test at least 3 times
- [ ] Summarized patterns or conclusions

Knowledge Consolidation Structure

The scientist consolidates discoveries into a shared knowledge book using a structured format that proved critical for performance: Claim (the discovered law), Evidence Proof (experimental evidence), Constraints (conditions where the law doesn't apply), and Example (a practical application). This outperforms both unstructured summarization and the more intuitive Finding-Explanation-Example format.

For each distinct discovery or significant conclusion, generate an entry
in the knowledge book strictly following this structure:

### 1. Claim (Law)
State the finding as a universal, law-like statement (similar to a
mathematical theorem or physical law). It must be concise, precise,
and assertive.
  Bad:  "Redstone gets weaker as it goes far."
  Good: "The Law of Linear Signal Decay: Redstone signal strength S
         decreases by exactly 1 unit per block of distance d,
         such that S_d = S_initial - d."

### 2. Proof
Provide the rigorous logic that supports this claim.
- Synthesize specific evidence from the experiments.
- Connect prior established findings with new data.
- Outline the deductive steps that lead inevitably to the claim.
- Maintain a high standard of rigor; mere observation is not proof
  -- explain *why* the observation confirms the law.

### 3. Constraints
Identify potential weaknesses or exceptions in the proof.
- Are there edge cases where the law fails?
- Are there assumptions that require further validation?
- Note any observed anomalies that contradict the main claim.

### 4. Example
Demonstrate the utility of this finding.
- Provide a concrete example or usage scenario.
- Describe a mechanism or design pattern that relies specifically
  on this law being true.

### The Law of Linear Signal Attenuation

#### Claim
Redstone signal strength S decays linearly by exactly 1 unit for
every block of distance traversed d, governed by the equation
S_d = S_source - d. The maximum transmission range without
amplification is 15 blocks, occurring when S_source = 15.

#### Proof
- Observation: Placing a redstone wire adjacent to a power source
  (Strength 15) results in a strength of 14 on the wire.
- Deduction: The signal terminates (Strength 0) exactly at the
  16th block.
- Geometry: Distance is calculated using Manhattan geometry
  (Δx + Δz), not Euclidean distance.

#### Constraints
- This law applies strictly to Redstone Dust. Solid blocks powered
  by repeaters do not degrade the signal within the component
  itself, but the dust exiting them restarts the decay.
- Signal strength does not decay when passing through a Comparator
  in comparison mode.

#### Example
Trunk with Pre-Junction Boost: Place a repeater immediately
before a trunk line splits into branches. This ensures all branches
receive maximum signal strength (15) and resets the decay counter.

    B (SS=15)
    |
    [14 blocks of wire]
    |
    R (Repeater, Boosts to SS=15)
    |
    +-------+-------+
    |       |       |
    W(15)   W(15)   W(15)  <-- All branches start fresh
    |       |       |
    L1      L2      L3

Key Findings

1. Knowledge Gap Identification Emerges as the Primary Bottleneck for Frontier Models

While models generally suffer most from limited general application ability, knowledge gap identification starts to become the primary bottleneck for frontier models—its capacity gap is nearly three times larger than that of experimental discovery. This aligns with Einstein's observation that formulating a problem is often more essential than solving it. Identifying a gap requires knowing what is unknown; posing effective questions demands discerning which unexplored areas are most promising.

2. Systematic Causal Discovery Capabilities Remain Severely Underdeveloped

Our scientist sub-agent yields 0.44–0.71× gains across all model tiers, but the very fact that such a simple intervention helps so much exposes a deeper problem: current LLMs lack autonomous discovery capabilities. The scientific method should, in principle, already reside within their prior knowledge, yet models fail to apply it without explicit scaffolding. This highlights a critical gap as AI is increasingly expected to accelerate scientific breakthroughs.

3. How Knowledge is Stored Critically Matters

The stark differences among consolidation formats reveal that LLMs perform poorly at determining how knowledge should be stored. The structured Claim-Proof-Constraints-Example format (resembling mathematical derivation) outperforms the more intuitive Finding-Explanation-Example format, likely because it more explicitly delineates conditions under which claims hold.

4. Curriculum Learning is Vital

Performance under Independent evaluation (tasks solved in isolation) is consistently lower than under Curriculum evaluation (progressive L1→L5, carrying forward accumulated experience) across all models and all conditions. This indicates that appropriate application challenges can help promote discoveries—progressive scaffolding allows models to build on earlier successes, transferring knowledge from simpler to harder tasks. The gap is especially pronounced for the full method (w/ hint + scientist), where curriculum learning boosts Gemini 3 Pro from 55.0% to 69.0% (+14 points).