Skip to main content
Back to timeline
arXivSource publication:

Closed-loop STEM navigation switches to typed decisions: 0.13–0.21 s per decision and 66–91% success on the W dopant task

Synopsis

Closed-loop scanning transmission electron microscopy (STEM) lets each acquired image set the next field of view and magnification, so the decision step must be both fast and correct. On simulated atomic-resolution images of monolayer MoS2, four vision-language models and three decision methods started from identical conditions; Typed Decision reads candidate-label probabilities directly from a pretrained model instead of generating text, cutting every model's decision to 0.13–0.21 s and giving the highest W dopant navigation success rate, 66–91%, for all models.

Source-provided article image: Fast closed-loop STEM navigation using vision-language models with typed decisions
arXiv

Interpretation

The decision step of closed-loop STEM navigation can skip token-by-token text generation and instead read candidate-label probabilities directly from a pretrained vision-language model, then convert the chosen label into a field-of-view shift or magnification change by fixed rules. Earlier autonomous electron-microscopy systems had to define the target for each task, either by training a model on labeled images or by hand-designing a quantity to maximize, so retargeting meant new data or a new design. On simulated atomic-resolution STEM images of monolayer MoS2, the W dopant and grain boundary tasks, four vision-language models and three decision methods all started from identical conditions over 100 episodes each; Typed Decision kept every model's decision time at 0.13–0.21 s.

The method turns a decision into one forward scoring pass: the model assigns a probability to each of six candidate labels (left, right, up, down, inside, none), the highest normalized probability is taken, and a fixed navigation policy maps it to an operation. The two text-generation comparators had the model emit a label in a constrained format, or first produce an explicit reasoning trace and then the label. Structured Outputs and Reasoning+Structured Outputs took 1.3–3.1 and 21–397 times longer per decision than Typed Decision; Typed Decision had the highest W dopant success rate for all models at 66–91%, and the highest grain boundary rate for the best model, Cosmos-Reason2 32B, at 43%.

The same label set and navigation rules do not depend on the target, so switching targets needs only a change in the natural-language description rather than per-target training. Earlier systems needed new labeled training or a redesigned objective to change targets, whereas here one model navigated to both the W dopant and the grain boundary. In the grain boundary task, a paraphrased description and a name-only description reached 59% and 62%, both above the 43% of the reference description; removing the description dropped the W dopant to 49% and the grain boundary to nearly zero.

Perspective

The evaluation is simulated, with two defect types in monolayer MoS2, a W dopant and a grain boundary, imaged as noise-free or Poisson-noisy simulated HAADF-STEM frames; specimen drift, instrument response delays and beam damage on a real microscope were not examined.

Grain boundary success rates are lower than W dopant rates and more sensitive to noise and wording: 43% without noise, 10% under low noise and 2% under high noise; Typed Decision's main failures were missed detection (36%) and reaching the step limit (19%), and reducing missed detection remains an open problem.