Skip to main content
Back to timeline
arXivSource publication:

EEG-to-text stops squeezing the brain signal into one vector: dual raw-and-sketch supervision reaches 76.08% mean subject accuracy

Synopsis

Visually evoked EEG-to-text generation has typically compressed the whole signal into one global representation and projected it straight into a large language model's hidden space, which can lose color, texture and other fine detail. This work instead learns two complementary EEG token streams under raw-image and sketch-image supervision, then uses a branch-specific semantic query bridge to aggregate each stream into compact semantic slots and turn them into LLM-readable soft prompts. On ImageNet-EEG, mean accuracy across the six subjects reaches 76.08%, and EEG-to-text improves ROUGE, BLEU, METEOR, BERTScore and GPT-4o adequacy over Thought2Text on all three evaluated LLM backbones, LLaMA3-8B, Mistral-7B-v0.3 and Qwen2.5-7B.

Source-provided article image: From Neural Tokens to Language: Dual-View EEG Semantic Bridging for Brain-to-Text Generation
arXiv

Interpretation

Visually evoked EEG-to-text generation no longer compresses the whole signal into one global vector; it keeps two complementary semantic token streams, one oriented to raw images and one to sketches. Earlier approaches typically compressed EEG into a single global representation and aligned it to visual embeddings, text embeddings or an LLM hidden space; single-pathway supervision struggles to preserve both relatively stable structural cues such as category, contour and shape and more variable appearance cues such as color, texture and local detail. On ImageNet-EEG, the encoder reaches a subject-average classification accuracy of 76.08% across six subjects and is highest on all six subjects among the compared methods with per-subject results; the all-subject overall accuracy is 58.94%.

The two token streams are not pushed directly into the language model; each is first aggregated into compact semantic slots and then converted into LLM-readable soft prompts. Directly projecting EEG features overlooks the distributional mismatch among neural, visual and linguistic representations and can cause category errors, missing details and generic descriptions; the branch-specific semantic query bridge preserves the dual-view organization during transfer instead of collapsing it through one projection. On Mistral-7B-v0.3, the full setting raises ROUGE-1 from 30.6 to 36.2, BLEU-4 from 6.1 to 8.5, METEOR from 26.2 to 30.7 and BERTScore from 0.890 to 0.900 over Thought2Text; removing the Stage-C EEG-conditioned language fine-tuning drops ROUGE-1 to 17.2.

The two EEG branches do learn different view preferences rather than duplicating one mapping. Downstream ablations only show the two branches are complementary and cannot directly establish view-specific representations; a branch-level cross-view alignment analysis supplies that representational evidence. Across all 1,987 test samples, the sketch branch's specialization margin toward the sketch view is 0.1900 (95% CI [0.1871, 0.1929]) and the raw branch's margin toward the raw view is 0.0195 (95% CI [0.0168, 0.0223]); 99.85% of samples show a positive sketch preference and 61.40% a positive raw preference.

Perspective

The demonstrated scope is visually evoked EEG on the ImageNet-EEG dataset with a predefined image-disjoint split, containing 7,959 training, 1,994 validation and 1,987 test samples, 128-channel inputs of 440 temporal samples, and six subjects. Generation evaluation follows Thought2Text in using the GPT-4-generated single-sentence captions it provides as textual supervision, and the sketch view is built with Gaussian smoothing followed by Canny edge detection. The authors propose future work on broader datasets and more diverse neural recording conditions.

The authors list limitations including multi-stage training cost, evaluation on a single EEG dataset with a limited participant population, and the lack of validation across broader recording conditions, electrode configurations and open-category settings. Inter-subject standard deviations are moderate, yet the authors still note that subject variability is non-negligible; raw-view specialization is weaker and more category dependent, positive for only 25 of the 40 categories.