Skip to main content
Back to timeline
arXivSource publication:

Screenshot-to-code moves beyond single HTML files: MESCoder lifts React, Vue and Angular compilation success to as high as 99.1%

Synopsis

Generating front-end code from UI screenshots has largely targeted a single HTML file, while React, Vue and Angular projects only count as usable when the whole project compiles. MESCoder first extracts a layout tree from the screenshot, turning whole-project generation into localized subtree and assembly decisions, then has a Project Agent choose between local repair and cross-file cluster repair under runtime sandbox feedback, trained by offline initialization plus online reinforcement learning. On the 200-page DesignBench test set, compilation success exceeds one-shot generation for all three frameworks across six LLM executors, with the largest gain on Angular, for example 71.1 to 89.2 under GPT-4o.

Source-provided article image: Toward Executable Multi-framework Front-end Code Generation with Self-Correction
arXiv

Interpretation

The generation target moves from a single self-contained HTML file to multi-file executable projects in React, Vue and Angular, where correctness depends on whether the whole project compiles and runs. Prior generation methods and benchmarks were HTML-centered, assuming a single file that renders without compilation or cross-file reasoning, whereas real front-end projects consist of multiple files in framework-specific structures. On the 200-page test set of the DesignBench design-generation split, MESCoder's compilation success exceeds the one-shot baseline on all three frameworks across six LLM executors, with the widest gap on Angular, 89.2 versus 71.1 under GPT-4o.

Generation is organized as a sequence of localized decisions that expand subtrees along the layout tree and assemble them, with self-correction inside the generation loop rather than as post-hoc repair. Generating the whole project in one pass entangles local implementation, global composition and framework constraints in a single decoding process, making intermediate errors hard to localize and fix. An ablation with GPT-4o as executor reports average compilation success across the three frameworks: removing the scaffold drops it to 80.8 against 95.4 for the full method, removing the PatchCluster action drops it to 85.5, and removing the Assemble action drops it to 89.7.

A lightweight policy network observes the current partial project, sandbox diagnostics and recent action history, and chooses among continuing construction, local repair, cluster repair, assembly and finishing, while a fixed LLM executor produces the concrete code edits. The offline stage initializes an action prior from clean-project replay and result-driven labels on corrupted states, and the online stage refines that policy with PPO-style reinforcement learning on sandbox feedback. On stagnated-cluster states the offline-only policy splits probability nearly evenly between PatchLocal and PatchCluster, while after online reinforcement learning PatchCluster becomes clearly dominant; on the stagnated-cluster diagnostic subset MESCoder reaches a 90.3% escalation rate and 76.2% final compilation success, above the offline-only policy's 57.8% and 58.3%.

Perspective

Evaluation is limited to the 200-page test set of the DesignBench design-generation split, covering React, Vue and Angular with six frozen LLM executors, measured by compilation success under strict compilation and runtime verification and by CLIP visual similarity; the method focuses on syntactic, structural and build-time executability.

The system depends on the quality of the scaffold preprocessor and the runtime sandbox, and incorrect structure extraction or incomplete framework checks can mislead the policy and executor; when the executor repeatedly fails to produce valid patches, the policy alone cannot recover the project; online reinforcement learning adds computational cost from repeated LLM and sandbox calls; broader software engineering properties such as security, accessibility and maintainability are not fully addressed.