Skip to main content
Back to timeline
arXivSource publication:

Constrained recall can't see structural contradictions: CoupledSGA cuts the contradiction rate from 45.92% to 4.07% at a cost in R@20

Synopsis

Continuous-time scene graph anticipation previously let each subject–object pair evolve independently, so relationships sharing an entity could not influence one another and one pair could be scored as both sitting on and standing on. CoupledSGA links relationships that share an entity through a Relationship Interaction Graph, adds graph attention inside the ODE vector field, and penalizes simultaneous activation of mutually exclusive predicates. On an 800-video Action Genome subset, enabling that penalty after two warm-up epochs lowers R@20 from 42.21% to 35.18% while cutting the contradiction rate under the No-Constraint protocol from 45.92% to 4.07%; the analysis also proves that constrained recall cannot reflect such violations when the within-group winner is fixed.

Source-provided article image: Coupled Scene Graph Anticipation: When With-Constraint Recall Hides Structural Inconsistency
arXiv

Interpretation

The evaluation protocol for scene graph anticipation itself hides structural contradictions: under the With-Constraint protocol the evaluator keeps only the top-scoring predicate per subject–object pair and predicate group, so two mutually exclusive predicates in the same group cannot both reach the scored output. Such methods inherited recall metrics from scene graph generation and video scene graph generation, and the With-Constraint and No-Constraint protocols were treated as complementary views, but whether the constrained protocol can measure structural consistency had not been examined separately. The work proves that when the winning predicate and its score in each group stay fixed, With-Constraint R@ and mR@ are unchanged regardless of whether a within-group exclusion rule is violated; every exclusion rule considered lies within a single predicate group.

CoupledSGA lets relationships that share an entity influence one another during temporal integration instead of evolving independently. SceneSayerODE previously assigned each subject–object pair an independent Neural ODE, so trajectories were integrated separately and relationships sharing an entity could not constrain each other during evolution. The Relationship Interaction Graph takes relationship pairs as nodes and joins pairs that share an entity, graph attention is placed inside the ODE vector field, and a Structural Consistency Loss is built on a matrix of mutually exclusive predicates; evaluation uses the PredCLS task on Action Genome with 600 training and 200 test videos.

The structural consistency penalty does reduce contradictions, but the cost shows up in recall, and constrained recall records only the cost, not the benefit. Under a matched training budget CoupledSGA reaches 40.48% R@20 against 44.78% for SceneSayerODE; coupling alone, without the penalty, lowers the contradiction rate from 54.40% to 45.92% at a cost of 1.94 points of R@20. In a three-seed controlled comparison, enabling the penalty after two warm-up epochs lowers R@20 from 42.21% to 35.18% while lowering the contradiction rate under the No-Constraint protocol from 45.92% to 4.07%; these results are not intended to establish state-of-the-art performance.

Perspective

The conclusions rest on the PredCLS task of Action Genome, using a fixed 800-video subset (600 training and 200 test videos) and a four-epoch training budget on a single 12 GB RTX 3060; the authors state that these results are not intended to establish state-of-the-art performance and that absolute numbers are not comparable to published full-split results.

Under the four-epoch budget, whether the added capacity was trained enough cannot be separated from a real deficit of the coupled formulation; two warm-up epochs were not enough for the penalty to avoid the recall cost; structural validity is assessed only through the top-two contradiction rate, and the stricter structural consistency score is not reported; the exclusion matrix is hand-curated and its coverage of the Action Genome ontology is unknown; explicit Euler is first-order and its truncation error is not quantified against an adaptive solver.