CULTESC Research · Evaluation note
What a Consistency Score Leaves Out
Reading consistency scores before they enter the creative brief.
A benchmark ranking, a director’s approval, and a viewer’s recognition require different evidence.
At CULTESC, the creative question is what can change while a character still reads as itself: its angle, mood, or performance. Before using a score to guide that decision, read what qualified for evaluation.
Read coverage beside identity.
In EntityBench v1, StoryMem leads EntityMem on the raw LLM face-identity score. EntityMem leads after the reported correction incorporates .
| Measure | StoryMem | EntityMem |
|---|---|---|
| Raw identity score | 0.718 | 0.678 |
| Evaluated-pair coverage | 0.31 | 0.60 |
| Gate-corrected score | 0.226 | 0.406 |
A conditional average describes the qualifying cases. Read admission rules, evaluation failures, and aggregation together. This comparison does not isolate which judge is better.
Ask what the judgment can support.
A supplied-image comparison does not establish memory after an episode or approval of a finished shot. The intended decision determines the evidence still needed.
Separate . Identifying a character, endorsing its portrayal, enjoying an episode, and returning are distinct outcomes.
Voice, relationships, and behavior are cues CULTESC wants to investigate. Their role, and the effect of familiarity, cannot be read from a visual score alone.
A useful score should be tested against the production decision or audience judgment it is expected to inform, on the same items with failures retained.
Sources and scope
This is a source-led evaluation note, not a CULTESC benchmark rerun or viewer study. The ranking example concerns the reported evaluation in He et al., EntityBench; it establishes no general production or audience ranking.
The score enters a creative decision with its own conditions attached. Keeping those conditions visible is part of understanding what the number can offer the next shot, the next episode, and the next viewer.