A model upgrade needs a question from the work

A new model arrives with images that seem to settle the matter before a comparison has begun. The light is more persuasive, the clothes have weight, and a face appears to hold an expression rather than merely display one. For someone developing an artificial talent, these are compelling possibilities. Yet the talent already has a history of accepted images and decisions. Moving to a different system means finding out what happens to that history when the means of making the next piece changes. The useful question is what the new system would allow this particular body of work to do better, and what evidence would make that improvement credible.

In his 20 October reflection on AI progress, Ethan Mollick describes delegating more stages of production to AI while acknowledging uneven capabilities. His account, which also promotes his new book, gives a timely reason to reconsider familiar working arrangements. It cannot establish which arrangement suits an artificial talent. That decision needs evidence from the assignments and judgements through which the character is being developed.

Give the comparison a real assignment

Imagine a hypothetical talent whose portraits have established a restrained presence: an ordinary room, practical clothing, an expression that leaves some distance between subject and viewer. A new system produces a spectacular fashion image of the same figure. It might be an excellent image and an unhelpful answer. If the next commission requires that reserved presence in a crowded setting, polish alone does not show whether the new system solves the difficulty. The comparison needs to begin with the crowded setting, the intended relationship to the camera and the details that must survive it.

This is where the question of adoption extends beyond recognising the same character across images. Continuity is one condition of success, but the work may also need a less rigid gesture, a readable exchange with another figure or a garment that remains intelligible at a distance. Naming the desired improvement before looking at the results prevents the most attractive surprise from quietly becoming the purpose of the exercise. It also makes room for a useful answer that is narrower than an overall winner.

An assignment drawn from the talent's ordinary work is a better starting point than an exceptional demonstration. A familiar portrait can reveal whether an established standard remains attainable; a recurring difficult composition can reveal whether something has improved. Their value comes from their relevance to future work. A handful of such attempts can expose a practical problem or justify further exploration, but it cannot establish a universal ranking or guarantee what will happen on the next commission.

Understand what an outside score measures

Published evaluations can help identify changes worth investigating, provided their scope remains visible. METR's explanation of task completion time horizons describes evaluations concentrated in software engineering, machine learning and cybersecurity, using tasks with comparatively clear success criteria. The organisation cautions against treating those results as a direct description of all real work. The page was last updated on 8 September and says it is no longer actively updated; it should not be read as today's leaderboard.

A creative brief asks questions that those tasks do not answer. Whether an invented person seems recognisable, whether a pose changes the reading of a garment and whether an image belongs in a developing series require judgements about the work itself. An improvement on a technical benchmark may be real and consequential without resolving any of these questions. The bridge between the two is a hypothesis to investigate, such as whether better handling of several constraints might help a complicated scene. It remains a hypothesis until the scene has been made and examined.

The broader principle has a useful precedent in NIST's discussion of validity and reliability. Its voluntary AI Risk Management Framework calls for testing that reflects intended use and representative conditions. Applied here, that principle supports choosing material from the actual creative task. It does not certify a small editorial comparison or supply an objective score for taste.

Compare the work that produces the result

A fair comparison also needs to account for how each system is used. Giving both the same brief and approved reference material establishes a common purpose. Requiring an identical instruction may be less informative if the systems interpret directions differently. There are two legitimate questions: whether a new system can replace the existing one with little adjustment, and whether an adapted way of working produces a worthwhile result. Those questions have different costs, so the distinction should remain visible when the results are discussed.

In the hypothetical crowded portrait, an initial attempt might preserve the face but make every surrounding figure look toward it. A revised direction could produce the intended indifference. That revision belongs in the account of the outcome, along with the time spent noticing the problem and deciding how to address it. If the old system achieves the same relationship more directly, the new image's greater surface detail may offer little advantage for this assignment. If the new system makes the relationship possible where it was previously elusive, the adaptation may be worth keeping.

This is one reason to look beyond a selected final image to the work surrounding a generated deliverable. Rejected attempts, corrections and review all contribute to the effort required to obtain something usable. Recording them can reveal a gain that a beautiful example conceals, or a burden that an inexpensive generation conceals. Any claim about saved time or lower cost needs observations from that comparison; it cannot be inferred from the result's appearance or from a provider's general promise.

Let the conclusion stay specific

Creative discovery complicates any attempt to keep the brief fixed. A surprising result may suggest a direction better than the one originally planned. That possibility deserves attention without being used to rewrite the terms of a comparison after the fact. The new direction can become a separate experiment, with its own purpose and judgement. Otherwise, a system appears to succeed simply because the assignment has been changed to suit what it happened to produce.

The eventual decision might therefore be modest. A new system could earn a place in complex compositions while an established approach remains useful for quieter portraits. It could justify further trials without yet justifying a change to regular work. These are proposals for interpreting evidence, not claims about a particular model's abilities. Their strength is that each decision can be connected to something observed and reconsidered when the work or the system changes.

For a continuing artificial identity, that specificity matters. Its history need not be remade whenever a more impressive demonstration appears. New capabilities become valuable when they enlarge what can be expressed while preserving the decisions that give the talent coherence. The next piece should offer a reason to change the tools. A persuasive upgrade is one whose benefit can still be explained after the excitement of the first image has passed.