Word Five

1"""The reward. This file is quoted verbatim on the demo page, so it is written2to be read by someone who does not write Python.3 4Three components, weights summing to 1.0. Two of them pull toward winning. The5third pulls against them on purpose.6 7# region: pig-demo/reward8"""9 10from __future__ import annotations11 12from dataclasses import dataclass13 14from .engine import GREEN, WORD_LENGTH, score_guess15 16SOLVED_WEIGHT = 0.5017ECONOMY_WEIGHT = 0.3018CONSISTENCY_WEIGHT = 0.2019 20 21@dataclass22class Episode:23    """Everything the reward needs from one played game."""24 25    answer: str26    guesses: list[str]27    patterns: list[str]28    rejected: int29    reference_depth: int30    max_guesses: int = 631 32 33def solved(ep: Episode) -> float:34    """Did it win? The objective, and half the reward."""35    return 1.0 if ep.patterns and ep.patterns[-1] == GREEN * WORD_LENGTH else 0.036 37 38def economy(ep: Episode) -> float:39    """How few turns it took, against how few the reference player needed.40 41    Always a ratio against a reference on the SAME hidden word, never an42    absolute count — otherwise the reward would mostly measure whether the word43    happened to be easy, and a model would be punished for a hard draw.44    """45    if not solved(ep):46        return 0.047    return min(1.0, ep.reference_depth / max(1, len(ep.guesses)))48 49 50def consistency(ep: Episode) -> float:51    """The share of guesses that were still possible answers when they were made.52 53    This is the counterweight, and it is genuinely in tension with the other54    two. A player maximising information will deliberately guess words that55    CANNOT win, because a word that splits the remaining candidates evenly56    teaches it more than a word that might happen to be right. That is good57    play, and it costs consistency. The reference solver scores below 1.0 here.58 59    So the three components describe a real trade: win, win fast, and do not60    spend turns on moves that could not have won. You cannot max all three, and61    which one you weight is a decision about what you actually want — which is62    the entire argument this demo exists to make.63 64    Scored over every turn SPENT, not every guess accepted — see below. Without65    that, a policy which never produces a parseable word scores a vacuous 1.066    here (no guesses, therefore no contradictions) and collects 0.2 for doing67    nothing at all.68    """69    spent = len(ep.guesses) + ep.rejected70    if spent == 0:71        return 0.072 73    viable = 074    for i, guess in enumerate(ep.guesses):75        # A guess is viable iff it is consistent with every earlier piece of76        # feedback — that is, it was still a possible answer at the moment it77        # was played.78        if all(score_guess(ep.guesses[k], guess) == ep.patterns[k] for k in range(i)):79            viable += 180 81    # The denominator is every turn SPENT, including replies the game refused.82    # Counting only legal guesses would hand a free 1.0 to a policy that plays83    # one word and then jams the parser five times: one guess, no84    # contradictions, perfect score. Measuring against turns spent asks the85    # question that actually matters — of the attempts you made, how many were86    # a move that could have won?87    return viable / spent88 89 90def score(ep: Episode) -> dict[str, float]:91    """The weighted components, unweighted. The caller applies the weights."""92    return {93        "solved": solved(ep),94        "economy": economy(ep),95        "consistency": consistency(ep),96    }97 98 99def total(ep: Episode) -> float:100    parts = score(ep)101    return (102        parts["solved"] * SOLVED_WEIGHT103        + parts["economy"] * ECONOMY_WEIGHT104        + parts["consistency"] * CONSISTENCY_WEIGHT105    )106 107 108# endregion: pig-demo/reward109 110 111def metrics(ep: Episode) -> dict[str, float]:112    """Diagnostics. Weight zero — reported, never summed into the reward."""113    return {114        "guesses_used": float(len(ep.guesses)),…