Score

When to use score#

Use score when the possible answers form a ladder: each step means more of the same thing. Being one step off is then a small mistake and three steps off a big one, and score takes that into account. Choice would treat "today" and "stop the pump now" as unrelated labels; score knows they are neighbours. If the options are just different (billing, shipping, returns), use choice. If there are only two, use noul.

Question object#

Example#

A plant's monitoring system writes a short status note for every pump. The maintenance planner has one crew and many pumps, and needs to know which to visit first. Every pump is somewhere on the same ladder, from "can wait" to "stop it now", so this is a score question.

• state is the status note as the monitoring system wrote it.
• instructions asks how urgent maintenance is.
• criteria lists four levels, lowest first. Each one is written as the action the planner would take, not as an abstract word like "medium". That way the answer maps straight onto the schedule.

{
  "state": "Pump P-204: grinding noise since this morning, vibration 2x baseline, temperature normal, no leak.",
  "questions": {
    "urgency": {
      "type": "score",
      "instructions": "How urgent is maintenance on this pump?",
      "criteria": ["Can wait for the next planned stop", "This week", "Today", "Stop the pump now"]
    }
  },
  "reasoning": "auto"
}

Most of the probability lands on "today". A grinding noise with double the normal vibration is a developing fault. Normal temperature and no leak mean it is not an emergency yet, so "stop now" gets a real but smaller share and "can wait" almost none.

"urgency": {
  "type": "score",
  "score": 2.02,
  "legend": { "0": "Can wait for the next planned stop", "1": "This week", "2": "Today", "3": "Stop the pump now" },
  "probabilities": { "0": 0.02, "1": 0.13, "2": 0.66, "3": 0.19 },
  "confidence": 0.32
}

The expected level is 2.02: "today", leaning slightly towards "stop now". Sort every pump by this number and the crew's route for the day writes itself.

Reading the answer#

• score is the expected level: the probability-weighted average of the level numbers. It can fall between levels, which is what makes it good for sorting a queue.
• probabilities gives each level's chance. For a hard rule, add up the levels you care about, e.g. act if p["2"] + p["3"] ≥ 0.8.
• legend maps each key back to your level text, so the answer is readable on its own.
• confidence says how concentrated the distribution is. It is not the chance of being right; see Probabilities & confidence.

The same score can mean different things

A score of 1.5 on a four-level scale could mean Wity is torn between the two middle levels. It could also mean a 50/50 split between "can wait" and "stop now". Both have the same score, and even the same confidence (0.50). The first is a mild pump; the second is a pump you know almost nothing about.

Sort by score, but make safety rules on probabilities. "Stop now above 0.3 pages the on-call engineer" catches the second pump. A rule on the average would miss it.

Splitting a big judgement#

"How good is this?" is rarely one scale. Quality has several parts, and a single score blends them in a way you cannot see or change. Ask one score question for each part in the same request, and do the blending in your code with weights you choose.

Here, a support team reviews its agents' replies for coaching. A customer wrote:

The reply is friendly and does something. But the team's policy says a customer with a deadline the trace won't meet should get an express replacement straight away:

Is this a good reply? It depends which part you look at, so the team scores four parts separately:

• resolved: did the reply meet the customer's real need, which is having the gift by Saturday, not just getting the parcel back eventually?
• policy: did the agent follow the written policy? It has three levels, because "broke it" and "missed part of it" are very different findings.
• effort: how much is left for the customer to do?
• tone: how the reply reads, on its own scale so a warm reply can't hide a wrong one.

All three texts go in the state as one object, so every question can compare the reply with the message and the policy.

{
  "state": {
    "customer": "My parcel was marked as delivered yesterday but it isn't here, and the neighbours don't have it. I need it for my daughter's birthday on Saturday.",
    "agent_reply": "Sorry to hear that! I've opened an investigation with the courier, which takes up to 3 working days. If they can't find it, we'll refund you in full.",
    "policy": "Parcel marked delivered but missing: open a courier trace. If the customer says they need the item by a date the trace won't meet, send a free replacement by express at once."
  },
  "questions": {
    "resolved": {
      "type": "score",
      "instructions": "How well does the agent's reply solve the customer's actual problem?",
      "criteria": [
        "Does not address the problem",
        "Addresses it, but leaves the customer's real need unmet",
        "Solves it, but next steps are unclear",
        "Solves it completely, with clear next steps"
      ]
    },
    "policy": {
      "type": "score",
      "instructions": "How closely does the reply follow the policy?",
      "criteria": [
        "Does or promises something the policy forbids",
        "Misses something the policy requires",
        "Follows the policy fully"
      ]
    },
    "effort": {
      "type": "score",
      "instructions": "How much is left for the customer to do after this reply?",
      "criteria": [
        "They will have to chase us to get anywhere",
        "They have to wait and probably write back",
        "One simple step",
        "Nothing, it is handled"
      ]
    },
    "tone": {
      "type": "score",
      "instructions": "How does the reply come across?",
      "criteria": ["Rude or dismissive", "Flat and scripted", "Polite", "Warm and personal"]
    }
  },
  "reasoning": "auto"
}

The four answers come back together (the legends are shortened here):

{
  "answers": {
    "resolved": { "type": "score", "score": 1.42, "confidence": 0.31,
                  "probabilities": { "0": 0.03, "1": 0.58, "2": 0.33, "3": 0.06 }, "legend": { … } },
    "policy":   { "type": "score", "score": 1.11, "confidence": 0.47,
                  "probabilities": { "0": 0.04, "1": 0.81, "2": 0.15 }, "legend": { … } },
    "effort":   { "type": "score", "score": 1.13, "confidence": 0.39,
                  "probabilities": { "0": 0.09, "1": 0.74, "2": 0.12, "3": 0.05 }, "legend": { … } },
    "tone":     { "type": "score", "score": 2.02, "confidence": 0.40,
                  "probabilities": { "0": 0.01, "1": 0.12, "2": 0.71, "3": 0.16 }, "legend": { … } }
  }
}

What the answers say

• resolved at 1.42: mostly "addresses it, but the real need is unmet". The trace and refund deal with a lost parcel, not with a birthday on Saturday. The 0.33 on the next level up shows it is not a total miss.
• policy at 1.11, with 0.81 on "misses something the policy requires": the express replacement. Only 0.04 goes to "forbidden". This is an omission, not a breach.
• effort at 1.13: the customer has to wait up to three days and will probably write back.
• tone at 2.02: polite, a little warm. The reply sounds fine. That is exactly why tone is scored on its own.

Acting on it

Level counts differ (policy has three levels, the others four), so the code first puts every score on 0 to 1 by dividing it by its top level. Then it combines them with the team's weights: solving the problem counts double. A possible policy breach skips the formula and goes to a team lead. Otherwise a low overall quality puts the reply in the agent's coaching queue, tagged with the weakest part.

a = r.json()["answers"]
 
# Put every score on 0..1 by dividing by its top level.
norm = {k: v["score"] / (len(v["legend"]) - 1) for k, v in a.items()}
 
quality = (0.4 * norm["resolved"]
         + 0.2 * norm["policy"]
         + 0.2 * norm["effort"]
         + 0.2 * norm["tone"])          # 0.51 for this reply
 
if a["policy"]["probabilities"]["0"] >= 0.3:
    send_to_team_lead()                  # possible policy breach: always seen by a person
elif quality < 0.6:
    add_to_coaching(agent_id, focus=min(norm, key=norm.get))   # weakest dimension
else:
    log_quality(agent_id, quality)

This reply scores 0.51 overall, below the 0.6 bar. The breach probability is only 0.04, so it goes to coaching rather than to a team lead, tagged with effort, its weakest part. The coaching note writes itself: look for the customer's deadline, and use the express replacement the policy provides.

Writing good levels#

• Order levels from lowest to highest, and make each step a real difference in what you would do.
• Keep each question to one dimension. If a reply can be warm and wrong, tone and correctness are two scores.
• Don't put numbers in the level text ("3 – High"). The position already gives the number, and the description should carry the meaning.
• Four or five levels is usually enough; more levels spread the probability thinner without adding signal.

Concrete levels

Level names like "low" and "high" mean different things to different people. Wity has to guess where your lines are, and answers drift between neighbouring levels. Take a team triaging bug reports:

"criteria": ["Low", "Medium", "High", "Critical"]

Is "export to PDF is broken" medium or high? It depends on whether there is a workaround, which the labels never mention. Describe each level as a situation, and add an example of a report that belongs there:

"criteria": [
  "Cosmetic: a typo or layout glitch. Everything still works. E.g. 'button text is cut off on mobile'",
  "A feature misbehaves but there is a workaround. E.g. 'export fails as PDF, CSV works'",
  "A feature is unusable for some customers, with no workaround. E.g. 'SSO users can't log in'",
  "Data loss, a security hole, or the product is down for everyone"
]

Now the line between levels is a fact Wity can check (is there a workaround? who is affected?) rather than a feeling. The examples show what a typical report at each level looks like, so it is best to take them from real reports.

Scores as forecasts#

Levels can be buckets of a future quantity: "how many days late will this shipment be?" with levels "on time", "1–2 days", "3–7 days", "more than a week". The answer is the chance of each bucket. Put the numbers that drive the outcome in the state. See Forecast questions.