Writing good questions

How Wity reads a question#

Every question has the same three ingredients, and each one has a job:

Most accuracy problems come from one of these three being thin: evidence left out of the state, a question that could mean two things, or options that are just labels. The worked example below fixes each in turn.

A worked example#

An online clothing shop wants to handle return requests automatically. For each customer message, the returns service needs to know what to offer: a refund, an exchange, a repair, or nothing. Here is a first attempt at the question, and a message it gets wrong.

{
  "state": "Hi, the jacket I ordered arrived last week but the zip broke the second time I used it. I'd like my money back please, not another one.",
  "questions": {
    "resolution": {
      "type": "choice",
      "instructions": "Resolution?",
      "criteria": {
        "refund": "Refund",
        "exchange": "Exchange",
        "repair": "Repair",
        "reject": "Reject"
      }
    }
  }
}

The answer is spread out. The customer asked for their money back, so refund leads, but exchange and repair both get real weight, and nothing tells Wity whether the shop is obliged to refund at all. (The probabilities on this page are illustrative.)

Nothing is wrong with the model here. The question simply does not contain enough to decide:

• The state has no policy and no dates. Whether a broken zip earns a refund depends on the returns window and on when the order arrived. Wity can only guess at both.
• "Resolution?" could mean two things. What the customer asked for, or what the shop should offer? These often differ, and here the answer depends on which you mean.
• The options are just names. "Repair" is a reasonable reading of a broken zip. Nothing says repairs are only for items past the refund window.
• There is no way out. A message too vague to apply the policy still has to land on one of four answers.

The fixed question

Each problem gets one fix:

• The state becomes an object with the message, today's date, the order (including the delivery date) and the relevant policy clause.
• The instructions say whose decision this is: what should we offer, under the policy.
• Each description says when the option applies, including the edge that separates it from its neighbour: repair only after 30 days, refund for faulty items inside the window.
• A needs_review option gives an honest landing place when the evidence is not there.

{
  "state": {
    "message": "Hi, the jacket I ordered arrived last week but the zip broke the second time I used it. I'd like my money back please, not another one.",
    "today": "2026-09-25",
    "order": { "item": "Trail jacket", "delivered": "2026-09-17", "price_eur": 129 },
    "policy": "Faulty items: refund or replacement within 30 days of delivery, customer's choice. Change of mind: 14 days, unused items only."
  },
  "questions": {
    "resolution": {
      "type": "choice",
      "instructions": "Under the policy, what should we offer the customer?",
      "criteria": {
        "refund": "Money back. The item is faulty within 30 days and the customer asks for money, or it is an unused change-of-mind return within 14 days.",
        "exchange": "A replacement or a different size or colour. The customer asks for one, or accepts one.",
        "repair": "Fix the item. Only when it is faulty after 30 days and still under warranty.",
        "reject": "Nothing is owed: outside every window, or the damage was caused by the customer.",
        "needs_review": "The message or order data is not enough to apply the policy."
      }
    }
  },
  "reasoning": "auto"
}

Now the answer is concentrated. The jacket arrived eight days ago, the zip is a fault, and the policy gives the customer the choice. They chose money back, which is exactly what the refund description says.

The same question now also handles the harder cases correctly. The same message 40 days after delivery moves to repair, because the description says that is when repairs apply. A message that only says "I want to send this back" with no reason lands on needs_review instead of a guess.

Write the instructions like a brief#

Instructions are the question itself. Write them for a capable colleague who can see the state but was not in the meeting where the process was designed.

• Say whose view you want. "What does the customer want?" and "What should we offer?" are different questions with different answers.
• Name the rule to apply. "Under the policy" tells Wity to read the policy in the state, not to use general sense about returns.
• Say what to ignore. If the state holds the whole ticket thread but only the latest message matters, say so: "Judge from the customer's most recent message."
• Keep it to one sentence where you can. Long instructions are fine (up to 4,000 characters), but definitions belong in the option descriptions, where each option can own its own.

Describe when each option applies#

An option's description is its definition. A good one says what makes the option true, where it stops, and, for options that are easily confused, a short example.

// weak
"criteria": { "urgent": "Urgent", "normal": "Normal" }
 
// strong
"criteria": {
  "urgent": "Production is stopped, safety is at risk, or a customer deadline is within 24 hours",
  "normal": "Everything else, including degraded but working equipment"
}

Look at the strong normal: it names the case people argue about ("degraded but working") and puts it on one side of the line. Every boundary you leave unstated is one Wity has to guess.

Make options exclusive and complete#

• Exclusive: each real case should fit exactly one option. If two can both be true (a message can be both faulty and late), they are separate judgements: ask two questions.
• Complete: add a catch-all (other, not_stated, needs_review) whenever a real input might fit nothing. Without one, Wity must pick the least-wrong option, and its probability will look more certain than the case deserves.
• Honest about missing information: when the state may not contain the answer at all (was it reported to the police?), a not_mentioned option turns "cannot tell" into a clear answer you can act on.

Put the evidence in the state#

Wity decides from the state you send. If a person would need to look something up to answer, put it in the state: the policy clause, the threshold, the dates, the customer's history. For forecasts, include the numbers.

• Use an object when there are several sources. message, order and policy as named fields are easier for Wity to tell apart than one pasted block.
• Send only what the decision needs. Every question is billed with the state it reads (see Billing), and irrelevant text is one more thing that can pull an answer the wrong way. The one policy clause beats the whole terms and conditions.
• Include the date when time matters. "Within 30 days" means nothing without both the delivery date and today's date in the state.

One judgement per question#

"Is this a refund request over €100 from a new customer?" hides three things: what the customer wants, how big the order is, and how new the customer is. Only the first is a judgement. The other two are facts your code already knows. Ask Wity only for the judgements, then combine everything in code.

{
  "state": { "message": "…", "customer": { "orders": 1, "since": "2026-09-02" }, "order": { "total_eur": 180 } },
  "questions": {
    "wants_refund": {
      "type": "noul",
      "instructions": "Is the customer asking for their money back?"
    },
    "reason": {
      "type": "choice",
      "instructions": "Why is the customer returning the order?",
      "criteria": {
        "faulty": "Broken, defective or not working as described",
        "wrong_item": "We sent a different item, size or colour than ordered",
        "changed_mind": "Nothing is wrong with it; they no longer want it",
        "not_stated": "The message does not say why"
      }
    },
    "frustration": {
      "type": "score",
      "instructions": "How frustrated is the customer?",
      "criteria": ["Calm", "Mildly annoyed", "Clearly frustrated", "Angry or threatening to leave"]
    }
  }
}

Each question is small and checkable on its own, and all three come back from one round trip. The order total and the customer's history stay in the state as evidence, but the comparisons with €100 and one order happen in your code, where they are exact:

a = r.json()["answers"]
state = payload["state"]
 
refund = a["wants_refund"]["noul"] >= 0.8
new_customer = state["customer"]["orders"] <= 1
large = state["order"]["total_eur"] > 100
 
if refund and new_customer and large:
    send_to("fraud_review")
elif refund and a["reason"]["choice"] == "faulty":
    auto_refund()
else:
    send_to("returns_queue")
 
if a["frustration"]["score"] >= 2:   # expected level: 0 = calm … 3 = angry
    set_priority("high")

This also makes failures easier to fix. If refunds are being misrouted, you can see which of the three answers was wrong instead of wondering which part of one compound question Wity misread.

Pick the primitive that fits the answer#

Start from what your code does next, and choose the primitive whose answer it can use directly:

A common mistake is a choice question with low, medium and high options. It works, but it throws away the order: a 0.5 / 0.5 split between low and high looks the same as one between low and medium. As a score question, the first gives an expected level in the middle and the second one near the bottom, which is what you want to threshold on.

Read spread as feedback#

Probabilities tell you how the question is working, not just how the case came out. When you look at answers in bulk, three patterns point at the wording:

• The same two options keep splitting. Their descriptions overlap. Find a case that splits and write the boundary into both descriptions.
• The catch-all wins too often. Either the state is missing evidence, or a common case has no option of its own. Read a sample of those cases and add what is missing.
• Confident and wrong. The description says something different from what you mean. Wity is applying the definition it was given; change the definition.

Some spread is genuine. Tone and intent are often ambiguous in the message itself, and a 0.6 / 0.3 answer is the honest one. For those, act on sums of options (frustrated plus angry) rather than the top one. See Probabilities & confidence.

Keep a small test set#

Wording changes are code changes: they can fix one case and break another. Before you change a question, write down twenty or thirty real cases with the answer you expect. Include the easy ones, the ones that went wrong, and the edge cases your descriptions now mention. Run them after every change.

The script below runs a test set against one question. It prints the score and every case that was wrong or close (the top two options less than 0.3 apart), so you can see what a rewording changed.

import json, requests
 
# One case per line: {"state": ..., "expected": "refund"}
cases = [json.loads(line) for line in open("resolution_cases.jsonl")]
 
right, close = 0, []
for case in cases:
    body = {"state": case["state"], "questions": {"resolution": QUESTION}, "reasoning": "auto"}
    ans = requests.post(URL, headers=HEADERS, json=body, timeout=60).json()["answers"]["resolution"]
    p = sorted(ans["probabilities"].values(), reverse=True)
    right += ans["choice"] == case["expected"]
    if ans["choice"] != case["expected"] or p[0] - p[1] < 0.3:
        close.append((case["expected"], ans["choice"], round(p[0], 2), round(p[1], 2)))
 
print(f"{right}/{len(cases)} right")
for row in close:
    print("expected %-12s got %-12s top %.2f  runner-up %.2f" % row)

Look at the close cases as well as the wrong ones. A rewording that keeps the score the same but turns a 0.55 answer into a 0.9 one has made the question sturdier, and your thresholds will let more of those cases through without a person. The Playground shows the expected answer next to each preset for the same reason.

Checklist#

• The state holds every fact a person would need, including dates and the rule to apply, and little else.
• The instructions say whose view you want and which rule to use.
• Every description says when its option applies and where it stops.
• Options are exclusive, and there is a catch-all whenever a real case might fit none.
• Each question asks for one judgement; facts and arithmetic happen in your code.
• Ordered answers use score, yes/no uses noul.
• A test set runs after every wording change.