Tuning Laya's Option Text Against a Labelled Ticket Set
Updated:
The local Laya and Ollama pipeline routes a support ticket three ways: an automatic reply, a human, or no reply at all. Laya makes that choice from three plain-English descriptions. On the tickets I tried first, it picked escalate_to_human every time, including a password reset and an obvious spam message. The 70% confidence gate in that version stopped anything wrong going out to a customer, and it also meant the automatic-reply route never ran.
The option text is part of the model’s input, so I treated it like code and tested it. This post is that test: 50 messages with a known right answer, ten wordings of each option, and the counts of what Laya chose against what the message actually was. laya_demo.py now uses the wording that won, and it routes on the highest probability instead of that gate.
The labelled set
Each message is one customer email, labelled by hand before any search:
| Label | Messages | What it covers |
|---|---|---|
auto_reply | 17 | A documented how-to: password reset, pairing, firmware, order lookup, a published policy |
escalate_to_human | 17 | A person has to act: a fault, a damaged or wrong order, a refund, safety, an account compromise |
ignore | 16 | Not a customer with a problem: spam, phishing, a sales pitch, recruitment, nonsense |
The original descriptions, the ones in laya_demo.py when this search started, are the first wording of each option (a0, e0, i0):
"auto_reply": "The issue is simple and can be addressed automatically.",
"escalate_to_human": "The hardware issue or replacement request needs a human agent.",
"ignore": "The message is spam or completely irrelevant."The other nine wordings per option are different hypotheses about the boundary, not paraphrases of that sentence. One names concrete FAQ tasks. One lists faults and refunds. One is just “Not a customer support request.” Each stays a single sentence, because Laya keeps only the first 48 tokens of an option.
The search score for a message is Laya’s answer_confidence when the chosen action matches the label, and 0 when it does not. That average rewards being right and being sure. Accuracy and the probability on the true label break ties. A full grid of the ten wordings is 1,000 combinations, and one combination takes about 18 seconds on CPU, so the grid is a few hours. I used coordinate search instead: try every wording of one action while the other two stay fixed, then repeat. It settled on round 3, after 64 combinations and about 22 minutes of model time. That is not a proof that some untried combination in the other 936 is worse. The top two that were tried scored 0.438 and 0.436, so a large hidden gain is unlikely, and a small one is still possible.
What the original wording does
Rows are the true label. Columns are the action Laya chose. The diagonal is the messages it got right.
| Actual | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 0 | 17 | 0 |
| escalate_to_human | 0 | 17 | 0 |
| ignore | 4 | 10 | 2 |
Nineteen of 50 are right, all of them either a real escalation or one of two ignores. Every how-to is sent to a human. Fourteen of the 16 non-tickets are treated as a customer. Credited confidence is 0.279. Twelve messages clear a 70% gate, and every one of them is an escalation. The gate is doing what it was built for: it refuses to act when the choice is unsure. It cannot create an auto_reply decision that the option text never wins.
The best wording
Coordinate search landed on one sentence each:
"auto_reply": "A how-to or FAQ: password reset, Bluetooth pairing, order tracking, firmware, or account settings.",
"escalate_to_human": "Broken hardware, a refund, a safety issue, or an account takeover.",
"ignore": "Not a customer support request."Same 50 messages:
| Actual | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 14 | 2 | 1 |
| escalate_to_human | 0 | 14 | 3 |
| ignore | 3 | 0 | 13 |
Forty-one of 50 are right. Credited confidence is 0.438. The original wording was rank 50 of the 64 combinations tried. The off-diagonal cells are the whole remaining problem, nine messages:
- Two how-tos called an escalation. “Where is the tracking number on the website?” written in a frustrated tone, and “What hours is customer support open?”
- One how-to ignored. “What is your returns window? I’m not sending anything back, I just want the policy.” The refusal in the sentence outweighs the question.
- Three real cases ignored. Someone changed the password and placed an order. A speaker hisses with nothing playing. “Cancel order 88210 and refund me.” The escalate sentence names a refund and an account takeover, and these three still lose.
- Three non-tickets given an automatic reply. An SEO pitch, a phishing mail that says “verify your mailbox password”, and a company offering to buy returns in bulk. The auto-reply sentence itself says “password reset”, and the phishing mail matches those words.
Correct decisions now sit around 0.51 to 0.58 confidence. One of the 50 clears 70%. The gate from the first post, applied to this wording, almost never lets a route run. It was useful while the old wording piled about 0.76 onto escalate for everything. Left in place, it would also block the 41 calls this wording gets right.
Averaging the probabilities of the nearest strong wordings stayed at 40 or 41 out of 50 and repeated these same nine mistakes. They disagree on messages the winner already gets right.
What the demo does now
laya_demo.py uses those three sentences. The 70% check is gone. The script takes the highest-probability action and runs that route, including a call whose confidence is in the 0.5s.
"criteria": {
# Wording from the labelled search (a1 / e3 / i9). One short sentence each:
# Laya keeps 48 tokens of an option, and three options stay in the calibrated bucket.
"auto_reply": "A how-to or FAQ: password reset, Bluetooth pairing, order tracking, firmware, or account settings.",
"escalate_to_human": "Broken hardware, a refund, a safety issue, or an account takeover.",
"ignore": "Not a customer support request."
}# 'answer_confidence' is max(p), the calibrated value.
# 'confidence' is normalised entropy over all options and is NOT calibrated.
# Route on the highest-probability action. With this wording a correct call usually
# sits around 0.5-0.6, so a 70% gate withheld every route.
answer_confidence = action_data.get("answer_confidence", 0)if chosen_action == "auto_reply":
print("Action triggered: Invoking Ollama to draft an automatic customer reply...\n")
elif chosen_action == "escalate_to_human":
print("Action triggered: Escalating to a human agent.\n")
else:
print("Action evaluated as 'ignore'. No response generated.")The holding reply and the two Ollama prompts are unchanged. What changed is which action reaches them. A how-to that wins at 55% now gets the automatic reply. A message whose highest probability is ignore is ignored, rather than held for a human because it missed 70%. The nine misses in the matrix above are the ones that still go the wrong way.
Piling the ten wordings into one option
Joining all ten sentences for an action, and scoring that single trio, got 17 of 50 right. Credited confidence fell to 0.159. Auto-reply was 0 of 17, escalate 9 of 17, ignore 8 of 16.
Laya renders each option as label: description and keeps 48 tokens. The joined texts were 152, 161 and 136 tokens. The model saw the original sentence plus the start of the next one. The winning escalate sentence and the winning ignore sentence both sat past the cut, so this run never read them. A longer description is not a richer description once it exceeds that cap.
Ten separate labels per action, summed back into one class afterwards, runs into the same budget from the other direction. All options share a 192-token head. Thirty of them are cut to about four tokens each, so the descriptions go and often the label does too. Eleven or more options also use the choice:11+ temperature, which this checkpoint clamps and tells you to treat as uncalibrated. I did not spend a run on that 30-option choice. Thirty separate yes/no questions, one sentence each, are a different shape, and that one is below.
A second question
The nine misses fall into clusters, so I tried a second 3-way question on the messages the first question had already called auto_reply. The idea was to separate “the customer wants steps” from “the sender wants us to buy, rank, partner, or enter a password or card.” Seventeen messages took that path. The three solicitations were among them, so the best possible score was 44 of 50.
Both wordings left those three as auto_reply. The SEO pitch went to 85% automatic reply. The first wording also moved “How do I turn off marketing emails?” from a correct automatic reply to ignore, and the score fell to 40 of 50. A sharper ignore sentence, naming SEO and password links explicitly, moved five real how-tos to ignore and scored 36 of 50.
A second question that double-checks ignore did no better. Seventeen messages, including the four that should not have been ignored. The re-check left all four as ignore, and it moved a gift-card scam and a “win a free phone” mail to auto_reply. Accuracy went to 39 of 50.
The errors that survive the first wording are stable. Asking the same three actions again, with a different sentence, does not pull them apart.
Scoring the mistakes differently
Accuracy treats every miss as equal. In this pipeline they are not. An automatic reply when a human was needed still sends the customer something. Ignoring that same case drops it. Escalating a how-to or a piece of spam costs a person’s time.
rated_score.py uses this cost. The row is the true class. The column is the action taken.
| True class | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 0 | 1 | 2 |
| escalate_to_human | 1 | 0 | 3 |
| ignore | 2 | 1 | 0 |
Lower is better. On the 50 messages the original wording costs 35: seventeen how-tos escalated at 1 each, four non-tickets given an automatic reply at 2 each, and ten non-tickets escalated at 1 each. The tuned wording, taking the highest probability, costs 19. Nine of those points are the three real cases in the ignore column of the second confusion matrix, at 3 each.
The same probabilities can pick the action with the lowest expected cost, treating each probability as the chance that class is the true one. That corrects the three dropped cases. It also escalates ten how-tos and thirteen non-tickets:
| Actual | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 7 | 10 | 0 |
| escalate_to_human | 0 | 17 | 0 |
| ignore | 1 | 13 | 2 |
Accuracy falls to 26 of 50 and the cost rises to 25. Needless escalation costs 1, and these probabilities usually sit between 0.3 and 0.5, so the cheap action wins whenever the model is unsure. The cost matrix is a good way to read a confusion matrix. It is a poor replacement for taking the top probability, at least with this gap between “drop a case” and “ask a human”.
One limit on all of these figures: the 50 messages are the set the wording was chosen on. Forty-one out of 50, and cost 19, are the training score of a1 / e3 / i9. The next two sections stay on that set. A second batch of 50, not used to pick a sentence, is scored after that.
Combining a second decision
I reloaded convaiinnovations/laya 0.3.20 and asked the tuned three-way question again. Every choice and every probability matched the search file. Cost 19, 41 of 50, reproduced rather than assumed. Asking that same question a second time produced the same choice and the same probabilities on all 50 messages. A majority over two samples has nothing to vote on. It still costs 19, and it spends two calls per message.
The second questions from earlier in this post asked for the same three actions in different words. This pass asks two questions that are not a copy of the route. The strings are in experiments/questions.py.
"instruction": "Who is writing, relative to this shop?",
"criteria": {
"customer": "A person writing about their own order, account, or device.",
"outsider": "Spam, a pitch, recruitment, or someone else's notice.",
"unclear": "Not enough to tell whether the sender is a customer.",
}"instruction": "What kind of help does the sender need?",
"criteria": {
"steps": "Steps, or the wording of a published policy. Nothing has failed.",
"staff": "A fault, a refund, a cancellation, a replacement, or an unsafe account.",
"none": "No help with their own order, account, or device.",
}need answered none on all 50 messages. A fixed map that ignores anyone need calls none therefore ignores every ticket. Cost 85. Accuracy 16 of 50, which is just the 16 non-tickets.
| Actual | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 0 | 0 | 17 |
| escalate_to_human | 0 | 0 | 17 |
| ignore | 0 | 0 | 16 |
A vote of the demo choice plus those two specialists does better than that and still loses. Because need is none everywhere, ignore starts with a vote on every message. Cost 45, 30 of 50. Ten how-tos the demo got right flip to ignore.
Using the specialists only to repair the expensive mistakes, with no confidence gate, also loses. need never says staff, so a dropped case is never pulled back to a human. Ownership says outsider often enough, including on real how-tos, that ten correct automatic replies become ignores. Cost 37, 32 of 50.
Fitting a threshold on the same 50 changes one message. The grid is 0.00 to 1.00 in steps of 0.05. The fit is t_own = 0.60 and t_need = 1.01. t_need cannot matter, because need never varies, and the tie-break then picks the highest threshold. Reapplied, m37 moves. It is a real non-ticket. The demo called it auto_reply at 0.4765, and ownership called it outsider at 0.6454, which clears 0.60. That miss cost 2 and now costs 0. Frozen cost 17, 42 of 50. Leave one message out and refit: forty-nine folds still choose 0.60, and the fold that holds out m37 chooses a threshold that never fires. Leave-one-out cost is 19. Leave-one-out accuracy is 41 of 50. The two-point saving exists only when the threshold is allowed to see m37.
The 70% gate, recomputed on these same probabilities, costs 33 and gets 17 of 50. Every message is escalated. The one score that clears 70% is m26, already an escalation at 0.8525, so the gate never opens another route.
Among these choice rules, the cost I can claim on the development set is still 19, from one call, the tuned wording, and the highest probability.
Thirty yes/no scores
Each of the thirty sentences can also be its own noul question: “Does this description match the customer message?” The true option is that sentence. The false option, on every question, is “This description does not match the customer message.” A cell is the probability that the sentence matches. The thirty cells do not compete, and a row does not sum to 1. Scoring the matrix is 1,500 questions, 30 per message. On this CPU the short wording took 466.6 seconds and the longer wording took 478.4.
Two rules read that matrix. The highest cell wins, and its group is the action (a is automatic, e is escalate, i is ignore). Or sum the ten scores in each group and take the largest total.
On the short sentences, the original one-line wordings:
| Rule | Cost | Accuracy |
|---|---|---|
| Highest cell | 19 | 40/50 |
| Sum of each group | 10 | 44/50 |
The highest cell ties the tuned choice on cost and is one message worse on accuracy. Five how-tos are ignored. On m04, “How do I change the email address on my account?”, the top cell is i1 at 0.7935, an ignore sentence about spam and phishing. a1, the winning how-to sentence, is 0.2164 on that row.
The group sum is the lowest cost I have measured on these 50. Every real case is an escalation. The six misses are five how-tos and one non-ticket. It is thirty yes/no questions per message, not one 3-option call, and the columns were not trained as a three-way decision. The sentences were written for this set. Cost 10 is a training score, the same kind of figure as cost 19.
A noul true option is still cut at 48 tokens. It is rendered as true: plus the sentence, and that prefix is 3 tokens, so the sentence itself can use about 45. A choice option spends more of the same 48 on the action label. escalate_to_human: is 8 tokens. I rewrote each of the thirty sentences to fill the noul budget and kept the short line as backup. The same two rules on the longer text:
| Rule | Cost | Accuracy |
|---|---|---|
| Highest cell | 74 | 19/50 |
| Sum of each group | 33 | 33/50 |
An ignore sentence wins 14 of the 17 how-tos and 14 of the 17 real cases under the highest-cell rule. Filling the spare tokens made both rules worse. The short lines are the ones that scored 10.
A second set of 50
data/heldout_messages.json is a new batch, written after the wording and the thresholds were frozen. Seventeen how-tos, seventeen real cases, sixteen non-tickets. None of the sentences are copied from the first file. The labelling rule is the same. The file was not used to pick a sentence.
The rule I ran is the short-wording group sum. experiments/heldout_group_sum.py reads the backup sentences, asks the same yes/no question, and refuses the original 50. 1,500 scores, 366.6 seconds. No group totals tied.
Cost 16. Accuracy 40 of 50. The development cost of 10 did not hold. One real case was ignored, h28, a subscription the website would not cancel, at cost 3. Four how-tos moved. Five non-tickets were answered.
| Actual | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 13 | 2 | 2 |
| escalate_to_human | 0 | 16 | 1 |
| ignore | 2 | 3 | 11 |
Two of the non-ticket misses are three totals within a few thousandths of each other. h37, a fake warranty renewal that asks for a card, has automatic 5.0025, escalate 5.0013, ignore 4.9987, so the automatic total wins. h49, a password-phishing link, is the same kind of three-way near tie, and escalate wins. The rule still takes the highest total. Both are wrong.
The tuned three-way choice, the rule in laya_demo.py, has now been run on this second file. experiments/apply.py asks only that question, one call per message, and does not refit a threshold. Loading took 13.1 seconds. The pass took 21.6 seconds. The options were inside the 48-token cap.
Cost 37. Accuracy 32 of 50. On the first 50 this question costs 19 and gets 41 right. On this file the short-wording group sum costs 16 and gets 40 right, from thirty yes/no questions rather than one call.
| Actual | auto_reply | escalate_to_human | ignore |
|---|---|---|---|
| auto_reply | 12 | 1 | 4 |
| escalate_to_human | 1 | 10 | 6 |
| ignore | 3 | 3 | 10 |
Eighteen messages go the wrong way. Six of them are real cases ignored, at 3 each, and those 18 points are about half the cost. They are a scratched speaker left in a bin, a protection plan that was never selected, a speaker that fell into a bath while plugged in, an account with a new overseas address and a pending order, a subscription the website would not cancel, and a trading-standards letter about the battery claim. The subscription, h28, is ignore at 0.7635. The group sum ignored it as well. The escalate sentence names broken hardware, a refund, a safety issue, or an account takeover, and the bath and the overseas order are in that list and still lose.
Four how-tos are ignored, one how-to is escalated, and one short-delivery complaint is given an automatic reply. Three non-tickets get an automatic reply: a bulk offer for broken speakers, a podcast sponsorship, and a prize that asks for a bank card. The prize is auto_reply at 0.6357. Three other non-tickets are escalated, including the fake warranty renewal and the password-phishing link.
The same decision on Jev
Jev is TypeSafe’s hosted model for this kind of question. Laya is the open-weight version of the same idea: a state, a choice with criteria, and probabilities back. laya-serve answers POST /v1/systemone, which is Jev’s endpoint, so a client can aim at either by changing the base URL. I have not sent these 50 messages to Jev. The call below is the documented request, filled in with this demo’s instruction and the three winning sentences.
| Jev | Laya in this demo | |
|---|---|---|
| Where it runs | TypeSafe’s API, https://api.typesafe.ai/v1/systemone. The ticket leaves the machine. | convaiinnovations/laya on this machine. Apache 2.0. The ticket stays local. |
Options in one choice | Up to 255. | They share a 192-token head, and each option keeps 48 tokens. Eleven or more land in the bucket this checkpoint treats as uncalibrated. |
| How much text | About 64k tokens for the request. State plus the longest question can run to 32k. | 512 tokens on the English checkpoint, with about 320 left for the ticket. |
| One question | Independently measured around 236–276 ms median, including the network. TypeSafe’s launch note says 70–500 ms. Neither figure is an SLA. | About 40 ms on a T4 once the weights are loaded. This demo is on CPU, and one pass over the 50 messages takes about 20 seconds. |
| Price | $0.042 per million input tokens. Output tokens are free. | The compute already on the machine. The coordinate search was about 22 minutes of CPU. |
| What you can change | The criteria, and which model version you pin. The weights stay closed. | The criteria, and a fine-tune. The demo loads the base English checkpoint. |
On the typed-decisions benchmark (2,000 decisions, four workflows), Jev 1.13.0 is published at 0.727 accuracy, 0.580 soft accuracy, and 0.144 calibration error. The base English Laya checkpoint, the one this demo loads, scores 0.362 on that set, under a majority-class baseline of 0.461. laya-typed-decisions, fine-tuned on that benchmark’s own training split, scores 0.766. Laya’s model card says the Jev column is copied from published results and was not re-measured on the same prompts. The 0.766 is a different checkpoint from laya_demo.py.
A wide label set is where those budgets diverge. Banking77 is 77 intents. Jev scores 0.870 on 72 of those labels. Laya scores 0.425 at the default head budget, because each label is left with a few tokens. This router has three labels. The first confusion matrix failed on the sentence, with every how-to sent to a human, and the option count was already small enough to fit.
The Jev call keeps Ollama for the customer email and the handoff note. Jev only picks the action. Pin jev-1.13.0 rather than jev-latest if a threshold has to stay still, because the docs say the moving alias can change behaviour.
answer = requests.post(
"https://api.typesafe.ai/v1/systemone",
headers={"Authorization": f"Bearer {os.environ['TYPESAFE_API_KEY']}"},
json={
"model": "jev-1.13.0",
"state": ticket,
"questions": {
"action": {
"type": "choice",
"instructions": "What action should be taken for this customer support ticket?",
"criteria": {
"auto_reply": "A how-to or FAQ: password reset, Bluetooth pairing, order tracking, firmware, or account settings.",
"escalate_to_human": "Broken hardware, a refund, a safety issue, or an account takeover.",
"ignore": "Not a customer support request.",
},
}
},
},
timeout=30,
).json()["answers"]["action"]
chosen_action = answer["choice"]choice is the option with the highest probability, as it is in Laya, and answer["probabilities"] is the distribution. The three branches in the demo stay as they are: draft a reply, send the holding reply plus a handoff note, or print nothing.
The confidence field is a different number. Laya’s answer_confidence is the top probability, and with this wording a correct call sits around 0.51 to 0.58. Jev’s confidence falls when that probability is spread across the options and rises when one option peaks. A 70% check on one field is a different rule from a 70% check on the other. I would score a labelled batch before choosing a cutoff. That is what pushed the 70% gate out of this demo, once the right answers landed in the 0.5s.
The wording search would also shrink. Joining all ten sentences scored 17 of 50 here because Laya cut each option at 48 tokens and the winning escalate and ignore lines sat past the cut. Jev’s documented budget is the 64k request, and one choice holds 255 options, so those sentences would be part of the input. The nine misses were stable across nearby wordings, and a longer rubric can still draw the boundary in the wrong place. The labelled set, the cost matrix, and the confusion matrix are the parts worth keeping. The parts tied to this checkpoint are the short sentence, the three-option limit, and a coordinate search built around an 18-second CPU call. On Jev those 50 messages are one API call each. The open measurement is whether the top probability is right, and whether confidence is high on the right rows and low on the nine misses.
Things to Know
The option text has a hard length. Forty-eight tokens per option, and 192 for the whole head at the default settings. I found the winning sentences by keeping each one short, then confirmed the failure mode by joining all ten and watching the winners get truncated. If a description matters, it has to fit in the first 48 tokens, not after them.
A follow-up question needs a different job. Both second questions asked for the same three actions in different words. The solicitations still looked like how-tos, and the dropped cases still looked ignorable. The follow-up also only sees the messages the first question already routed that way, so it cannot fix a how-to that was escalated.
Flat probabilities make a cost rule collapse. The three dropped cases are exactly the ones a cost-aware rule should save, and it does save them. It saves them by escalating almost every close call. The original wording plus the 70% gate was already a version of that policy: when unsure, send it to a person. The demo now takes the other side. It trusts the top probability of the tuned wording, so a how-to at 0.55 gets an automatic reply, and those three dropped cases stay dropped.
The choice:11+ warning still matters. The first post’s three-option question is in the calibrated 3-to-5 bucket. Anything I considered with eleven or more labels is the bucket Laya warns about on startup. I kept every measured choice at three options. The thirty-sentence run is thirty separate noul questions, two options each, not one 30-way choice.
A second question can fail by never varying. need answered none on all 50 development messages, so every rule that trusted it ignored tickets or started ignore with a vote. A different job on paper is not a different job in the scores.
Filling the token budget is a different change from finding a better sentence. The longer noul lines use the spare tokens past the short wording. Highest-cell cost went from 19 to 74, and the group sum went from 10 to 33.
A threshold fit on the reported set can be one message. t_own = 0.60 saves m37 and nothing else. Leave that message out of the fit and the saving disappears. The number I keep is the leave-one-out cost, 19, not the reapplied cost, 17.
Key Takeaways
- Test option text the way you test code. A labelled set of 50 turned “it always escalates” from a hunch into a 0-of-17 count, and showed which sentence moved it.
- One short sentence beats a stack of sentences when the model truncates the option. The text past 48 tokens is not a soft hint. It is absent.
- Read the off-diagonal, not just the accuracy. 41 of 50 hides three dropped cases and three solicitations that received a customer reply. Those are different failures.
- A cost matrix is for scoring a confusion matrix. Using it to pick the action, while probabilities are spread across all three classes, buys the cheap action on every close call.
- Don’t retune on the same messages you report. The wording was chosen on the first 50. Cost 19 is that training score. On a second 50 the same three-way question costs 37 and gets 32 of 50. The group-sum rule, which cost 10 on the first file, costs 16 on the second.
- Repeating the same prompt does not make a vote. Two calls of the tuned question matched on every message. The extra call buys nothing.
- A confidence gate has to sit where the scores actually land. 70% matched the old wording, which piled about 0.76 onto escalate. The tuned wording’s correct calls sit around 0.5 to 0.6, so the same number blocks the routes it just made possible.
- The same question can hide a different confidence field. Laya’s usable number on this 3-option call is the top probability. Jev’s
confidencedescribes how peaked that distribution is. Copying a 70% cutoff from one field onto the other changes the rule.
The Code
The demo is pyardley/laya-ollama-demo, written up in A Local Decision + Generation Pipeline with Laya and Ollama. laya_demo.py now uses the three winning sentences and routes on the highest-probability action. The search that picked them lives next to it:
tune_criteria.pysearches the wordings and writescriteria_search_results.jsondata/customer_messages.jsonholds the 50 labelled messagesdata/criteria_variants.jsonholds the ten sentences per actionsecond_pass.pyre-asks messages already classified asauto_replyrated_score.pyholds the cost matrix, the expected-cost decision, and the ignore re-checkexperiments/EXPERIMENTS.mdis the log of the combined-decision runs and thenoulsumsexperiments/run_all.pyscores thechoicerules on the original 50 and refuses any other fileexperiments/questions.pyholds the demo question plusownershipandneednoul_matrix.pyscores every wording as a yes/no against every development messageexperiments/heldout_group_sum.pyruns the short-wording group sum on a new filedata/heldout_messages.jsonis the second set of 50
There is no Jev client in the repo. The request in the Jev section is TypeSafe’s documented POST /v1/systemone shape, with this demo’s instruction and criteria filled in.