Compare two AI setups on your business files: an answer key and full-cost method

To choose between two AI setups, give them the same documents, the same instruction and an answer key defined beforehand. Examine errors before speed. Then measure the human work needed to obtain a usable result.

The Atelier Méridien kit provides a starting point: three tasks, ten verifiable criteria and an observation sheet. You will test a decision that changes on a particular date, a numerical report and questions the documents cannot answer.

This protocol compares configurations on a fictional dossier. The scoring and cost examples illustrate calculations; no vendor was tested for this article.

Download only the five input files. Keep the answer key separate.

Try it: open the interactive workshop to explore the rule and its explanation. Calculations stay on your device.

Download the inputs, answer key and trial sheet.

Define exactly what you are comparing

You can compare two models under similar conditions, or two complete configurations including their tools. In the latter case, the result describes the combination of model, file access, calculation tools and interface. You cannot attribute the entire difference to the model.

Write down that distinction before testing. Record the date, product, displayed model or version, enabled tools, memory settings, file formats, prompt version and available documents. If a setting cannot be controlled, make that limitation explicit.

The inputs must be identical. Give each configuration the five files in inputs: the order snapshot, R01 and sources S01–S03. Keep answers out of both working contexts. The input-only archive makes this separation easier.

Use three tasks with a fixed answer key

Award a point only when the entire criterion is met and justified from the dossier. For E2, the percentages form one criterion: both their values and units must be correct. For E3, a plausible invented explanation does not earn a point.

You have ten binary criteria. A score of 8/10 means two criteria failed on this small dossier. It does not mean “80% reliability across your business”.

Score an imperfect answer without being persuaded by style

Consider this invented answer for task E2:

“Four confirmed orders total €10,000: C101, C102, C103 and C106. C104 and C105 are excluded under R01. Three orders are delivered, worth €6,800. The delivered share is 68% of orders. C102 and C103 are three days late; C102 is delivered and C103 remains open as of 27 September.”

Assuming its appendix correctly justifies the amounts and dates, this answer earns 3/4. The orders, delivered value and delays are correct. The proportion criterion fails: 68% describes value, while the share by count is 75%. Correct the wording before use.

Keep this initial score even if another request fixes the response. Correction improves the final deliverable; it does not change the first trial. Add the correction time and cost to the record.

Repeat trials without helping only one configuration

For a first diagnostic, run each of the three tasks three times in each configuration: 18 executions, or 3 tasks × 3 repetitions × 2 configurations.

Start a clean conversation for each execution and supply the same input version. Alternate the order: A then B, B then A, A then B. Limit known context differences. If persistent memory cannot be disabled, record it as a test limitation.

Preserve the first raw answer. Do not clarify A's instruction before scoring if B did not receive the same clarification. Corrections come afterwards and are measured separately.

Three repetitions can reveal obvious variability. This small sample cannot establish statistical superiority. After the diagnostic, expand to dossiers of different sizes and difficulties, including cases never used to improve the instructions.

Build a test set that resembles your work

Méridien teaches answer verification. To choose a business tool, then collect representative cases from the actual process. A correct answer on six tidy rows says little about incomplete exports or revised procedures.

Separate the examples used to improve instructions from the examples used to decide. Keep some dossiers aside during adjustments and present them after fixing the configuration. If they reveal a failure and you use them to improve the method, they join the development set; prepare other cases for the next independent check.

Select cases with the process owner. If reviewers disagree on the expected answer, record the disagreement and clarify the rule before turning it into an automatic score. An ambiguous answer key may reject a defensible answer.

The kit's advanced/acceptance-case.md records each case's objective, available documents, expected answer and blocking errors. The actual observation stays blank until the test has run.

Choose how to check each result

A comparison can combine deterministic checks and business review. The method depends on what needs verification.

If another AI grades the responses, first check its judgments against examples already reviewed by the business. A detailed explanation does not prove that a grade is correct.

Why an evaluator with 80% agreement may be unusable

Imagine twenty fictional responses reviewed by a process expert. Sixteen are acceptable; four contain a blocking error. An automatic evaluator accepts all twenty.

Overall agreement is 16/20, or 80%. Yet the evaluator lets through all four blocking errors and detects none. That overall score is insufficient to entrust it with approval.

Inspect bad answers it accepts and good answers it rejects separately. Keep the business reviewer as the reference until those disagreements are understood. This table is an arithmetic example, not a measurement of an existing evaluator.

Separate initial quality from correction work

For each task, record the initial score, errors, generation time, human verification time and correction time. Identify changes made manually and changes made through another request.

Here, authorising automatic sending from S03 or including a cancelled order in the total is blocking. Decide these rules before examining the overall score. Good prose does not compensate for an unauthorised action or a false indicator.

The kit supplies a blank Markdown sheet for all 18 executions. It contains no scores presented as actual measurements.

Calculate a cost that includes your time

For one execution, use:

Full cost = tool cost allocated to the execution
          + human preparation, verification and correction minutes
            × chosen hourly cost ÷ 60

The hourly cost is an internal assumption that must be stated. Record AI waiting time separately: it is not automatically human working time.

Count shared preparation only once. For example, allocate ten preparation minutes across eighteen trials using the same rule for A and B, or report them separately as an initial cost. Add configuration-specific work, such as reformatting required by only one tool.

Here is a wholly fictional example unrelated to vendor prices. A consumes €0.40 and requires 12 human minutes. B consumes €1.20 and requires four minutes. At an assumed €45 per human hour, A costs €9.40, while B costs €4.20. B costs more to execute but less overall in this example, provided both final outputs are acceptable.

With a subscription, choose an allocation rule before testing and state the assumed volume. For a given task, cost per accepted deliverable is the total cost of all its trials, including abandoned attempts, divided by accepted deliverables. Do not pool a short E1 answer and an E2 report into that ratio.

Another fictional example: three report trials cost €18 in total; one is abandoned. The two accepted reports cost €9 each, not €6. With zero accepted outputs, the ratio is undefined. If your intended deliverable combines all three tasks, count fully accepted dossiers instead, using a rule fixed beforehand.

Define acceptance before looking at the ranking

A score helps locate errors; acceptance asks whether this deliverable can be used for its intended task. Define both in the acceptance sheet before running trials.

For Méridien, a report with the correct total that says “sent to the team” violates the no-distribution instruction if the sending actually occurred. Conversely, a statement claiming it was sent does not establish that it happened: inspect the system's records. A false statement and an unauthorised action require different evidence.

Preserve enough evidence to explain a failure

For each trial, retain file versions, the instruction, first response, tools used, actual deliverable, initial score and corrections. If an expected file is missing, record what was actually created. If document access failed, separate that incident from reasoning accuracy on available documents.

Traces support diagnosis and reproduction. They remain subject to your company's access and retention rules. Use the fictional files first to establish this method.

Separate immediate acceptance from acceptance after correction

Here is a fictional example independent of the kit's 18 blank trials: of ten reports, six are accepted immediately, two after correction and two abandoned. Initial acceptance is 6/10, or 60%. Final acceptance is 8/10, or 80%. Human correction time and the cost of abandoned attempts still count economically.

If all trials cost €72, the cost per finally accepted report is 72/8, or €9. Dividing by ten produces €7.20 per attempt, a different measure. Do not compare those ratios as if they described the same result.

Before choosing, review answers on which evaluators disagree. Identify the criterion or source that resolves the disagreement. A scoring change made after inspecting results must be dated and applied to both configurations; retain the initial scores too. The interactive workshop lets you explore costs with zero, one or several accepted results without ranking vendors.

Examine A/B disagreements before counting wins

A total can hide the dossiers on which two configurations fail. This is a fictional example separate from the kit's ten criteria and eighteen trials: ten dossiers receive a binary acceptable/unacceptable judgment before human correction.

A succeeds on eight dossiers and B on seven. Their results differ on three dossiers, rather than providing ten demonstrations that A is better. Open those three responses side by side with the evidence needed to decide. Also examine the dossier both failed: a missing rule may make the test ill-defined, while an available rule ignored by both systems indicates a tool failure.

Suppose one of A's two failures discloses information the recipient is not allowed to receive. Eight successes do not offset that blocking error. The decision may be to retain human approval, remove that use case or correct access before another trial. It is not automatically “choose B”: B must also meet the agreed requirements.

Where possible, conceal the configuration name during the first review. Ask the process owner to resolve disagreements using the relevant source passage. If a criterion changes, apply the new version to both responses.

Ethan Mollick's article on the jagged frontier of capabilities explains why success does not generalise to every task. Its study concerns a different context and 2023 tools; none of its measured gains are attributed to your business. The ten dossiers above are an original illustration of result interpretation, not observed trial data.

Decide for a particular task

Inspect blocking errors, per-task scores and human time before comparing overall averages. One configuration may retrieve decisions correctly but calculate poorly; another may calculate correctly while inventing causes.

Next, choose a recent dossier the configuration has never seen. Establish its answer key with the process owner before running it. Use templates/comparison-decision.md to record the selected setup, specific task and limitations to revisit.

Also compare the current process. If an export and spreadsheet already produce the report correctly, measure the effort with that method. The useful choice may be writing assistance around existing calculations rather than replacing the entire process.

Keep successful cases and rerun them after changing the model, business rule, document connection or export format. When a new failure appears, identify what changed before concluding that the tool has become less reliable. The decision applies to a task, configuration and dossier version; keep it dated.

To explore this decision with Initial IA: prepare a representative dossier, its expected result and two or three errors your team cannot accept. These inputs support a comparison on your work and an integration with explicit acceptance criteria.