Arm P: production code 10dd338b. Arm E: Goal E code 658929c5. Both used the launcher environment without provider or model overrides. Both smokes observed DeepSeek response model deepseek-flash, requested through the deepseek-v4-flash fast-tier alias. Verifier off; two workers; three sequential repeats per arm. F1B used glm-5.3, the non-production generator.
| Measure | P: DeepSeek | E: DeepSeek | F1B: non-production glm |
|---|---|---|---|
| Runs | 90 | 90 | 90 |
| Matches | 0 | 0 | 0 |
| Mismatches | 41 | 21 | 41 |
| Clarifications | 30 | 31 | 30 |
| Refusals | 0 | 5 | 3 |
| Errors | 49 | 64 | 46 |
| p95 seconds | 9.343 | 15.388 | 27.557 |
Deploy line: FAIL. Matches may not fall and mismatches may not rise beyond observed repeat spread. The predeclared tolerance is three times the larger within-arm range of the three repeat counts. Match tolerance: 0; mismatch tolerance: 18. E p95 must be at most 10.277s; observed ratio 1.6470.
| Paired E minus P metric | Difference | 95% family-cluster interval |
|---|---|---|
| matches | 0.0000 | 0.0000 to 0.0000 |
| mismatches | -20.0000 | -34.0000 to -9.0000 |
| refusals | 5.0000 | 0.0000 to 15.0000 |
| errors | 15.0000 | 8.0000 to 23.0000 |
| clarifications | 1.0000 | 0.0000 to 3.0000 |
| p95_ms | 6045.0070 | -10271.1940 to 8113.7760 |
| p95_ratio | 1.6470 | 0.4631 to 1.9672 |
Intervals resample 15 whole families, preserving both questions, all repeats and arm pairing; 20,000 samples, fixed seed. Count intervals are differences across 90 runs; p95_ms is a millisecond difference and p95_ratio is E divided by P. These are exploratory intervals over the selected families, not a population-wide guarantee.
| Arm / repeat | Matches | Mismatches | Refusals | Errors | Clarifications | p95 seconds |
|---|---|---|---|---|---|---|
| P / 1 | 0 | 14 | 0 | 16 | 10 | 8.726 |
| P / 2 | 0 | 13 | 0 | 17 | 10 | 20.548 |
| P / 3 | 0 | 14 | 0 | 16 | 10 | 8.407 |
| E / 1 | 0 | 4 | 1 | 25 | 11 | 18.378 |
| E / 2 | 0 | 10 | 2 | 18 | 10 | 8.859 |
| E / 3 | 0 | 7 | 2 | 21 | 10 | 7.527 |
| Question | P: repeats 1 / 2 / 3 | E: repeats 1 / 2 / 3 |
|---|---|---|
| Q01 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q02 | error; none; other / error; none; other / error; none; other | error; none; other / error; none; other / error; none; other |
| Q03 | clarified_then_mismatch; partial; wrong_grain / clarified_then_mismatch; partial; wrong_grain / clarified_then_mismatch; partial; wrong_grain | error; partial; other / clarified_then_mismatch; partial; wrong_grain / clarified_then_mismatch; partial; wrong_grain |
| Q04 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q05 | clarified_then_mismatch; partial; wrong_join_or_key / clarified_then_mismatch; partial; wrong_join_or_key / clarified_then_mismatch; partial; wrong_join_or_key | error; partial; other / error; partial; other / error; partial; other |
| Q06 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q07 | clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent | error; none; other / clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent |
| Q08 | error; all; other / error; all; other / error; all; other | error; all; other / error; all; other / error; all; other |
| Q09 | error; all; other / error; all; other / error; all; other | error; all; other / error; all; other / error; all; other |
| Q10 | clarified_then_mismatch; partial; definition_absent / clarified_then_mismatch; partial; definition_absent / clarified_then_mismatch; partial; definition_absent | error; partial; other / clarified_then_mismatch; partial; definition_absent / clarified_then_mismatch; partial; definition_absent |
| Q11 | error; none; other / error; none; other / error; none; other | error; none; other / error; none; other / error; none; other |
| Q12 | error; none; other / error; none; other / error; none; other | error; none; other / error; none; other / error; none; other |
| Q13 | mismatch; partial; wrong_grain / error; partial; other / mismatch; partial; wrong_grain | error; partial; other / mismatch; partial; wrong_grain / error; partial; other |
| Q14 | error; partial; other / mismatch; partial; wrong_grain / mismatch; partial; wrong_grain | mismatch; partial; wrong_grain / mismatch; partial; wrong_grain / mismatch; partial; wrong_grain |
| Q15 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q16 | clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent | clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent / clarified_then_mismatch; none; definition_absent |
| Q17 | mismatch; partial; definition_absent / mismatch; partial; definition_absent / mismatch; partial; definition_absent | mismatch; partial; definition_absent / mismatch; partial; definition_absent / mismatch; partial; definition_absent |
| Q18 | mismatch; partial; wrong_window / mismatch; partial; wrong_window / mismatch; partial; wrong_window | error; partial; other / error; partial; other / error; partial; other |
| Q19 | mismatch; partial; definition_absent / mismatch; partial; definition_absent / mismatch; partial; definition_absent | error; partial; other / mismatch; partial; definition_absent / mismatch; partial; definition_absent |
| Q20 | mismatch; all; definition_present_not_applied / mismatch; all; definition_present_not_applied / mismatch; all; definition_present_not_applied | mismatch; all; definition_present_not_applied / mismatch; all; definition_present_not_applied / error; all; other |
| Q21 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q22 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q23 | error; none; other / error; none; other / error; none; other | error; none; other / mismatch; none; wrong_window / error; none; other |
| Q24 | clarified_then_mismatch; all; wrong_window / error; all; other / clarified_then_mismatch; all; wrong_window | error; all; other / error; all; other / error; all; other |
| Q25 | mismatch; none; definition_absent / mismatch; none; definition_absent / mismatch; none; definition_absent | refused; none; definition_absent / refused; none; definition_absent / refused; none; definition_absent |
| Q26 | mismatch; none; wrong_window / mismatch; none; wrong_window / mismatch; none; wrong_window | error; none; other / refused; none; definition_absent / refused; none; definition_absent |
| Q27 | error; partial; other / error; partial; other / error; partial; other | error; all; other / error; all; other / error; all; other |
| Q28 | error; partial; other / error; partial; other / error; partial; other | error; all; other / error; all; other / error; all; other |
| Q29 | clarified_then_mismatch; partial; definition_absent / clarified_then_mismatch; partial; definition_absent / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
| Q30 | error; partial; other / error; partial; other / error; partial; other | error; partial; other / error; partial; other / error; partial; other |
Each per-question entry gives outcome, relevant-definition coverage, and defect class. Errors and refusals are outside the match/mismatch counts. Definitions reflect captured prompts; a bundle or route being present does not prove every relevant rule was present.
| Arm | Route | Prompts with material | Prompt denominator | Rate |
|---|---|---|---|---|
| P | default | 0 | 95 | 0.00% |
| P | resolver | 21 | 95 | 22.11% |
| P | catalog | 95 | 95 | 100.00% |
| P | exemplar | 0 | 95 | 0.00% |
| E | default | 95 | 95 | 100.00% |
| E | resolver | 22 | 95 | 23.16% |
| E | catalog | 95 | 95 | 100.00% |
| E | exemplar | 0 | 95 | 0.00% |
Arm P relevant coverage: all 12, partial 54, none 24, no generation prompt 0. Captured prompts: 95/95. Scored SQL calls: 95; without a response model: 0. All observed calls including smoke and ancillary routing: 170.
Arm P post-clarification outcomes: clarified_then_mismatch: 19, error: 11.
Arm P terminal error classes: sql: 47, schema: 1, semantic: 1.
Arm P clarifications settled by existing rulings or explicit wording: Q05: 3, Q07: 3, Q15: 3, Q16: 3, Q24: 3, Q29: 3, Q30: 3.
Arm E relevant coverage: all 18, partial 48, none 24, no generation prompt 0. Captured prompts: 95/95. Scored SQL calls: 95; without a response model: 0. All observed calls including smoke and ancillary routing: 151.
Arm E post-clarification outcomes: clarified_then_mismatch: 9, error: 22.
Arm E terminal error classes: sql: 62, data: 1, schema: 1.
Arm E clarifications settled by existing rulings or explicit wording: Q05: 3, Q07: 3, Q15: 3, Q16: 3, Q24: 3, Q29: 3, Q30: 3.
Settled clarification topics: professional partners; all acquisition taxonomies; both referral classes; metal-band high value; settled commission on completion; executed-trade revenue; explicit sell-side SAR with a monthly denominator. Material oracle ambiguities remain pending owner ratification.
Both smoke generator calls completed. P smoke returned an answer; E smoke subsequently ended in a SQL execution error. The same sealed F1B oracle and clock anchor were reused unchanged. Product clocks and read-only source replicas remained live, as in F1B; snapshot equivalence and drift are unverified. The provenance patch required an observation-only compatibility fallback on pre-Goal-E code; four prompt fixtures per arm remained byte-identical to each arm’s uninstrumented generator. P preflights failed before SQL generation and are retained separately. The launcher lacked embedding settings, and the local definition index was empty. Both arms used production’s RunPod embedding settings and production definition-index searches through an isolated tunnel; initialization and writes were disabled. No scored answer was rerun to improve a result. An arithmetic review checks counts, not oracle SQL semantics.
One E clarification, Q08 repeat 1, used a legacy question/options payload with no ruling-mappable resolution state. The inherited follow-through supplied no selected ruling and its second turn ended in a SQL error. That error is retained; the outcome after a properly mapped reply is unverified. The 21 established ruling-settled clarification events per arm exclude this extra event.
Arithmetic review: PASS. Public output contains only validated categories, counts and timings; private answers stay on the evaluator host.