Harmostes Consulting

Harmostes is one person: Katherine Pratt, working the gap between what AI produces and what the decision actually needs. The work is for anyone whose AI output feeds an expensive decision: contracts, procurement, compliance, financial review. The domain changes per engagement. The shape of the failure doesn't — the demo below is one instance, not the niche.

An email starts a scoped conversation — what's being evaluated, for what decision, at what stakes. Every engagement is fixed-scope and fixed-fee, and it ends with a deliverable you can act on.

A frontier model was given a real procurement package: an RFP, an evaluation procedure, and two vendor proposals for a K-12 school with over 200 warrantied LG VRF units on campus.

Its job: evaluate both vendors and recommend who to move forward with.

What the model produced

Authorization premium set in Step 2a: Not provided.

Executive recommendation. Do not make a final award yet. Move Sandhill Mechanical forward as the only currently responsive proposal, but only to a "resolve before award" step. Sandhill is the lower-price vendor and passes the visible insurance requirement, but it is not LG-authorized, and references have not been contacted.

Do not move Tidewater forward under the procedure as submitted. Tidewater is technically stronger because it is an LG VRF Certified Service Provider, but its certificate of insurance shows "N" for additional insured.

Sandhill — responsive, pending references. $30,500 total (65 points plus references). LG authorization absent; references unverified. Tidewater — disqualified as submitted, not scorable. Certificate-of-insurance defect; bundled base price.

Recommended next move: advance Sandhill to reference checks and LG-authorization clarification, while issuing Tidewater a limited cure request only if procurement rules allow correction.

It scored both vendors. It built a tiered comparison. It caught a real paperwork defect in Tidewater's insurance certificate. It even wrote a question about the warranty risk. Read that output cold, and it looks like professional work product.

The evaluation procedure told the model to stop before scoring and get a number from the decision-maker: "how much is current LG authorization worth to this campus?" That line at the top — "Authorization premium: Not provided" — means the model noted the blank and kept going.

It flagged Sandhill's missing LG authorization. It wrote a cure question about warranty coverage. Then it advanced the cheaper, unauthorized vendor anyway — and archived the fully authorized one.

The warranty risk on over 200 LG VRF units appeared in the report. It was never allowed to change the recommendation.

A facilities director acting on this output would advance a vendor who could void the manufacturer's warranties across the school's dominant HVAC system — based on a report that looked thorough, scored correctly, and never told them it had skipped the question it was supposed to ask first.

Same model. Same documents. Different questions.

Decision status: Do not award yet — resolve 3 items first.

How much is current LG authorization worth annually? The campus is LG VRF-heavy; Sandhill is not LG-authorized, Tidewater says it is. Needed now: the decision-maker sets a maximum premium.

Can Tidewater cure its paperwork and pricing? Tidewater may be safer technically, but it cannot be scored yet. Needed now: two questions go to Tidewater.

Is Sandhill's LG risk acceptable if the savings are material? Sandhill is the clean, low-cost bidder but lacks LG authorization. Needed now: a warranty and training question goes to Sandhill.

If Tidewater cures its certificate of insurance and its price lands within the LG premium, prefer Tidewater. If Tidewater cannot cure the certificate defect, do not award Tidewater. If Tidewater's premium exceeds the LG premium, prefer Sandhill, if the LG risk plan is acceptable. If Sandhill cannot provide an acceptable LG or warranty plan, prefer Tidewater if curable; otherwise reconsider.

No award today. The decision-maker does not need to re-read either proposal. They need to ask the listed questions, set the LG premium, then choose under the decision rule.

The warranty risk on 200+ units reached the decision-maker as the first item on the table — not as a footnote, not as a deferred note, but as a question that had to be answered before anything else moved.

It wasn't a fluke

The model ran the same evaluation two more times. It failed differently each time.

Run 2 scored both vendors, flagged the warranty risk, and archived the fully authorized vendor — advancing the cheaper, unauthorized one.

Run 3 dropped one vendor entirely. Given two proposals, it produced a single-vendor report. The second vendor was never mentioned.

Three runs. Three different failures. A tool that produces a different wrong answer every time is one you can't build a process on.

Katherine Pratt

Write to katherine@harmostes.com.

What you send is held in confidence and used only to scope the engagement — nothing goes further without a signed letter.

AI judgment consulting. The difference between complete and actionable.