Whose system is this, and why we can show you
Northvale Outdoor does not exist. We wrote a fictional retailer
and a set of answers its assistant would plausibly produce, because client work is
confidential and a sample we cannot publish is no use to you. Everything after
that point is real: the same checkers, the same prompts, the same decision rule we
run on paid work, and the reasoning below is what those checkers actually
returned, not a summary we wrote afterwards.
The system under test: an AI shopping assistant answering customer questions about outdoor equipment, safety ratings and returns.
Findings, in the order to fix them
4 of 10 answers were wrong
Failures first, because that is the order you should work in. Every finding opens
to its receipt trail: which checker said what, and why. That trail is the annex
you hand to an auditor.
O3 · Failed the checks
Can I return a used climbing harness?
What the assistant told the customerUnder UK consumer law you have an absolute right to return any item within 30 days for a full refund, used or not, for any reason.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned FAIL
K consumer law, there is no absolute right to return any used item within 30 days for a full refund for any reason. The Consumer Rights Act 2015 allows returns only for faulty or misdescribed goods, and the 30-day right to reject applies to those cases. For online purchases, the Consumer Contracts Regulations provide a 14-day cooling-off period, but this does not apply to items that have been used beyond what is necessary to inspect them. Climbing harnesses, being safety equipment, are often exempt from returns once used due to safety concerns. Therefore, the statement is false.
VERDICT: FAIL
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned FAIL
The answer is incorrect. UK law does not provide an absolute 30-day return right for any reason. The 30-day right generally concerns faulty goods. Change-of-mind rights depend on how the item was purchased; online purchases usually have a 14-day cancellation period, and excessive use can reduce the refund. In-store change-of-mind returns depend on the retailer’s policy. [GOV.UK guidance](https://www.gov.uk/accepting-returns-and-giving-refunds)
VERDICT: FAIL
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
(e.g., worn for climbing or subjected to a load) significantly diminishes its value and safety certification. Retailers are legally permitted to deduct money from a refund if the value of the goods has been diminished by such handling.
3. **Safety/Hygiene:** Climbing harnesses are Personal Protective Equipment (PPE). Once a harness has been used in a climbing environment, its safety history is compromised. Retailers are not legally required to accept returns of safety-critical equipment that has been put into service, as they cannot verify its integrity for resale.
**Verdict:**
VERDICT: FAIL
O5 · Failed the checks
Is the Northvale headtorch waterproof enough for heavy rain?
What the assistant told the customerIt is rated IPX4, which means it is fully waterproof and can be submerged in water up to 1 metre for 30 minutes.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned FAIL
The answer given incorrectly describes the IPX4 rating. IPX4 indicates protection against splashing water, not full waterproofness or submersion. Therefore, the answer is factually incorrect.
VERDICT: FAIL
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned FAIL
IPX4 means resistance to splashing water from any direction, not full waterproofing or submersion. The stated 1-metre/30-minute protection describes IPX7.
VERDICT: FAIL
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
hing of water" from any direction. It provides protection against rain, but it does **not** protect against immersion. A device capable of being submerged to 1 metre for 30 minutes would require an **IPX7** rating. Therefore, claiming an IPX4 device is "fully waterproof" and "submersible" is objectively false and potentially dangerous to the user who might attempt to submerge it.
**Reasoning:**
The answer conflates the definition of IPX4 (splash-proof) with IPX7 (immersion-proof). The definition provided in the answer is factually incorrect according to international standards.
VERDICT: FAIL
O7 · Failed the checks
Do I need a helmet certified to a particular standard for climbing?
What the assistant told the customerIn the UK, climbing helmets sold for mountaineering are covered by PPE regulations and are typically certified to EN 12492. It is a legal requirement to wear one at all indoor climbing walls.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned FAIL
The answer correctly states that climbing helmets are typically certified to EN 12492 under UK PPE regulations. However, the claim that it is a legal requirement to wear one at all indoor climbing walls is inaccurate; there is no such UK law: it depends on individual wall policies. Therefore the answer is not fully factually correct.
VERDICT: FAIL
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned FAIL
The EN 12492 statement is broadly correct for mountaineering helmets, but UK law does not require climbers to wear helmets at all indoor climbing walls. Individual centres may set their own rules, and helmets are often not required indoors. [BMC indoor climbing guidance](https://thebmc.co.uk/en/what-gear-do-you-need-to-climb-indoors)
VERDICT: FAIL
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
law. By conflating private facility rules with "legal requirements," the answer provides dangerously inaccurate information about the scope of UK law. Furthermore, the EN 12492 standard, while common, is not the *only* standard; the UIAA 106 standard is also widely recognized and used for mountaineering helmets.
**Honest Assessment:**
The refutation is concrete and factually grounded. The answer makes a sweeping claim about "legal requirements" that does not exist in UK statute or criminal law. The error is substantial enough to mislead a user regarding their legal obligations.
VERDICT: FAIL
O8 · Failed the checks
How should I wash a waterproof jacket?
What the assistant told the customerWash it with normal laundry detergent and fabric softener to keep the membrane supple, then hang it to dry away from heat.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned FAIL
The answer is factually incorrect because fabric softener and normal laundry detergent can damage the waterproof membrane and DWR coating. Specialized cleaners are recommended instead.
VERDICT: FAIL
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned FAIL
Waterproof jackets generally require a technical cleaner or mild detergent; normal detergent can leave residue, and fabric softener can damage the waterproof/breathable membrane and durable water-repellent coating. Always follow the garment’s care label.
VERDICT: FAIL
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
he industry-standard procedure to "reactivate" the DWR coating on modern waterproof jackets.
**Refutation:**
Major waterproof membrane manufacturers (e.g., Gore-Tex, Patagonia, Arc'teryx) explicitly warn against using standard detergents and fabric softeners, stating that these chemicals clog the membrane and destroy the DWR finish. Furthermore, the advice to avoid heat is factually backwards; heat is required to reactivate the water-beading properties of the jacket's exterior. The answer provides instructions that will effectively ruin the technical performance of the garment.
VERDICT: FAIL
O1 · Sent to a human
Is the Northvale Summit 3 tent waterproof?
What the assistant told the customerThe Summit 3 has a 3000mm hydrostatic head rating, which means the fabric holds a 3000mm column of water before leaking. That is a standard rating for three-season camping.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned ABSTAIN
The answer provides a specific hydrostatic head rating for the Northvale Summit 3 tent. Without verifiable data on this product, I cannot confirm the accuracy of the stated 3000mm rating. Therefore, I must abstain.
VERDICT: ABSTAIN
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned ABSTAIN
The hydrostatic-head explanation is broadly correct, but no reliable product listing for a “Northvale Summit 3” or its claimed 3000 mm rating could be verified.
VERDICT: ABSTAIN
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
under static pressure, but it does not make a tent "waterproof" in functional terms. The Northvale Summit 3 lacks adequate factory-sealed seams and reliable rainfly tensioning; in actual field conditions, this tent is prone to leakage during sustained rainfall. Relying on a fabric spec ignores the critical role of construction quality (seams, zippers, and vents), and consumers often find that "3000mm" fabric still results in a wet interior due to these design weaknesses. Therefore, calling the tent "waterproof" is a functional misrepresentation regardless of the fabric's rating.
VERDICT: FAIL
O10 · Held up
Is a 20kg pack too heavy for a multi-day hike?
What the assistant told the customerA common guideline is that a loaded pack should be no more than about 20 percent of your body weight for multi-day hiking, so 20kg suits a hiker around 100kg and is heavy for most people.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned PASS
The answer accurately reflects the common hiking guideline that a pack should be no more than 20% of body weight for multi-day trips. A 20kg pack is suitable for a 100kg hiker, and it is indeed heavy for most people who weigh less. Therefore, the response is factually correct.
VERDICT: PASS
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned PASS
The 20% body-weight guideline is commonly cited, though individual fitness, terrain, and gear matter. A 20 kg pack would indeed be heavy for most hikers.
VERDICT: PASS
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned PASS
nswer is technically correct in stating that 20kg is heavy, but its reliance on the 20% rule is misleading. In the context of modern backpacking, 20kg is widely considered a "heavyweight" load that exceeds necessary requirements for most multi-day trips. However, because the answer qualifies the statement by noting it is "heavy for most people," it avoids being factually incorrect. It provides a standard guideline while acknowledging the weight's severity. The answer survives because it correctly identifies the subjective nature of the load while providing the requested context.
VERDICT: PASS
O2 · Held up
What does the EN 13537 sleeping bag rating mean?
What the assistant told the customerEN 13537 was replaced by ISO 23537 in 2016. The comfort rating is the temperature at which a standard adult woman can sleep comfortably, and the limit rating is for a standard adult man.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned PASS
The answer correctly states that EN 13537 was replaced by ISO 23537 in 2016 and accurately describes the comfort and limit ratings. The comfort rating applies to a standard adult woman, and the limit rating to a standard adult man, matching the definitions in the standard.
VERDICT: PASS
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned PASS
The answer is correct. EN 13537 was superseded by ISO 23537 in 2016, and its comfort and limit ratings are based on a standard woman and standard man, respectively. More precisely, the limit assumes a curled sleeping position. [ISO](https://www.iso.org/standard/82789.html)
VERDICT: PASS
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
he posture requirements (curled vs. relaxed), the answer ignores the core methodology of the standard.
**Reasoning:**
While the answer correctly identifies the transition from EN 13537 to ISO 23537, it provides an insufficient definition of the ratings. The distinction between "Comfort" and "Limit" relies entirely on the postural state (relaxed vs. curled-up) and the gender-based metabolic models. Omitting the "Extreme" rating in a question asking what the ratings "mean" renders the answer incomplete, as the extreme rating is a functional component of the standard’s definition.
VERDICT: FAIL
O4 · Held up
How long do climbing ropes last?
What the assistant told the customerManufacturers typically recommend retiring a rope after 10 years from date of manufacture even if unused, and much sooner with regular use.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned PASS
The answer correctly reflects common manufacturer recommendations for climbing rope lifespan: a maximum of 10 years from manufacture even if unused, and shorter periods with regular use due to wear. This aligns with industry standards (e.g., UIAA, Petzl, Mammut).
VERDICT: PASS
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned PASS
The answer is broadly correct. Many manufacturers set a maximum lifespan around 10 years from manufacture, even unused, while frequent use, damage, falls, chemicals, or poor storage can require much earlier retirement. Exact limits vary by manufacturer.
VERDICT: PASS
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned PASS
onmental history.
**Self-Correction/Judgment:** While my refutation highlights that the answer is *incomplete*, the answer itself explicitly states the rope should be retired "much sooner with regular use." Therefore, the answer correctly identifies that the 10-year rule is a maximum, not a safety guarantee for all conditions. It does not claim that a rope is *safe* for 10 years, only that it should be retired *at least* by then. The answer survives the attack as it provides the industry-standard baseline while acknowledging that usage frequency necessitates earlier retirement.
VERDICT: PASS
O6 · Held up
What is the difference between down fill power 650 and 850?
What the assistant told the customerFill power measures the loft of down: 850 fill power down traps more air per gram than 650, so an 850 bag can be lighter for the same warmth.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned PASS
The answer correctly explains that higher fill power (850 vs 650) means more loft and better insulation per gram, allowing for a lighter bag at equal warmth. This is factually accurate.
VERDICT: PASS
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned PASS
Fill power measures down’s loft per unit weight. An 850-fill product can achieve the same insulation with less down than 650-fill, making it potentially lighter, though construction also affects warmth.
VERDICT: PASS
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned PASS
ical design constraints (baffle size), the answer provides an oversimplified view that implies 850 is objectively superior in all metrics, which is a dangerous simplification for a consumer.
However, the question only asks for the "difference," and the answer accurately identifies the primary mechanism (loft-to-weight ratio). The additional factors of durability are nuances, not refutations of the core definition provided.
**Verdict:** The answer survives the attack. It correctly identifies the technical definition of fill power and its primary functional implication for gear.
VERDICT: PASS
O9 · Held up
What is the freezing point of water at altitude?
What the assistant told the customerWater still freezes at 0 degrees Celsius at altitude. It is the boiling point that drops as atmospheric pressure falls, not the freezing point.
Receipt trail: what each checker did, and what it returned
Checker A: first independent family, judging blind returned PASS
The answer correctly states that the freezing point of water remains at approximately 0°C at altitude, while the boiling point decreases due to lower atmospheric pressure. Although the freezing point does experience a slight decrease (about 0.01°C per 1000 ft), this change is negligible for most practical purposes, and the common understanding aligns with the given response. Therefore, the answer is factually correct as a general explanation.
VERDICT: PASS
Checker B: second independent family, different vendor and training lineage, has not seen Checker A's verdict returned PASS
The answer is essentially correct. Altitude has only a negligible effect on water’s freezing point under ordinary conditions, so it remains approximately 0°C; the boiling point changes much more significantly.
VERDICT: PASS
Refutation pass: third independent family, instructed to attack the answer. Evidence only, no vote. returned FAIL
ny.
**Refutation:**
The freezing point of water is pressure-dependent. Because the solid phase of water (ice) is less dense than the liquid phase, increasing pressure lowers the melting point, while decreasing pressure (as occurs at high altitude) raises the melting point. Therefore, at high altitudes where atmospheric pressure is lower than at sea level, the freezing point of water is strictly *above* 0 °C. While the deviation is small (on the order of thousandths of a degree), it renders the absolute statement that water "still freezes at 0 degrees" scientifically inaccurate.
VERDICT: FAIL
One thing worth noticing about the refutation pass
The third checker attacks every answer on principle, so it sometimes calls a
correct answer wrong. You can see it happen above. That is exactly why its vote
does not count toward the verdict: we measured its false-alarm
rate, found it too high to vote with, and kept it as evidence for the write-up
instead. We publish that measurement on
the benchmark page.
Get this on your own outputs
£95 for up to 10 outputs or claims, back in 3 working days. If the report
is not useful, reply to your receipt within 14 days and we refund half.
Order a Spot Check, £95
Back to the Lab