Natural Habitat.
AI Testing and Analysis: Quality, Costs and Repeatability
A model comparison is valuable when the same set of tasks is assessed against explicit criteria. One successful conversation is not a stability test.
Measure information quality, output completeness, time, cost and the need for manual corrections separately.
How to build a test sample
The sample should include typical and difficult tasks, plus cases where the correct response is to decline an answer because data is missing. For extraction, include empty fields, invalid formats and documents with conflicting information.
When testing AI models, use identical inputs for every comparison. Otherwise, result differences may come from changed questions rather than the capabilities of the models.
Separate assessment criteria
Factual accuracy, fidelity to a document, field completeness and format validity are different properties. For a particular task, identify critical failures separately from errors that only require correction.
When analysing AI response quality, record the specific reason for each error as well as the score. This helps distinguish retrieval problems from interpretation problems.
Repeatability and versions
Changing the model, date, instructions or documents can produce different results. Test logs should therefore record the solution version, settings and input identifiers. The same sample can then be rerun after future changes.
When optimizing AI systems, success on one dataset does not establish that an improvement will hold for other types of data. Testing needs different usage scenarios.
Calculations others can reproduce
Using the published Eurostat figures, the change in the share of Polish businesses can be calculated from 5.9% in 2024 to 8.4% in 2025: 8.4 − 5.9 = 2.5 percentage points. This is an original calculation based on published figures, not a test result for an AI system.
For your own tests, specify the denominator, unit, rounding rule and study conditions. Only then can another person check or reproduce the calculation.
Example task assessment criteria
| Criterion | What to check | Example failure |
|---|---|---|
| Source fidelity | Ability to verify a claim | Claim unsupported by the document |
| Completeness | All required fields | Missing date or identifier |
| Format | Schema compliance | Text instead of a number |
| Handling missing data | No unsupported assumptions | Invented value |
Sources
- Eurostat, “20% of EU enterprises use AI technologies”, 11 December 2025, dataset isoc_eb_ai.