When a computer makes an error, people tend to judge it harshly. No matter the complexity of the task, we usually start complaining about the software program or stop trusting the algorithm at all. But do we assess our mistakes in the same way? Can we even realize how many mistakes we make?

Let’s look at a complex task of identification of healthcare-associated infections (HAIs) during the point prevalence study conducted in 2017 (see the complete report by Suetens et al.). Each patient staying at that time in the hospital was assessed against the HAIs definitions proposed by the European Centre for Disease Prevention and Control (ECDC). This assessment was done by medical experts, i.e., members of infection prevention and control teams or hospital hygiene departments.

In some countries, national validation teams carried the validation study of the reported HAIs independently from the primary surveyors (i.e., acute care hospitals) to check the results. They re-examined 12,228 patient charts in total. The authors of the paper considered the validation teams’ results as a gold standard and they used it to assess the quality of the original survey results.

On average, experts found less than 70% of all HAIs during the point prevalence study. Besides that, 20% of reported HAIs were not identified correctly.

Suetens et al. state that the mean recall (sensitivity) of HAI detection was 69.4%. This means, that less than two-thirds of the HAI patients were discovered and reported by the reviewed hospitals. 30.6% of HAI patients were not reported at all! The mean precision of the original results was 79.7%, i.e., only 79.7% of the originally reported HAI patients truly had an HAI and the rest was reported falsely.

The poor validation results show, that even the highly qualified experts sometimes err. Therefore, we might be a little more cautious and tolerant next time we assess the performance of computers and automated algorithms. However, you may still be wondering who would win: humans or the machines? In our experience, nothing can beat human-machine cooperation.

• Blog posts