The model that learned the X-ray machine

In 2018 a group of researchers trained a neural network to do something nobody would ever pay for. It looked at a chest X-ray and guessed which hospital the image came from. It got the hospital right for 99.95% of images from the National Institutes of Health and 99.98% of images from Mount Sinai. We tend to think an X-ray is a picture of lungs. To the model it was also a picture of the machine.

The paper, by Zech and colleagues in PLOS Medicine, was really about pneumonia. The authors pulled 112,120 X-rays from the NIH, 42,396 from Mount Sinai Hospital and 3,807 from Indiana University, and trained models to detect pneumonia. Then they asked a question that was rare in AI studies at the time: how does the model do on a hospital it has never seen?

In 3 of 5 comparisons, it did worse. The clearest case was a model trained on both Mount Sinai and NIH images. On unseen images from those same two systems, its AUC was 0.931. On unseen images from Indiana, it was 0.815.

AUC needs a sentence. Pick one sick patient and one healthy patient at random and show both X-rays to the model. AUC is how often the model gives the higher score to the sick one. A coin flip is 0.5 and a perfect model is 1.0. The gap between 0.93 and 0.82 is the gap between a model that rarely mixes the two up and one that mixes them up roughly once in five tries.

The reason is in the data. About 34% of the Mount Sinai X-rays showed pneumonia. At the NIH it was 1.2%, and at Indiana it was 1.0%. If a model can tell a Mount Sinai image from an NIH image, it has a shortcut. When the picture looks like Mount Sinai, guess pneumonia.

And it could tell. Each hospital leaves a fingerprint in its images, from the scanner, the way the image is processed, and small artifacts from its imaging system. None of that has anything to do with pneumonia, but in this dataset it tracked pneumonia closely. A model will use the best predictor it can find, and the best predictor isn't always the one we hoped for.

I've watched this happen in my own work. I trained a tuberculosis model on a mix of public X-rays. On unseen images from those same sources it scored above 99%. On images from a hospital that wasn't in the training mix it scored somewhere between 67% and 86%, depending on which external set I used. That was the polite version.

The impolite version came from a different experiment. A model trained on several pooled datasets had gotten so good at recognizing scanners that, on external TB X-rays, its sensitivity (the share of real TB cases it caught) fell to 0.5%. Half of one percent. It was calling nearly every sick patient healthy, because in the training data "healthy" and "this particular scanner" had become close to the same thing.

When I retrained on one consistent dataset, sensitivity rose to about 22%. That is still nowhere near good enough, but it's a model that has at least started looking at lungs. The 0.5% model was looking at the scanner.

I should be careful here. These models come from a student project, and the results aren't published. I don't know how much of the improvement came from using a consistent dataset and how much came from other changes I made along the way. I wouldn't hang anything on the exact numbers. What I'd hang on them is the pattern: a model can look excellent on its own data and collapse at another hospital, and that has nothing to do with medicine.

This is what external validation is for, and it's worth saying what the term actually means. Holding back 20% of your own data and testing on it is not external validation. That data came from the same hospital, the same scanners, and often the same patients on different days. External validation means a different hospital with a different imaging system, ideally in a different country with a different disease rate, using data collected after the model was frozen so nobody could have peeked.

Models already on the market haven't been required to do this. In a 2021 Nature Medicine study, Wu and colleagues reviewed 130 AI devices the FDA cleared between 2015 and 2020, looking at what the public reports disclosed. Of the 130, 93 had no publicly reported multi-site evaluation. Of the remaining 37, 4 were evaluated at a single site and 8 at only two.

The same study also built pneumothorax models on data from each of three sites and tested each model at the other two. Performance dropped by 0.072 AUC on average and by as much as 0.124. That's close to the drop Zech saw, this time measured on the kind of tool a hospital could actually buy.

None of this means the models are bad. It means nobody can say whether they work on a different imaging system, and a hospital's imaging system is always somebody else's.

The FDA publishes a summary for every cleared device. When I read summaries for radiology AI, the sentence I look for starts with "The test set was collected from" and then names sites that don't also appear in the training set. I'd like to see how many sites there were, which scanner makes and models were involved, which countries, and what the disease rate was at each one, because the 34% against 1% in Zech's data matters that much. I'd want performance reported site by site and not pooled, since pooling hides which site fell apart. And I'd want to know whether any test images came from a hospital that also supplied training images.

A model that can tell which hospital an X-ray came from has done exactly what it was trained to do. Nobody asked it for more. The summary is where that should show up.