Our assessment:

An artificial intelligence (AI) model may be able to rule out COVID-19 infection among people arriving at the hospital, before the results of virus swab tests are ready.

Researchers at Oxford University used data from more than 115,000 hospital visits by adults and five million routine laboratory tests, to ‘train’ an algorithm to predict who had SARS-CoV-2, the virus that causes COVID-19.

They used information which could be gathered within one hour of the patient arriving at the hospital. Routine blood tests and vital signs (such as how fast the patient was breathing) gave the most useful information and this type of information was used to build their algorithm. Testing the algorithm on a further sample of adults attending the hospital suggested it performed well at ruling out COVID-19, although it doesn’t identify everyone who has an infection.

At present, results of swab tests for the virus to confirm COVID-19 can take 24 hours or more to arrive. If hospitals could rule out COVID-19 in some people before test results were available, it could make early treatment (and the need for infection control) more straightforward. The study still needs to be checked by expert reviewers, and further tests of the algorithm in other settings are likely to be needed before it can be more widely used.

Where did the story come from?

The Daily Mail was one of the news outlets that reported on the ‘new AI test’ for COVID-19. The information came from Oxford University’s John Radcliffe Hospital, which has made available the results of their study. The study has not yet been published in a peer-reviewed journal, so it has not been checked by expert reviewers to make sure the methods and results are reliable.

What is the basis for the claim?

The researchers at Oxford University used all routinely-collected medical information about two groups of people attending the accident and emergency department, or admitted to the hospital for care of an acute (i.e. not chronic) condition:

  • those attending between 1 December 2017 and 1 December 2019, none of whom would have had COVID-19 (114,957 attendances)
  • those attending between 1 December 2019 and 19 April 2020, who had a positive test for COVID-19 (534 attendances)

The information included people’s age, sex and ethnic background, as well as basic information such as results of routine blood tests, blood oxygen levels, and vital signs, such as temperature, heart rate and breathing.

They fed the information to a computer artificial intelligence model, to teach it to spot differences in patterns of information between people who had positive tests for the virus, and people who had not had the virus. They used 80% of the data to “train” the algorithm and then tested the algorithm on the remaining 20% of the data.

When looking at either people presenting to the emergency department, or those admitted to hospital, the model correctly identified 77% of people who had the virus, and correctly identified around 95% of those who did not have the virus. These results meant that if the algorithm predicted that a person did not have SARS-CoV-2, doctors could be over 99% sure that they didn’t have it.

After this ‘training’ period, the researchers tested the model on data collected from all patients who had attended the hospital from April 20 to May 6, 2020, to see how well the model could predict who tested positive for SARS-CoV-2. This sample included 3,326 people who attended the emergency department and 1,715 people who were admitted to the hospital. In these samples, the algorithm also performed well, and over 97% of those who the algorithm said did not have COVID-19 did turn out to test negative for SARS-CoV-2.

What do trusted sources say?

As this is early-stage research, which is still awaiting peer-review by experts, no official bodies have commented on the study.

The algorithm will need further testing to determine whether it works similarly on patients in other hospitals and in other settings (such as in different countries) and over longer periods of time as infection rates vary.

Doctors will also need to think about how useful the algorithm would be in their setting, and how the algorithm and its results might be incorporated into their testing and infection control processes. For example, as the algorithm does miss some individuals with COVID-19, doctors would need to decide whether to still use swab testing to identify those who are missed by the algorithm and to decide what infection control measures would be appropriate when managing patients predicted to be negative for COVID-19 on the algorithm.

Analysis by EIU Healthcare, supported by Reckitt Benckiser

Citation

  1. Soltan AS et al. Artificial intelligence driven assessment of routinely collected healthcare data is an effective screening test for COVID-19 in patients presenting to hospital. Pre-print available on medRxiv 2020.07.07.20148361; doi: https://doi.org/10.1101/2020.07.07.20148361 (Accessed 25 August 2020).