Person A: “It’s a game changer!” “There are tons of studies showing that they work!”
Person B: “It’s anecdotal!”
Watching this unfold on TV is painful. Science and medicine rely on research to guide or inform decisions. An important fact here is that there are many types of research studies and there is a scale, or hierarchy of evidence, that each study (type) contributes to our total knowledge on a particular subject. The current level of knowledge on experimental COVID-19 treatments is based mainly on case series, which is one of the lowest levels of evidence on a scientific subject. There many weaknesses in this type of study including, lack of control groups. That is, no comparison is made with an untreated cohort or with a group that is receiving placebo or some other treatment. Such a comparison is important because we want to be able to identify a causal inference regarding the relationship of a proposed intervention and the outcome.
The shortcomings of case reports or case series can be illustrated like this:
During World War II, rescue workers, digging in the ruins of an apartment house blown up in the London blitz, found an old man lying naked in a bathtub, fully conscious. He said to his rescuers, “You know, that was the most amazing experience I ever had. When I pulled the plug and the water started down the drain, the whole house blew up.”
Why are studies needed before treatment guidelines are recommended?
Scientific approach to treatment incorporates the best evidence from well-designed large clinical studies with the goal of promoting consistency of treatment, optimal outcomes and establishing national standards of patient care. When looking at the evidence in the published literature, it is important to identify the study design and understand what value they contribute to the total knowledge based on the hierarchy of evidence spectrum. The hierarchy provides an indication of the validity and trustworthiness of different types of research. This process assists in the selection of the best evidence to guide clinical practice.
As an example, let’s use this spectrum or pyramid when considering the use of antibiotics to treat the common cold, which is caused by a virus. Prescribing antibiotics for colds was commonly done for many decades. To this day, a practitioner still might have to fight off antibiotic requests for the common cold. Antibiotics are effective against bacteria, but not viruses. Kenealy and Arroll ( in 2002 and 2013) conducted two separate systematic reviews of randomized controlled trials including thousands of participants – the highest level in the hierarchy of evidence – and concluded that antibiotics are not effective against the common cold and could cause adverse effects. Yet most people believe and report that cold symptoms improve after antibiotic treatment. These are testimonials, or case reports, based on individual experiences which is one of the weaker forms of evidence since individual anecdotes are not able to control for bias or other factors, for instance secondary bacterial infections.
Recently, a few medication combinations have been touted to be effective in the treatment of Coronavirus; however, very little data is available about the use of these medication combinations in controlled studies. Much of the data is anecdotal from case reports. Using the theory of hierarchy of evidence, you can see why many experts are encouraging caution. While some of this early data may look promising, randomized control studies are needed to account for any bias, weaknesses or other factors before medical practice changes are recommended.
There are significant gaps in our knowledge of this novel and rapidly mutating Coronavirus. The situation is fluid and evolving during this world-wide outbreak. Cogency will strive to stay on top of this situation to keep you informed. For more information about this Coronavirus outbreak and how to characterize your risk and reduce risk to those you are charged to protect, contact the experts at Cogency at solutions@cogencyteam.com