Finding and using knowledge

An evidence hierarchy can be a useful tool to help identify the best available evidence to answer a question. Study types are ranked on the rigour of research methods which minimise bias and strengthen the reliability of evidence: Evidence pyramid | How to search

However, that doesn’t mean that the study types at the top of this hierarchy will always be best for meeting your information need. The type of study that will get you to the best information will depend on what type of question you are asking. There is no point in reading a quantitative paper if you are interested in the lived experience of individuals, or reading a qualitative paper if you are interested in the statistical relationship between two variables. You may even find that your question requires a combination of both qualitative and quantitative methods.

Below we have some example questions and our recommendations for the best study type to answer a question and/or provide relevant information.

1. Case report
   • Example question: What factors led to delayed diagnosis of knee ligament injury in a young shinty player in Scotland?
2. Qualitative studies
   • Example question: Barriers faced by women re-entering employment after mental illness
3. Case control studies
   • Example question: You are interested in the causes of knee osteoarthritis in men aged 40-60 living in Braemar
4. Cohort studies
   • Example question: Is long term participation in ceilidh dancing associated with reduced risk of falls in older adults in Scotland
5. Randomised control trial
   • Example question: Is the Glasgow wonder vaccine effective against the Scottish flu?
6. Systematic Review
   • Example question: Is the Glasgow wonder vaccine effective against the Scottish flu?

​

If you are looking at a rare presentation, often the best available evidence will be a case report.

A case report is a description of a single case. To get published it usually needs to be about an unusual presentation or difficult diagnoses, but they can also complement bedside learning. Case studies can include useful discussion around differential diagnosis and decision making. You can take valuable learning from a case study but they are not intended to be a basis for clinical decision making.

One step up you have case report series – these are a collection of individual case reports. The medical histories of more than one patient with a particular condition can be described to illustrate a point. These can be used to identify commonalities that would lead to a hypothesis. And that hypothesis could then be tested using one of the analytical study types (RCT, cohort study).

Case reports and case series are descriptive, not analytical, studies and are consequently considered to be weak evidence. However, they are often the first study to raise awareness of a new threat. For example, one of the first studies on Covid was a case series in January 2020, looking at 41 adults. The speed as which case reports and case series can be published is of key importance. It was also famously, through case studies that concern was first raised about thalidomide. For rare presentations case studies and case report series may be the only evidence available.

​

You may find quantitative studies addressing aspects of this question, but if you want to explore human experience, qualitative studies are ideal for this.

Qualitative studies are about exploring an idea and skimming beneath the surface. They focus on lived experience, feelings and perspectives and can provide detailed granular responses to help us understand health behaviours. The method uses and often combines interviews, focus groups and questionnaires. It can also include ethnography (observation) and discourse analysis (detailed study of words or phrases – both written and spoken). There tends to be very little statistical analysis and where quantitative data has been historically preferred, this can lead to bias against research more aligned to social sciences. However, as Trisha Greenhalgh notes, qualitative research has become increasingly popular because quantitative methods have failed to answer important questions in both clinical care and service delivery. (Greenhalgh, How to read a paper, 2019, p218). Where sample sizes are small, qualitative studies offer appropriate methods because these focus on in-depth understanding rather than statistical answers.

The findings are based on the subjective experience of the researcher and the subject. The focus of the research may not be clear at the start of the study or may change based on what is uncovered. However, by the time the paper is written it should address a clearly formulated question.

Qualitative research is intrinsically different from quantitative research. For example, when recruiting for a randomised control trial it is essential participants are properly randomised to minimise bias, but for a qualitative study we need to recruit participants that meet the needs of the research. For our example question, that would mean recruiting women who are recovering from mental illness, but we may also want to examine the question across different geographical areas, or employments. Randomisation would not meet this specification.

​

If you want to know more about the causes of knee osteoarthritis, a case control study starts with an outcome and looks back at various exposures.

The objective of a case control study is to identify correlations between an outcome and an exposure. A case control study starts off with a set of cases which have a specific outcome, in our example knee osteoarthritis. There is an eligibility criteria to be included in the study – for example 40-60 year old men living in Braemar who have developed knee osteoarthritis. There is also a control group – the control group should be as similar as possible with the exception of the outcome. The study then looks back at exposures for both groups to see if a difference can be identified that might explain the difference in outcome.

In our example there is a clear correlation between tossing the caber and our group of 40-60 year old men living in Braemar with knee osteoarthritis.

Case control studies cannot establish causality, only a correlation.

Weaknesses of case control studies include the difficulty of creating a control group which is sufficiently similar to the case group – differences between the control group and case group can lead to confounding factors, so ‘tossing the caber’ may not cause knee osteoarthritis.

Another weakness of case control studies is that the information about past exposures is either captured from medical records or by asking patients. Medical records can be incomplete and patients might not recall all information correctly.

While case control studies are more subject to bias, because they start with the outcome, they are good for the study of rare conditions. They are also conducted retrospectively which means they are quick and less expensive than other analytical study types.

​

A cohort study is an observational study that tracks a group of people over time. Participants will be recruited and tracked to meet the needs of the research. For example:

• A cohort diagnosed with an illness may be tracked over time to learn about the progress of the condition, or the recovery rates of those affected.
• A birth cohort tracks a group from infancy, sometimes for decades, and collects information at various stages. The information collected varies across a range of socio-economic and health factors.
• A group who share a similar exposure (good or bad) is tracked forward in time to observe and measure an agreed outcome(s).

Cohort studies may include control groups, or results may be interpreted using before/after criteria, by comparing to earlier studies or other comparative material.

In some cases further people are brought into a study after it has commenced. This may be to improve representation of different groups, or to account for drop-off in participation numbers.

A cohort study which is prospective tracks participants through in real time. A retrospective cohort study uses historical data. Although prospective studies are generally ranked higher in the research hierarchy, these can be time consuming and expensive to reach results, and also prone to drop out from participants. On the other hand, while retrospective studies are quicker and cheaper to conduct, they must work with historical data that has already been collected, often for another purpose.

Advantages of cohort studies over other study types include being suitable to measure rare exposures, and as they start with the exposure and follow through to see what happens, they can measure multiple outcomes. Where interventional studies are not possible for ethical reasons, cohort studies can reliably indicate correlation between an exposure and outcome.  For example you cannot ask people to start smoking to see what effect this has. 

For our example question, we are going to focus on a prospective cohort study which starts with an exposure and follows forward in time to measure an outcome.

If you want to know about possible benefits of ceilidh dancing, a cohort study is ideal.

In our example three community centres hosting ceilidh dancing within the same local authority area were identified. Participants between the ages of 60 and 70 were recruited from those present in the community centres on days when ceilidh dancing was taking place. They were then separated into two groups – those who regularly participated in ceilidh dancing, and those who did not participate in ceilidh dancing. Other activities happening in community centres at this time included a coffee morning, visiting the library, art class and tai chi. For the purposes of this study those who participated in ceilidh dancing only occasionally were excluded from the study.

​

Randomised control trials (RCTs) are often used to test the efficacy of new drugs or surgical procedures, but they can also be used to measure the effects of diet or exercise regimes. These are intervention studies, and the controlled exposure is an important factor in minimising bias. There are a range of controls in RCTs and these are all designed to increase the reliability of results by minimising bias. There is a clear eligibility criteria. Participants are then randomised into either the intervention group or a control group who often receive a placebo - randomisation should ensure that each person has exactly the same chance of being placed in the control or intervention group. Double blinding means neither patients nor doctors know who is in each group.

Relative risk versus absolute risk reduction

Relative risk is the probability an event will happen in the intervention group compared to the control group. You should be wary of a paper which quotes only relative risk as it can make the results look more impressive than they really are, especially in smaller trials. Absolute risk reduction is the actual difference in risk between the intervention and control groups. Absolute risk is generally considered a more reliable statistic than the relative risk. The diagram below show how relative risk and absolute risk reduction are calculated for our example question.

The intervention group receives the Glasgow wonder vaccine. The control group receives a placebo.

In the diagram shown here we can see that the number needed to treat is the inverse of the Absolute risk reduction.

In a group of 20 receiving the intervention, one additional person benefits, compared to the control group; or 20 people need to receive the treatment for 1 person to benefit. Confidence intervals can also be useful. A 95% confidence interval provides a range between a lower and upper limit which you are 95% sure that the true effect of the intervention lies between. A big and well done study should have a narrow confidence interval.

The controlled exposure and the randomisation of participants in RCTs are important factors in minimising bias. However, there are still factors to be aware of when reading RCTs. For example, the effects of an intervention can only be detected if a large enough number of participants have been enrolled. RCTs with small sample sizes are often criticised for being underpowered. Nevertheless, recruiting large sample sizes can bring its own difficulties. It’s notoriously difficult to recruit participants for RCTs, so while a large sample size would seem best, if there are vast differences in the participants, geographical or age or some other difference, again there could be confounding factors at work.

It is also worth asking whether RCT participants are representative of the population that would receive that intervention. For example, participants with existing conditions may be excluded to control confounding factors, with the aim of ensuring observed outcomes are the result of the intervention being tested and not something else. But if a treatment is for a condition where comorbidities are common, that may mean the participants in the RCT are quite different from the group of patients who would actually need that intervention. It should be noted that exclusion criteria must be justified on scientific, safety or ethical grounds.

It’s also worth noting that getting results from an RCT can be both costly and time-consuming.

​

Systematic reviews synthesise available evidence on a given topic to produce reliable evidence suitable to inform clinical decision-making. A systematic review needs a:

• Focused and well-designed question which specifies how the authors will measure the outcome
• Robust search strategy, so all relevant studies are included.
• Transparent selection criteria for including or excluding studies
• Clear review process

The synthesised results should provide clear conclusions about how the evidence can be used in practice, even if that conclusion is that further research is required. By synthesising the results of smaller studies, systematic reviews can sometimes resolve contradictions and by providing generalisable conclusions are key to getting knowledge into action.

A forest plot shows the conclusions of a study, and allows the benefits of meta analysis to be seen in a glance. This Forest plot shows the meta analysis for our question about the efficacy of the Glasgow wonder drug for Scottish Flu.

The scale at the bottom indicates the vaccine is offering benefit. A 95% confidence interval is shown for each individual study. The black diamond shows the results when all the individual studies are combined together and averaged.

The horizontal points of the diamond are the limits of the 95% confidence interval.

While a systematic review of RCTs is generally considered the best level of evidence, synthesising the results of a range of study types can provide robust generalisable conclusions and are key to getting knowledge into practice. The method used to synthesise data should be appropriate for the study type, research question and individual study data. Cochrane provide guidance on appropriate synthesis methods for different study types (Cochrane handbooks).

As well as an eligibility criteria for inclusion in the study, it is important to appraise the quality of individual papers. A systematic review will only ever be as good as the original studies, and is likely to magnify any flaws.

The search strategy is key to ensure all relevant studies are captured, so look out for bias in the selection of studies. Publication bias is common, so successful published trials are included, but it’s common to miss data from unpublished, unsuccessful trials. There is also an English language bias in included papers.

Finally be aware that knowledge is always changing. Is your evidence still up to date? Look out for up to date reviews of evidence. Cochrane, for example, regularly review their systematic reviews to keep their evidence current.

Was this page helpful? Tell us what you think.