Competency A: Is the study of interest to the target population?

Subcompetency

Justification

Data Requirement

1. Is the study objective of interest?

The goal of study may or may not be relevant to the target population?

a. study objective, as II.F.c

What is the scientific question that the study is designed to answer?

b. primary hypothesis, as II.F.b.i

What other scientific questions is the study designed to shed light on, but not necessarily answer?

c. secondary hypotheses, as II.F.b.ii

Intention should be equivalent to the target objective, e.g. primary or secondary prevention, or acute or chronic intervention

d. intervention objective

2. Are the study outcomes of interest?

Primary outcome, for which the trial is best powered, may or may not be of interest to target population

a. primary outcome definition, as II.F.1.a and c

Other measured outcomes may or may not be of interest to target population

b. definition of all other outcomes, as II.F.1.a and c

3. Is the comparison of the intervention against the control group of interest?

The control group should be relevant to the target population

a.i. definition of control, and ii. reason for using this control

4. Were side effects of concern to the target population studied?

Important side effects may affect use of intervention, and evaluation of its effectiveness

a. definition of side effects monitored

Subjects that make it past a washout period are more selected

b. washout period, as II.E.1.a

Competency B: Are the trial subjects and the target population likely to belong to the same overall population? 

Subcompetency

Justification

Data Requirement

1. Was the source from which trial subjects were drawn similar to the target population?

Large differences may exist if study sites serve populations with different socioeconomic and racial distributions than the target popualtion

a. final study sites i. name, ii. location

Subjects from tertiary referral centers are generally sicker than those from primary referral centers

b. referral level of each study site

Different types of subjects go to different care settings

c. type of inpatient or outpatient facility 

Payment mechanism associated with illness of population

d. payment method of each site

2. Were eligible subjects randomly selected from the source population?

How subjects were initially identified can be a main source of selection bias

a. method of sampling for potential subjects

Highly selected subject populations limit generalizability

b. recruitment flowchart

3. Were enrolled subjects an unbiased subset of those eligible?

In conjunction with III.B.2.a-b., tells how selected is the enrolled population

a. number of subjects excluded for each exclusion criterion

Subjects that make it past a run-in period are more selected

b. run-in period

Subjects that make it past a washout period are more selected

c. washout period, as II.E.1.a

4. Were the trial subjects comparable to those of the target population?

Clinical characteristics of enrolled subjects should be similar to those of the target population

a. inclusion criteria

Application of exclusion rules yields more highly selected, and less generalizable, trial populations

b. exclusion criteria

Large differences in the clinical characteristics of the included and excluded groups suggests that the included group is less representative of those sampled

c. baseline characteristics (including age and sex), of included and excluded groups

Effects may not generalize to populations with higher or lower rates of the outcome then in the control group

d. outcome results in control group

Competency C: Is the setting of the trial sufficiently different from that of the target population that the trial's results are not applicable?

Subcompetency

Justification

Data Requirement

1. Might geographical or cultural differences limit the generalizability of the results?

Site(s) of study may be associated with unobserved variables that affect outcomes

a. final study site i. name, ii. location, as III.B.1.a

Knowing distribution of enrollment across sites helps assess impact of any one site

b. enrollment at each site

2. Was the health care delivery system different in such a way that limits generalizability?

Subjects from tertiary referral centers are generally sicker than those from primary referral centers, contributing to spectrum bias

a. referral level of each study site, as III.B.1.b

Unobserved variables associated with place of intervention may affect outcomes

b. type of inpatient or outpatient facility, as III.B.1.c

Unobserved variables associated with payment structure may affect outcomes

c. payment method of each site, as III.B.1.d

3.Has the clinical context changed since the time of the trial?

Technologies and co-interventions may have changed since the time the study was conducted, such that the trial results are not applicable anymore

a. start and end enrollment dates

External circumstances may have made follow-up more difficult for subjects at different times of the trial

b. date of close of follow-up 

Competency D: Is the tested intervention reproducible for the target population?

Subcompetency

Justification

Data Requirement

1. Is the intervention described clearly enough for local duplication?

Customization of intervention to local constraints may reduce applicability of trial results?

a. intervention description, as II.D.1.a

Overall compliance with assigned intervention should be comparable to local expectations

a. completion of assigned intervention, as II.D.3.a

Degree of compliance may not be achievable in the field if compliance enhancing method of trial very intensive

b. method of increasing compliance

Analysis of the study should appropriately adjust for degree of compliance

c. actual compliance, as II.D.3.b-d

Intervention effects may be due to co-interventions that may not be generalizable

a. co-interventions, as II.E.2.a-c