Meta-Analyses of Randomized Controlled Clinical Trials to Evaluate the Safety of Human Drugs or Biological Products

Administrative draft guidance, not enacted legislation. This captured publisher edition preserves the listed document and attachments; later developments are not independently consolidated.

Meta-Analyses of Randomized
Controlled Clinical Trials to
Evaluate the Safety of Human
Drugs or Biological Products
Guidance for Industry
DRAFT GUIDANCE
This guidance document is being distributed for comment purposes only.
Comments and suggestions regarding this draft document should be submitted within 90 days of
publication in the Federal Register of the notice announcing the availability of the draft guidance.
Submit electronic comments to https://www.regulations.gov. Submit written comments to the
Dockets Management Staff (HFA-305), Food and Drug Administration, 5630 Fishers Lane, Rm.
1061, Rockville, MD 20852. All comments should be identified with the docket number listed in
the notice of availability that publishes in the Federal Register.
For questions regarding this draft document contact (CDER) Scott Goldie at 301-796-2055 or
(CBER) Office of Communication, Outreach, and Development, 800-835-4709 or 240-402-8010.
U.S. Department of Health and Human Services
Food and Drug Administration
Center for Drug Evaluation and Research (CDER)
Center for Biologics Evaluation and Research (CBER)
November 2018
Drug Safety

Meta-Analyses of Randomized
Controlled Clinical Trials to
Evaluate the Safety of Human
Drugs or Biological Products
Guidance for Industry
Additional copies are available from:
Office of Communications, Division of Drug Information
Center for Drug Evaluation and Research
Food and Drug Administration
10001 New Hampshire Ave., Hillandale Bldg., 4th Floor
Silver Spring, MD 20993-0002
Phone: 855-543-3784 or 301-796-3400; Fax: 301-431-6353
Email: druginfo@fda.hhs.gov
https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm
and/ or
Office of Communication, Outreach and Development
Center for Biologics Evaluation and Research
Food and Drug Administration
10903 New Hampshire Ave., Bldg. 71, Room 3128
Silver Spring, MD 20993-0002
Phone: 800-835-4709 or 240-402-8010
Email: ocod@fda.hhs.gov
https://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm
U.S. Department of Health and Human Services
Food and Drug Administration
Center for Drug Evaluation and Research (CDER)
Center for Biologics Evaluation and Research (CBER)
November 2018
Drug Safety

Contains Nonbinding Recommendations
Draft — Not for Implementation
TABLE OF CONTENTS
I. 	INTRODUCTION............................................................................................................. 1
II. 	BACKGROUND AND SCOPE ....................................................................................... 1
III. 	THE QUALITY AND RELEVANCE OF CANDIDATE TRIALS ............................. 4
A. 	Basic Principles .............................................................................................................................. 4
B. 	Consistency with Standards for Adequate and Well-Controlled Trials ................................... 4
C. 	Outcome Definition and Ascertainment ...................................................................................... 5
D. 	Duration of Exposure and Length of Follow-Up......................................................................... 7
E. 	Subject Populations ....................................................................................................................... 8
F. 	Dosing and Comparators .............................................................................................................. 8
G. 	Relevance to Current Medical Practice ....................................................................................... 8
H. 	Availability of Subject-Level Data................................................................................................ 9
I. 	Quality over Quantity .................................................................................................................... 9
IV. 	THE IMPORTANCE OF PRESPECIFICATION AND TRANSPARENCY .......... 10
A. 	Potential for Bias, Multiplicity, and Other Errors ................................................................... 10
B. 	Meta-Analysis Protocol ............................................................................................................... 11
C. 	Reporting Results from a Meta-Analysis ................................................................................... 12
V. 	STATISTICAL METHODOLOGY CONSIDERATIONS ........................................ 13
A. 	Overview ....................................................................................................................................... 13
B. 	Statistical Properties of Risk Estimates and Hypothesis Tests ................................................ 14
C. 	Heterogeneity ............................................................................................................................... 16
D. 	Sensitivity Analysis ...................................................................................................................... 17
VI. 	STRENGTH OF EVIDENCE AND REGULATORY DECISIONS ......................... 18
A. 	Critical Factors in Determining the Strength of Evidence ....................................................... 18
B. 	Hierarchy of Evidence for Decision-Making ............................................................................. 19
VII. 	EXAMPLES .................................................................................................................... 21
A. 	Example 1: Antidepressant Use and Suicidal Events in Adults............................................... 21
B. 	Example 2: Tiotropium and Cardiovascular Events ................................................................ 24
VIII. 	REFERENCES ................................................................................................................ 25

Contains Nonbinding Recommendations
Draft — Not for Implementation
1
Meta-Analyses of Randomized Controlled Clinical Trials to	1
Evaluate the Safety of Human Drugs or Biological Products	2
Guidance for Industry1
3
This draft guidance, when finalized, will represent the current thinking of the Food and Drug	4
Administration (FDA or Agency) on this topic. It does not establish any rights for any person and is not	5
binding on FDA or the public. You can use an alternative approach if it satisfies the requirements of the	6
applicable statutes and regulations. To discuss an alternative approach, contact the FDA staff responsible	7
for this guidance as listed on the title page.	8
I. 	INTRODUCTION	9
This document provides guidance to applicants submitting investigational new drug applications	10
(INDs), new drug applications (NDAs), biologics license applications (BLAs), or supplemental	11
applications on the use of meta-analyses of randomized controlled clinical trials (RCTs) to	12
evaluate the safety of human drugs or biological products within the framework of regulatory	13
decision-making.2 This guidance is also intended for FDA reviewers and for third-party entities	14
that prepare or evaluate meta-analyses assessing the safety of drug products. Specifically, this	15
guidance describes the factors FDA intends to consider when evaluating the strength of evidence	16
provided by a meta-analysis studying the safety of drugs.	17
18
In general, FDA’s guidance documents do not establish legally enforceable responsibilities.	19
Instead, guidances describe the Agency’s current thinking on a topic and should be viewed only	20
as recommendations, unless specific regulatory or statutory requirements are cited. The use of	21
the word should in Agency guidances means that something is suggested or recommended, but	22
not required.	23
II. 	BACKGROUND AND SCOPE	24
Evaluating the safety of drug products, both before approval and after marketing, is a	25
fundamental responsibility of the FDA. This evaluation often requires combining and	26
summarizing information from multiple sources, and meta-analysis is a useful tool for this	27
purpose. The term meta-analysis, as used in this document, refers to the combining of evidence	28
from relevant studies using appropriate statistical methods to allow inference to be made to the	29
population of interest. The most common reason for performing a meta-analysis is to provide an	30
estimate of a treatment effect or measure of relative risk associated with an intervention and to	31
quantify the uncertainty about the estimated effect or risk, when data from a single existing study	32
are insufficient for this purpose, and the conduct of a new, large study would be impractical, take	33
1 This guidance has been prepared by the Office of Biostatistics in the Center for Drug Evaluation and Research, in
cooperation with the Center for Biologics Evaluation and Research, at the Food and Drug Administration.
2 For the purposes of this guidance, all references to drugs include both human drugs and biological products unless
otherwise specified.

Contains Nonbinding Recommendations
Draft — Not for Implementation
2
too long, or be unethical. The term meta-analysis sometimes refers to the quantitative synthesis	34
in a systematic review (Cochrane Handbook 2011) and the term systematic review refers to the	35
broader effort, including defining objectives and selecting and evaluating studies, as well as	36
synthesis. We use the term meta-analysis more broadly to include consideration of study	37
selection as well as overall design issues such as prespecification and reporting.	38
Unless a randomized controlled clinical trial is prospectively designed with a particular safety	39
outcome as its primary endpoint and sized accordingly, the trial may not have sufficient sample	40
size to detect important adverse consequences of drugs and to reliably evaluate whether there is	41
increased risk of such events. This is because most serious drug-induced adverse events (1) are	42
rare or (2) occur at only slightly increased frequency compared to background rates and are not	43
obviously drug-related (e.g., cardiovascular events, cancers). Meta-analysis is most useful in the	44
latter case, to detect and quantify an increased risk over the background rate of the safety event.	45
For the former case, when events are rare and not expected to occur in the target population,	46
meta-analyses may still be useful for improving the precision of the estimate of risk.	47
Meta-analysis factors into FDA’s evaluation of potential safety issues in a variety of ways:	48
• 	Meta-analyses may be conducted by sponsors and submitted to FDA as part of an	49
IND, NDA, BLA or supplemental submission.	50
• 	FDA may ask a sponsor to conduct a prospective meta-analysis, as it has	51
recommended for sponsors of new antidiabetic therapies to treat type 2 diabetes in the	52
draft guidance for industry, Diabetes Mellitus – Evaluating Cardiovascular Risk in	53
New Antidiabetic Therapies to Treat Type 2 Diabetes.3	
54
• 	FDA may initiate its own meta-analysis in response to safety signals that FDA is	55
aware of, using study data FDA has access to, but that may be unavailable to sponsors	56
and other researchers. These meta-analyses typically have prospective protocols to	57
address issues of bias and multiplicity, as discussed later in this document.	58
• 	FDA may evaluate a meta-analysis conducted by an external party that raises a safety	59
concern about a marketed product.	60
Because regulatory actions may stem from a meta-analysis, it is important that rigorous	61
principles be applied to such studies. In this guidance, the important principles underlying best	62
practices for safety meta-analysis and the way that FDA intends to factor adherence to those	63
principles into its decision-making are described. An overview of the most important principles	64
presented in this guidance is as follows:	65
• 	Prespecification and transparency are recommended, as they enable a thorough	66
evaluation of the meta-analysis.	67
• 	The criteria for selecting which trials to include should be determined prior to	68
conducting the meta-analysis. The selection of the studies should not be based on the	69
3 We update guidances periodically. To make sure you have the most recent version of a guidance, check the FDA
Drugs or Biologics guidance web pages at:
https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/default.htm.
https://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guidances/default.htm

Contains Nonbinding Recommendations
Draft — Not for Implementation
3
trial outcomes, but rather on trial quality and consistency of critical design elements,	70
and should be executed by parties masked to the outcomes of the trials, whenever	71
possible.	72
• 	The quality and relevance of the individual trials and the quality of the trial data are	73
critical determinants of the quality of the meta-analysis itself. Outcome ascertainment	74
and adequacy of exposure periods are two of the most important determinants of trial	75
quality.	76
• 	Meta-analysis conducted to meet safety objectives often requires re-purposing trials	77
that were originally designed to meet efficacy objectives. This can be challenging,	78
particularly if subject-level data are not available.	79
• 	Meta-analysis based solely on published trials is particularly problematic because of	80
the potential for bias and error, both known and unrecognized.	81
• 	Generally accepted principles of good statistical practice should be followed in	82
selecting the statistical methods to be used for meta-analysis (but this guidance is not	83
prescriptive as to the choice of method).	84
This guidance applies to meta-analyses conducted in both pre-market and post-market settings.	85
In the pre-market setting, the number and scope of trials may be limited, because the drugs are	86
not yet approved for marketing, and these limitations may affect the ability to address the safety	87
question of interest. In the post-market setting, the number and variety of trials available for	88
inclusion are usually larger, as is the number of parties able to conduct the meta-analysis. In both	89
pre- and post-market settings, the important principles guiding a well-planned and well-executed	90
meta-analysis apply.	91
This document focuses specifically on meta-analyses conducted for purposes of safety evaluation	92
using data from RCTs. Meta-analyses conducted to evaluate a product’s effectiveness, either	93
overall or within specific subgroups, are occasionally of interest to FDA, but the primary use of	94
meta-analysis in the regulatory setting is for assessment of product risk. While meta-analyses of	95
non-randomized studies may be informative for the assessment of certain safety outcomes, the	96
issues related to such a meta-analysis are more complex, and the interpretation of the results	97
more controversial. Meta-analyses of observational studies are therefore not addressed in this	98
guidance.	99
Meta-analyses are conducted for both exploratory and confirmatory purposes. The primary focus	100
of this guidance, however, is on meta-analyses with predefined hypotheses that are designed to	101
confirm a suspected risk associated with a drug rather than on exploratory meta-analyses.	102
The subsequent sections of this guidance provide a detailed discussion of the important elements	103
used in evaluating meta-analyses for regulatory purposes. In section III, the importance of the	104
quality and relevance of the component trials included in a meta-analysis and the quality of the	105
data from those trials are discussed. In section IV, the importance of prespecification and	106
transparency in designing, conducting, and reporting a meta-analysis is described. In section V,	107
the use of recommended statistical methods is discussed. In section VI, we summarize these	108
technical considerations and discuss how they may be factored into a regulatory decision.	109

Contains Nonbinding Recommendations
Draft — Not for Implementation
4
Section VII provides two examples illustrating the range of meta-analyses conducted for safety	110
evaluation and FDA’s use of the evidence provided by each.	111
III. 	THE QUALITY AND RELEVANCE OF CANDIDATE TRIALS	112
A. 	Basic Principles	113
Deciding what trials to include in a meta-analysis is an important step in the design and conduct	114
of a high-quality meta-analysis. The major determinants for this decision should be the quality	115
and relevance of the individual trials and the data collected in those trials. The component trials	116
of a meta-analysis should be able to address the safety objectives of the analysis and be of	117
sufficient quality to provide evidence useful for regulatory decision-making. The following are	118
important factors to consider in determining whether the individual trials and associated data are	119
of sufficient quality and relevance to ensure the validity of the meta-analysis:	120
• 	The extent to which the component trials are consistent with established standards for	121
the design and conduct of adequate and well-controlled clinical trials	122
• 	The quality and completeness of safety outcome ascertainment in each trial	123
• 	The appropriateness of exposure and follow-up periods for estimating risk	124
• 	The appropriateness of the component trials’ inclusion/exclusion criteria for defining	125
the population at risk	126
• 	The appropriateness of the comparator used in each trial and of the doses for the test	127
drug and comparator	128
• 	The relevance of the candidate trials to current medical practice	129
• 	The availability of subject level data from each trial	130
These factors are discussed further in the subsections that follow.	131
B. 	Consistency with Standards for Adequate and Well-Controlled Trials	132
The knowledge base, literature, and published guidelines for designing, conducting, and	133
analyzing well-controlled clinical trials to demonstrate efficacy in support of an NDA or BLA	134
are extensive and well-known (see, e.g., E9 Statistical Principles for Clinical Trials,	135
International Council on Harmonisation (ICH) of Technical Requirements for Pharmaceuticals	136
for Human Use). The same principles apply to the individual component trials of a meta-	137
analysis, and the extent to which the component trials satisfy these principles has strong bearing	138
on the quality of the meta-analysis to which they contribute. Notably, however, trials that are	139
well-designed to measure the effect of a drug on a particular efficacy outcome may not	140
necessarily be well-designed to measure an effect on another outcome, particularly an	141
uncommonly occurring safety outcome, as discussed further in section III.C.	142
Some study designs may cause a candidate trial to be discouraged from inclusion in the meta-	143
analysis. For example, randomized withdrawal studies, in which all subjects initially receive the	144
drug and are then randomized to either remain on the drug or withdraw to a placebo or active	145
control drug, may not be recommended for a safety meta-analysis. In these studies, subjects who	146
cannot tolerate the test drug are excluded from the randomized portion of the trial, and the study	147

Contains Nonbinding Recommendations
Draft — Not for Implementation
5
population may therefore not accurately represent the population at risk. Additionally, depending	148
on the period of exposure needed for the adverse effect to occur, the initial exposure to the drug	149
may result in events in both randomized groups and an underestimate of the relative risk.	150
Crossover studies in which subjects receive different treatments at different periods of time may	151
not be recommended for evaluating safety outcomes, if exposure to a treatment in one period can	152
result in an adverse event occurring in a later period. Washout periods for safety outcomes may	153
need to be longer than for efficacy outcomes. Other non-standard study designs such as	154
enrichment trials, trials with add-on therapies, adaptive trials, and trials stopped at interim may	155
raise similar issues.	156
C. 	Outcome Definition and Ascertainment	157
A high-quality meta-analysis has a carefully defined outcome variable with appropriate	158
ascertainment procedures prospectively implemented in the component trials such as specific	159
protocol-defined procedures for data collection and adjudication of safety outcomes. For	160
example, if the outcome of interest is myocardial infarction, the protocol might instruct the	161
investigators to collect laboratory and electrocardiogram data for suspected events during the	162
trial. The results of these procedures might then be subject to adjudication by an independent	163
panel to strengthen the evidence that a case event is real. Such procedures, however, are used	164
primarily to assess effectiveness outcomes (does the treatment reduce myocardial infarction	165
rates) and are not commonly used to assess safety outcomes, unless there is a specific concern	166
known and planned for prior to study start (e.g., cardiovascular outcomes in studies of Type 2	167
diabetes drugs; suicidal events in studies of antidepressant drugs). Although prospective	168
collection and adjudication of safety outcomes are desirable, they are usually not feasible,	169
particularly in the most common setting of evaluating a new, unanticipated safety signal with	170
data from trials already completed.	171
When the component trials are not prospectively designed to produce accurate ascertainment of	172
the meta-analysis safety outcome, retrospective identification and adjudication of events will	173
usually be recommended. In this situation, the safety outcome of interest should be clearly	174
defined, and the identification and adjudication of events should be performed while masked. For	175
example, in the antidepressants and suicidal events meta-analysis (section VII, Example 1),	176
where suicidality was not specifically assessed, predefined search criteria were applied to	177
adverse event data collected in the component trials. Based on the results of the search,	178
narratives of candidate events were created, and a group of experts masked to treatment	179
assignment classified the events into validated suicidal outcome categories. This resource	180
intensive effort required subject-level data not directly available in the original trial datasets. A	181
detailed meta-analysis protocol was developed that described the procedures necessary for	182
obtaining and adjudicating the outcome data of interest prior to implementing those procedures.	183
Measurement bias (such as an over- or under-estimation of the rate of events because of	184
imprecise or individualized interpretation of adverse event reporting) factors into determining	185
whether outcome ascertainment is sufficient for a high-quality meta-analysis. Biases common to	186
both treatment and control groups can occur when an outcome variable does not accurately	187
represent the safety outcome of interest (e.g., is not specific enough, causing many irrelevant	188
events to be reported, or is too narrowly defined, causing many events to be missed). Both	189

Contains Nonbinding Recommendations
Draft — Not for Implementation
6
reporting problems may result in reduced power or a biased effect measure, but will not	190
completely eliminate the ability to detect an effect. Of greater concern are reporting problems	191
that can affect treatment groups differently, as they can eliminate the ability to detect an effect	192
when one exists or that create the appearance of an effect when one does not exist.	193
Biased ascertainment of outcomes is one important concern in unmasked trials, where	194
investigators or subjects may unconsciously, or consciously, under- or over-report medical	195
events based on the known treatment assignment. Even in double-masked trials, there is a	196
potential for differential bias to occur in safety reporting, especially when safety outcomes were	197
not of primary interest in designing the trial. For example, a drug may cause discoloration of the	198
urine, which in turn may lead to more evaluations and subsequent diagnoses of kidney disease. If	199
anticipated, the trial protocols could have included an evaluation for kidney disease at scheduled	200
times during the trials, thereby reducing the potential for biased reporting of that safety outcome.	201
Several strategies should be considered to minimize the impact of measurement bias. The use of	202
safety outcomes that can be diagnosed readily and unambiguously, often called hard outcomes,	203
can help minimize bias due to outcome ascertainment in a meta-analysis. For example, if vital	204
status at the end of the study is known for all patients in all of the component trials, then use of	205
death as the safety outcome effectively avoids any potential for ascertainment bias. If ischemic	206
cardiovascular outcomes are of interest, ascertainment of myocardial infarction and stroke will	207
be less prone to ascertainment bias than less specific events such as transient ischemic attack or	208
angina. Excluding the less specific events or events that are difficult to ascertain objectively will	209
probably reduce the power of the meta-analysis to detect a safety signal as well as the precision	210
of the risk estimate that results, but the reduction in ascertainment bias may outweigh these	211
losses. Precision and power can be quantified and reported with the meta-analysis results,	212
whereas bias is typically unknown and difficult to measure. In general, reducing bias in a meta-	213
analysis should be given greater weight than increasing precision and power.	214
It is important to define the period within which the safety outcome of interest is to be measured.	215
For example, a safety outcome corresponding to the occurrence of anaphylactic events may call	216
for the primary focus to be placed on the period of initial drug exposure, with a secondary focus	217
on the entire drug exposure period. Including events beyond the initial exposure period may	218
result in underestimation of the risk attributable to the drug. In cases where it is known that the	219
effect of the drug diminishes when the drug is stopped, it might be recommended for the primary	220
analysis to count outcomes only during the time a subject is on the drug (such as an on-treatment	221
analysis).	222
Ideally, outcome definition and ascertainment should be as uniform as possible across the	223
component trials. Trial-to-trial differences can introduce heterogeneity in safety outcomes,	224
increasing the variability of the meta-analytic estimate of risk. Differences in outcome definition	225
and ascertainment may be confounded with other trial design or subject population	226
characteristics, making observed differences in risk measures difficult to interpret.	227
Outcome definition and ascertainment are particular problems for meta-analyses that rely	228
exclusively on published trial data. Information taken from published articles about the	229
component trials may be incomplete or lack specificity. Publications may not report on the safety	230

Contains Nonbinding Recommendations
Draft — Not for Implementation
7
outcome of interest, and even when the outcome is reported, important details may be lacking,	231
including whether the event occurred on or off randomized treatment and whether the outcome	232
was defined a priori and uniformly across trials. Protocols, study reports, and subject-level data	233
from the component trials are often important to determine whether the trial outcomes are	234
adequate for supporting a high quality meta-analysis.	235
The definition of the safety outcome, the source data and any adjudication procedures that may	236
have been employed should be prespecified in the meta-analysis protocol and consistently	237
applied to all component trials, if possible (see Section IV.B).	238
D. 	Duration of Exposure and Length of Follow-Up	239
The duration of exposure and length of follow-up for each of the candidate trials should be	240
factored into the criteria for trial inclusion. For an outcome with delayed appearance, such as	241
cancer or bone injury, the inclusion of short-term trials may not be recommended. When subject-	242
level data are available, analysis methods can be used to identify and account for differences in	243
trial duration across studies (see Section V). Without subject-level data, it may not be possible to	244
account for differences in duration, depending on the level of detail provided by the summary	245
information available from each trial, and some trials may need to be excluded as a result.	246
Subjects prematurely stopping assigned drug or withdrawing from the trial can affect the	247
comparability of subject groups with respect to safety outcomes ascertained over the course of	248
the treatment or study period. The dropout pattern may result in dissimilar observation time	249
between the two groups, resulting in more opportunity to observe the safety outcome in one	250
group compared to the other. Simple adjustments for person-time of observation may not be	251
sufficient to correct for non-comparability, because these adjustments assume constant hazards	252
across time. The risk of the event may not be constant over time if, for example, the safety	253
outcome tends to occur either early or late during treatment. Time-to-event analysis may also be	254
insufficient if the dropout rates are indicative of informative censoring; for example, if the	255
adverse events resulting in early discontinuations are similar to or predecessor events of the	256
safety outcome.	257
When reviewing the component trials of a meta-analysis, it is important to consider the	258
possibility of differential follow-up and informative censoring. Examining summary statistics	259
and graphics by subject group of on-assigned drug time and follow-up time is usually helpful for	260
this purpose, as is an examination of the stated reasons for stopping assigned-drug or	261
discontinuing participation in the trial by subject group. The criteria for excluding individual	262
trials for these reasons should be specified a priori and described in the meta-analysis protocol	263
and analysis plan (see Section IV). If incorporated in the trial inclusion and exclusion criteria	264
(applied to determine the component trials of the meta-analysis), a review to identify differential	265
dropout rates should be performed masked to the safety outcome measurements. Regardless of	266
the decision on inclusion, data summaries should be provided in the meta-analysis report to	267
permit consideration of these issues.	268

Contains Nonbinding Recommendations
Draft — Not for Implementation
8
E. 	Subject Populations	269
Wherever possible, the subject population for component trials should reflect the patient	270
population hypothesized to be adversely affected by the drug. For cardiovascular safety	271
outcomes, for example, trials that enrolled subjects with pre-existing cardiovascular risk factors	272
may improve the ability of the meta-analysis to detect any risk associated with the drug.	273
Conversely, including trials that excluded subjects with certain risk factors may limit the ability	274
to detect risk. The inclusion/exclusion criteria of the component trials should be reviewed to	275
determine if the corresponding subject populations are consistent with the objectives of the meta-	276
analysis.	277
F. 	Dosing and Comparators	278
Although uniformity of dosing regimens and therapeutic indications studied across component	279
trials is desirable, it may be that trials including other doses or conducted in other indications can	280
contribute to the meta-analysis. For example, in some circumstances, it may be assumed that if a	281
safety event is not observed at doses higher than the dose or doses approved, it should not occur	282
at the approved dose or doses. In this scenario, including trials with dosing higher than the	283
approved dose might be used to rule out an association. Information on dose response	284
relationships may also support a possible relationship between drug use and safety outcomes.	285
Similarly, including trials for indications outside the indication of specific interest may be useful	286
in a safety meta-analysis, if it can be assumed that the association would not depend on the	287
indicated use. Such assumptions can be examined to some extent through sensitivity analyses	288
conducted on subsets of trials at particular doses or in particular indications (see Section V.D).	289
The suitability of the comparator drugs in the candidate trials should also be factored into the	290
meta-analysis inclusion criteria. In some situations, the ideal comparator is a placebo, since a	291
placebo cannot cause the safety outcome under investigation. However, placebo-controlled trials	292
may not be feasible or ethical in certain disease areas. If trials with active drug comparators are	293
used, attempts should be made a priori to determine if the active comparator is associated with	294
the safety outcome of interest. The protocol specifications for concomitant therapy in the	295
individual trials are also relevant, since concomitant therapies may be associated with the safety	296
outcome.	297
G. 	Relevance to Current Medical Practice	298
Changes over time in the practice of medicine may affect the usefulness of some trials for	299
contributing data to a meta-analysis. Older trials may no longer be relevant, if medical practice	300
has changed such that current practices are able to prevent or reduce the occurrence of the safety	301
outcome under investigation. Sensitivity analyses can be used to examine estimated risks as a	302
function of the dates the component trials were conducted to determine if calendar trends pose a	303
problem.	304

Contains Nonbinding Recommendations
Draft — Not for Implementation
9
H. 	Availability of Subject-Level Data	305
The availability of subject-level data is an important consideration in deciding which studies to	306
include in the meta-analysis. For reasons already discussed, subject-level data improve the	307
quality of the meta-analysis by providing the ability to evaluate important quality factors of the	308
component trials and possibly correct for any deficiencies identified, particularly poor outcome	309
assessment or insufficient exposure periods. Subject-level data also allow for a broader range of	310
analysis methods to be used and an examination of subgroups (see Section V). Note, however,	311
that in some cases, meta-analyses based on only trial-level summary data may be able to identify	312
or rule out risks associated with a drug. If so, then the criteria for determining which trials to	313
include in a trial-level meta-analysis should be carefully considered; the principles described in	314
this section apply to trial-level meta-analyses just as they do to subject-level meta-analyses.	315
I. 	Quality over Quantity	316
There is often a desire to include as many trials as possible in a meta-analysis to both increase	317
the sample size and enhance the generalizability or external validity of the findings. Including	318
trials that are of poor quality, however, does not accomplish this. The findings from a meta-	319
analysis of a limited set of trials, selected with careful attention to trial and data quality, the	320
intended use of the product, and combined using appropriate statistical methods, will yield a	321
more informative answer to the safety question under investigation than a broader set of trials	322
that includes trials of poor quality.	323
The criteria used to decide which of the candidate trials will be included in a safety meta-analysis	324
should be carefully developed, taking into consideration outcome ascertainment and exposure	325
periods as well as other factors described in the previous subsections. The choices of subject	326
populations, dosing regimens, comparator arms, background therapy, standard of care, and other	327
trial features that comprise the meta-analysis inclusion criteria will affect the validity and	328
interpretation of the meta-analysis findings. Broad inclusion criteria (such as including trials	329
where outcomes may not be reliably assessed) will likely compromise the internal validity of the	330
meta-analysis without necessarily improving the external validity. The criteria for trial inclusion	331
should be well-documented in advance of conducting the meta-analysis. This topic is discussed	332
in section IV.	333
Trial inclusion decisions are particularly important for network meta-analyses, which are	334
designed to assess safety concerns about one drug relative to another, when the two may not	335
have been studied in the same randomized trial (Ohlssen, Price et al. 2014). Direct comparisons	336
between drugs within the individual trials included in a network meta-analysis are used to form	337
indirect comparisons between the two drugs of interest. Because some of the subject group	338
comparisons are made across trials, it is important that the trials involved in a network meta-	339
analysis be similar in design, subject populations, outcome definitions, and medical practice.	340
Although the principles in this guidance apply to network meta-analyses, network meta-analyses	341
have unique considerations beyond what is discussed in this guidance.	342

Contains Nonbinding Recommendations
Draft — Not for Implementation
10
IV. 	THE IMPORTANCE OF PRESPECIFICATION AND TRANSPARENCY	343
The extent of the information that should be considered both before and following the conduct of	344
a meta-analysis to adequately establish prespecification and transparency is discussed in this	345
section.	346
A. 	Potential for Bias, Multiplicity, and Other Errors	347
Meta-analysis is a form of retrospective research in that most meta-analyses are conducted based	348
on published clinical trials or trials already completed and whose results are known. It is	349
important to minimize the potential for bias and other errors from sources that are often	350
characteristic of retrospective research, including:	351
• 	Prior knowledge of individual study results when selecting the studies to be included	352
in the meta-analysis	353
• 	Inclusion of the hypothesis-generating study in a meta-analysis designed to confirm	354
the hypothesis	355
• 	Inability to determine the impact of multiplicity on the reported results	356
Special care is recommended when including trials whose results regarding the safety outcome	357
of interest are known prior to the conduct of the meta-analysis. Information describing the	358
knowledge base at the time the meta-analysis was planned will aid in determining the extent of	359
possible bias that may affect interpretation of the results (e.g., trial outcomes influencing	360
selection of trials). Prespecification of the criteria used to decide which trials to include before	361
decisions about individual trials are made is a major mechanism to minimize bias and can help	362
lessen the impact of this knowledge on the validity of the meta-analysis findings.	363
As stated earlier, our focus is on meta-analyses conducted to confirm a hypothesized safety risk.	364
If a safety hypothesis was generated from the results of a specific clinical trial, then drawing	365
inference from a meta-analysis that includes that trial is problematic. In this case, hypothesis test	366
results and confidence intervals about the risk estimate are not readily interpretable. If the goal of	367
the meta-analysis is to summarize existing information and not to make formal inference, then	368
including the motivating trial may be reasonable. If the motivating trial is included, sensitivity	369
analyses should be performed with and without the motivating trial to investigate its impact on	370
the meta-analysis results (See Section V).	371
Another problem frequently encountered when evaluating the evidence provided by a meta-	372
analysis is the potential for spurious findings due to multiple hypotheses being tested, multiple	373
outcomes being evaluated, multiple or iterative analyses being conducted and multiple subject	374
subgroups being investigated (Bender, Bunce et al. 2008). The result is inflation of the Type I	375
error probability associated with the tests of hypotheses, making the meta-analysis conclusions	376
difficult to interpret. When each of these sources of multiplicity is not well-described in advance,	377
it is impossible to apply a statistical method of adjustment for multiplicity because the full range	378
of factors that were evaluated cannot be determined. And even when the analysis plan does	379
contain a clear description of the sequence of tests to be conducted (across hypotheses,	380
outcomes, subgroups, etc.), there may be too little power available for each of the tests to	381

Contains Nonbinding Recommendations
Draft — Not for Implementation
11
confirm the hypothesized safety signal, when appropriate adjustments are applied. Adequate	382
planning and prespecification of meta-analysis objectives and tests of hypotheses may help	383
minimize, to some extent, problems due to multiplicity.	384
B. 	Meta-Analysis Protocol	385
Prespecification, completeness, and transparency are important principles in the reporting of a	386
meta-analysis, and the reporting begins with the meta-analysis protocol. The protocol should	387
contain a detailed description of the information available prior to designing the meta-analysis	388
that motivated the research. Potential problems anticipated in designing the meta-analysis and	389
the methods planned to manage those problems should be documented. The protocol should be	390
finalized prior to conducting the meta-analysis and, importantly, be in place prior to the selection	391
of the component trials.	392
The meta-analysis protocol should be available through advance publication or other methods of	393
distribution. This practice has been widely adopted for clinical trials via use of the web site,	394
https://clinicaltrials.gov/. There are several repositories for the protocols, such as PROSPERO	395
(Chien, Khan et al. 2012). Having protocols appear in the same publication as the meta-analysis	396
findings is generally insufficient to provide such assurance.	397
Following is a list of the broad topics a meta-analysis protocol should include. Each is discussed	398
further in the paragraphs that follow:	399
• 	The planned purpose of the meta-analysis	400
• 	The background information available at the time of protocol development that	401
motivated the meta-analysis	402
• 	The design features of the meta-analysis, including outcome definition and	403
ascertainment, exposure periods and assessment, comparator drugs, and target subject	404
population	405
• 	A description of the search strategy that will be used to identify candidate trials and	406
the criteria that will be applied for trial selection	407
• 	The analysis strategy for conducting the meta-analysis, including planned subgroup	408
analyses and sensitivity analyses	409
Planned purpose: The planned purpose should be clearly stated in the protocol, with sufficient	410
background material to explain the reason for conducting the meta-analysis. Examples include:	411
to estimate a specific risk, to evaluate risk in a subgroup of patients, to identify risk factors or	412
effect modifiers, to examine whether risk changes over time, or to assess accumulating evidence	413
on product safety as ongoing studies of the product complete. The weight of evidence provided	414
by a meta-analysis planned specifically to provide new information or update existing	415
information about a hypothesized risk of a drug would be considered more compelling than that	416
from a meta-analysis designed to explore safety signals or relationships among variables with no	417
stated hypothesis. The distinction is analogous to that made between exploratory and	418
confirmatory clinical trials in drug development, with the latter guided by pre-specified	419
objectives reflected in a final protocol prior to study start.	420

Contains Nonbinding Recommendations
Draft — Not for Implementation
12
Background information: The protocol should describe the information available prior to	421
designing the meta-analysis that served as motivation for the research. Examples include safety	422
risks identified in a randomized clinical trial of the drug or another drug in the same class,	423
potential relationships between exposure and safety outcomes shown in post-marketing studies	424
of health care data, or potential relationships identified during review of spontaneous adverse	425
event reports.	426
Design elements: A clear prospective plan can help protect a meta-analysis against bias and	427
inflation of Type I error by providing the rationale for each design element based on the	428
knowledge and information available during planning. Without such a plan, it is difficult to	429
determine which analyses were planned and which were exploratory or suggested as the analysis	430
progressed. Important among the design elements is outcome ascertainment, including whether	431
the outcome data were collected as part of the design of the individual trials or retrospectively	432
collected as part of the meta-analysis; whether the outcome was actively collected from subjects	433
or passively collected via subject adverse event reports; and whether the outcome was	434
adjudicated, and, if so, how. Clear definitions of the outcome variable and the follow-up period	435
for its ascertainment should be stated, with rationale for the choices thereof. The protocol should	436
state the specific exposure of interest and the comparator. If multiple exposures (multiple doses	437
of one drug or multiple drugs within a class) or comparators are to be combined, this should be	438
stated, and the primary exposure and comparator should be identified.	439
Search and selection criteria: The protocol should describe the search algorithm that will be used	440
to identify candidate trials to be considered for inclusion in the meta-analysis. Details should	441
include a description of the sources to be searched, such as the literature or online resources (e.g.,	442
https://clinicaltrials.gov/, https://www.accessdata.fda.gov/scripts/cder/drugsatfda). The trial	443
inclusion criteria should be described in detail, with the rationale given for each factor used as a	444
basis for trial selection (see Section III.D). The selection process should be masked to study	445
outcome and described in the meta-analysis protocol. Note that even if results are known to some	446
parties, it may be possible to find others who could apply the trial selection criteria for the meta-	447
analysis in an unbiased manner.	448
Analysis strategy: The protocol should describe the primary analysis strategy for achieving the	449
study objectives as well as any sensitivity analyses and subgroup analyses planned. The	450
statistical methods for the primary analysis should be stated in the protocol, with additional	451
details provided in the statistical analysis plan. The analysis plan should be finalized prior to	452
conducting the meta-analysis, analogous to the recommendation that a clinical trial’s analysis	453
plan be finalized prior to unmasking of treatment codes. Sensitivity analyses should be planned a	454
priori to assess the impact of any unverifiable assumptions on the meta-analysis results. The	455
factors that should be considered in choosing the statistical methods are discussed in section V.	456
C. 	Reporting Results from a Meta-Analysis	457
Results of a meta-analysis should be reported in a way that provides transparency and full	458
disclosure of the many decisions involved in conducting the meta-analysis. The report should	459
provide enough detail about the selection of trials, the statistical methods applied in the analyses,	460
the results of those analyses, and the rationale for and results of any sensitivity analyses carried	461

Contains Nonbinding Recommendations
Draft — Not for Implementation
13
out to enable an evaluation of the impact of bias and multiplicity on the findings and to assess	462
their strength and credibility. The Preferred Reporting Items for Systematic Reviews and Meta-	463
Analyses (PRISMA) statement provides some recommendations on the reporting of systematic	464
reviews and meta-analyses (Moher, Liberati et al. 2009). Although not all the PRISMA	465
components directly apply to meta-analyses that are the focus of this guidance, they should be	466
considered.	467
The report should include the results of the search algorithm used to identify candidate trials and	468
contain enough detail to evaluate the search. The selection process used to determine which of	469
the candidate trials were selected for inclusion in the meta-analysis, which should be by applying	470
the pre-specified criteria, should also be reported. Accounting for the trials that were not selected	471
and the reasons for their exclusion is as important as accounting for the trials that were selected.	472
Characteristics of the individual trials included in the meta-analysis should be summarized,	473
including individual trial design features, durations of exposure and follow-up periods, and	474
patient populations. The report should describe the sources of any trial-level and subject-level	475
data used in the meta-analysis. Summaries of subject-level characteristics should also be	476
provided for the trials to be included in the meta-analysis, including basic demographics,	477
concomitant medication usage, and other important factors thought to impact the exposure-risk	478
relationship under investigation.	479
Any departures from the planned statistical methods should be described, as well as the rationale	480
for those departures. Additional sensitivity analyses determined to be needed after the protocol	481
was finalized, because of characteristics of the particular trials selected, unanticipated data issues	482
encountered during analysis (e.g., zero-event trials), or preliminary findings needing further	483
exploration, should be described and justified.	484
Results corresponding to the pre-specified test of hypotheses, supporting analyses, and	485
sensitivity analyses should be provided in a clear and concise manner, with sufficient detail to	486
aid in interpretation. Point estimates of absolute or relative risk should be accompanied by	487
measures of uncertainty, e.g., confidence intervals. Forest plots are recommended for providing	488
visual summaries of the results from each of the component trials relative to the results of the	489
meta-analysis. These plots are useful in describing study-to-study heterogeneity.	490
V. 	STATISTICAL METHODOLOGY CONSIDERATIONS	491
A. 	Overview	492
In this section, general recommendations for selecting the statistical methods that will be used to	493
combine evidence from the component trials in a safety meta-analysis are discussed. It is not the	494
goal of this guidance to propose any best method, as no method performs best in all settings, nor	495
is it the goal to restate the relative performance of methods that are well-established and have	496
been compared in the literature pertinent to safety meta-analyses. Rather, this guidance	497
recommends that the statistical methods used in a meta-analysis be aligned with the analysis	498
objectives and hypotheses under investigation and be consistent with the study designs and data	499
collected in the individual trials. The choice of methods should be justified based on the stated	500

Contains Nonbinding Recommendations
Draft — Not for Implementation
14
objectives and documented in the protocol or analysis plan (see Section IV); sensitivity of the	501
results to departures from assumptions required for correct application of the methods should be	502
examined as part of the planned analysis strategy. Note that although this guidance generally	503
recommends the use of subject-level data when available, it is recognized that meta-analyses	504
conducted based on trial-level data only may be useful in certain settings (see Section VI.B.).	505
The recommendations in this section apply to trial-level meta-analysis as well.	506
The material in this section falls into three broad areas: statistical properties of the analysis	507
methods, heterogeneity, and sensitivity analysis.	508
B. 	Statistical Properties of Risk Estimates and Hypothesis Tests	509
The statistical approach for a safety meta-analysis should ensure that the estimator and/or	510
hypothesis test have good statistical properties, namely that the resulting risk estimate is	511
approximately unbiased and sufficiently precise, the standard error of the estimated risk is	512
accurate, and the associated confidence intervals have accurate coverage properties. Tests of	513
hypotheses about the risk should have good operating characteristics, i.e., the Type I and II error	514
probabilities should be accurate, and the power maximized given the data available.	515
An important principle involved in estimating risk from a meta-analysis is that the randomized	516
comparisons of the individual trials should be maintained when analyzing the combined data. In	517
other words, when comparing drug A to drug B, subjects randomly assigned to drug A in a single	518
trial are compared to subjects assigned to drug B from the same trial and not to subjects from	519
other trials. In the statistics literature, this is referred to as stratifying the analysis by trial.	520
Intuitively, this implies that the overall comparative measure of risk is based on combining the	521
comparative risk measures from the individual trials using recommended statistical methods.	522
Stratifying the analysis by trial is preferred to combining data across all subjects in the	523
component trials by subject group prior to estimating risk, often referred to as simple pooling, as	524
this ignores the randomized comparisons of the individual trials and can produce misleading	525
findings.	526
When one or more of the trials included in the pooling does not employ a one-to-one	527
randomization scheme, simple pooling of trial data can result in a phenomenon known as	528
Simpson’s paradox (Chuang-Stein and Beltangady 2011). When there are large sample size	529
disparities among the trials with different randomization allocations, the impact of this	530
phenomenon can be quite large. The hypothetical example in Table 1 illustrates an extreme	531
example of Simpson’s paradox in which, for each of four trials, the estimated risk of a safety	532
event is identical for both Drug A and Drug B. With simple pooling, however, the risk for Drug	533
A appears to be more than twice as high as that for Drug B (12.8 percent vs. 6.2 percent).	534

Contains Nonbinding Recommendations
Draft — Not for Implementation
15
Table 1. An Illustration of Simpson’s Paradox from Incorrect Pooling of Data	535
Drug A 	Drug B
Trial 	Events 	Patients 	Risk 	Events 	Patients 	Risk
1 	1 	100 	1.0% 	2 	200 	1.0%
2 	1 	100 	1.0% 	2 	200 	1.0%
3 	200 	1200 	16.7% 	50 	300 	16.7%
4 	2 	200 	1.0% 	2 	200 	1.0%
Total 	204 	1600 	12.8% 	56 	900 	6.2%
It is sometimes of interest to combine multiple doses of a drug in one or more of the component	536
trials to gain statistical power and improve the precision of the risk estimate in a meta-analysis.	537
The combination of arms should be performed within each trial and the overall analysis should	538
still be stratified by trial to avoid Simpson’s paradox in this setting.	539
Sparse data resulting from rare safety outcomes pose particular problems in a meta-analysis. The	540
statistical methods chosen for the analysis should perform well when the number of outcome	541
events is very small in one or more of the component trials or in one or more treatment groups	542
within a trial. Some commonly used methods perform well when there are ample events, but not	543
so well when events are sparse (Bradburn, Deeks et al. 2007). For example, inverse variance	544
weighting involves estimating risk with a weighted estimate of trial results, where weights are	545
computed as the inverse of the trial level variance estimates. With sparse data, the estimated	546
variances may not be well-determined, resulting in an unstable risk estimate. If some of the	547
component trials have no events, the choice of methods is even more limited.	548
We do not recommend the use of continuity corrections, one approach for handling zero-event	549
trials or trials with zero events in one or more treatment groups. Because it may not be apparent	550
with some software packages if and how continuity corrections are incorporated, caution is	551
needed to avoid their inadvertent use. Continuity corrections approaches generally involve	552
adding small quantities to the zero event counts prior to analysis. Although their use allows zero-	553
event trials to be included in a meta-analysis, the results may be biased. Note that the use of ratio	554
effect measures, such as the risk ratio or hazard ratio, is more challenging in the presence of	555
zero-event trials than is the use of risk difference measures, such as the Mantel-Haenszel risk	556
difference (Greenland and Robins 1985). Another approach is to consider Bayesian methods for	557
meta-analysis (Sutton and Abrams 2001) (Spiegelhalter, Abrams et al. 2004), which can	558
incorporate information on trials with no events, even when a relative risk measure is used. The	559
performance of any proposed method for dealing with zero-event trials should be established and	560
the choice justified for a particular meta-analysis application.	561
The ability to replicate the results of a meta-analysis with an independent study will increase the	562
persuasiveness of the findings. One such approach is to analyze one or more newly available	563
trials to see if the results agree quantitatively and/or qualitatively with the results of an existing	564
meta-analysis. Alternatively, an existing meta-analysis can be updated as new trials become	565
available. Although this sequential approach to meta-analysis provides an efficient way to update	566
risk estimates with new study results, the impact of repeated hypothesis tests about that risk	567
should be taken into account (Whitehead 1997).	568

Contains Nonbinding Recommendations
Draft — Not for Implementation
16
C. 	Heterogeneity	569
In any meta-analysis, heterogeneity of the drug effect among the component trials is expected	570
and should be addressed at the design stage. If there is strong reason to believe trials will have	571
importantly different drug effects based on known factors such as characteristics of the trial	572
populations, the specific interventions, or other trial design features, then the statistical analysis	573
should account for this expected variation. This may involve use of a statistical model that allows	574
for different effects based on known factors. Alternatively, it may be of interest to conduct	575
separate analyses for distinct groups of trials that vary with respect to one or more important	576
design factors. For example, if the set of component trials consists of both placebo-controlled	577
and active-controlled trials, a reasonable approach would be to perform a meta-analysis for each	578
group of trials separately, taking into account what is known about the active control effect. In	579
some situations the trials may be so heterogeneous that it is not possible to conduct a meta-	580
analysis.	581
The most common approach to account for residual heterogeneity in drug effects across trials,	582
after accounting for expected heterogeneity attributable to known factors, is to incorporate	583
individual-trial treatment effects in the analysis model as either fixed or random effects. The	584
meta-analysis literature includes a great deal of discussion about choosing between the two	585
(Borenstein, Hedges et al. 2010). In the context of a meta-analysis, use of a fixed effects model is	586
often interpreted as assuming a common effect exists across the trials, in contrast to the use of a	587
random effects model, where the effects are assumed to vary across trials according to some	588
probability distribution. This distinction is not usually made in other, similar areas of application,	589
e.g., in managing centers in a multi-center trial (Senn 2000). In the statistics literature on multi-	590
center trials (see, e.g., ICH E9), use of a fixed effects model is not as restrictive in that the model	591
can specify either a common effect across centers or different, but non-random, effects for each	592
center (i.e., by including the center by treatment interaction terms in the model). In the latter	593
case, interest lies in estimating an average effect across the centers. Similarly, in meta-analysis, it	594
may be desirable to allow effects to vary by trial with the inclusion of treatment by trial	595
interaction terms in the fixed effects model, and, in this case, averaging across trials with	596
appropriate methods provides the drug effect of interest.	597
Use of a random effects model in a meta-analysis implies an interest in estimating the average	598
effect for some larger population of trials that are believed to be adequately represented by the	599
trials in the analysis, and this parallels use of a random effects model in a multi-center trial; i.e.,	600
interest lies in estimating the average effect for a larger population of centers for which the trial’s	601
centers provide adequate representation. Arguments may be made against the use of random	602
effects models in a meta-analysis based on the belief that the trials available for analysis are not a	603
random sample of some larger population of trials — that is, all relevant trials are included in the	604
meta-analysis. It has been pointed out, however, that even when there is no interest in making	605
inference to a larger population of trials, use of a random effects model may produce more	606
appropriate results, due to the better characterization of the between- and within-trial variance in	607
the estimation process (Permutt 2003).	608
Both frequentist and Bayesian methods are available for random-effects meta-analysis, and the	609
difference between the two lies in the assumptions made about the distributions of the random	610

Contains Nonbinding Recommendations
Draft — Not for Implementation
17
effects, with Bayesian methods offering more flexibility (Muthukumarana and Tiwari 2016).	611
Bayesian methods also allow multiple sources of variation to be incorporated in the modeling	612
and estimation process. For example, in a meta-analysis designed to examine a specific risk for a	613
class of drugs, one may assume there is a component of variation among different drugs within	614
the class and a separate component among trials involving a single drug. To date, the Agency has	615
limited experience in evaluating meta-analysis submissions that use Bayesian methods, but	616
supports the consideration of Bayesian and other methods that achieve the desired properties	617
discussed in this section.	618
For safety meta-analysis, the goal is to determine whether a significant risk is causally related to	619
exposure to the drug, and the power available for that test should be maximized. Use of a fixed	620
effects model will usually provide optimal power for detection of risk and also reflects a primary	621
interest in the average effect among only those trials included in the meta-analysis. The parallel	622
with the establishment of efficacy for drug approval is relevant here. The selective populations	623
included in premarket efficacy trials may not fully represent the broader patient populations seen	624
in clinical practice, but are still central in making regulatory decisions. However, for the	625
quantification of the risk itself, a random effects model might be more appropriate, as the	626
incorporation of the between-trial variance might better reflect the uncertainty associated with	627
the risk estimate. Under all scenarios, the statistical inference should properly reflect the	628
assumptions made for the fixed or random effects model used; in particular, the variance of the	629
estimator should properly reflect whether the trial effects are constant, non-constant, or random.	630
D. 	Sensitivity Analysis	631
Sensitivity analyses play an important role in examining the impact of meta-analysis design	632
decisions on the findings as well as the strength of evidence provided by the meta-analysis. The	633
goal of any sensitivity analysis should not be to search for additional findings, but to support and	634
understand the primary findings of the meta-analysis. Trial inclusion criteria, outcome definition,	635
time period within which the safety outcome of interest is to be measured, and analysis method	636
are examples of design characteristics that may be varied as part of a sensitivity analyses.	637
For example, if a meta-analysis protocol and statistical analysis plan called for including only	638
those safety events that occurred during exposure periods in the risk estimate, then a sensitivity	639
analysis that included all reported events, regardless of whether subjects were on or off drug,	640
could provide important information about the observed risk estimate. A decreased event rate in	641
off-treatment periods could, in this example, support causality (depending on the hypothesized	642
mechanism). Similarly, a meta-analysis that included one very large study contributing a large	643
proportion of subjects and events could raise a concern that it was overly influencing the meta-	644
analytic results. A sensitivity analysis that excluded that study would have reduced numbers of	645
subjects and events and lower power to yield a significant finding, but a risk estimate that was	646
consistent with the original estimate would add to the weight of evidence of the finding.	647
It is often of interest to examine the consistency of findings from a meta-analysis across	648
subgroups based either on trial-level or subject-level characteristics. Trial-level factors that might	649
be of interest include the comparator treatment, dose and duration of treatment, background	650
therapy, and subject inclusion criteria. Subject-level factors may vary within trials, and subject-	651

Contains Nonbinding Recommendations
Draft — Not for Implementation
18
level data are required to provide estimates for each subgroup. In the antidepressant meta-	652
analysis example of section VII, age was an important factor of specific interest, because the	653
meta-analysis was motivated by an earlier meta-analysis of pediatric subjects. The number of	654
subgroups to be examined should be kept to a minimum to avoid the consequences of multiple	655
testing. Given the multiplicity issues, subgroup findings are seldom viewed as definitive in safety	656
meta-analyses.	657
VI. 	STRENGTH OF EVIDENCE AND REGULATORY DECISIONS	658
A. 	Critical Factors in Determining the Strength of Evidence	659
Regulatory decisions related to drug safety are generally taken after considering the totality of	660
available evidence, which may include meta-analytic findings, as well as other factors such as	661
risk-benefit considerations, availability of alternative treatments, biological and clinical	662
plausibility of the drug-risk relationship, and available regulatory options. The strength of	663
evidence provided by the meta-analysis may influence a regulatory decision by FDA. The factors	664
discussed above that FDA generally considers in determining the strength of evidence with	665
respect to a safety-related regulatory decision can be summarized as follows:	666
• 	Quality and appropriateness of the individual trials for the meta-analysis objectives	667
 	Quality and completeness of safety outcome ascertainment	668
 	Appropriateness of studied populations and exposure and follow-up periods	669
 	Protocol adherence in the individual trials (e.g., compliance with investigational	670
treatment, loss to follow-up, etc.)	671
 	Availability and quality of subject-level data	672
• 	Prespecification and adequacy of documentation	673
 	Prespecification and documentation of objectives, available knowledge, trial	674
inclusion criteria, and choice of comparators, outcomes, statistical methods, and	675
subgroups	676
 	Documentation that trial outcomes were not used as part of the trial selection	677
criteria	678
 	Documentation of meta-analysis results including summaries of trials, subjects,	679
outcomes, effect estimates, measures of uncertainty, and sensitivity analyses	680
• 	Appropriateness of statistical methods	681
 	Approach used for combining trials	682
 	Methods to handle sparse data or rare events	683
 	Methods to address heterogeneity	684
 	Sensitivity analyses	685
 	Validity of uncertainty estimates (e.g., confidence intervals or credible intervals)	686
Although not previously discussed, the magnitude of the estimated risk and associated measures	687
of uncertainty are also important. A large estimated risk will generally be more convincing than a	688
small to moderate one, because it will provide more assurance that an effect is real even in the	689
presence of potential biases. Similarly, smaller p-values or narrower confidence intervals, both	690
measures of uncertainty, provide additional assurance on the findings of the meta-analysis. For	691

Contains Nonbinding Recommendations
Draft — Not for Implementation
19
safety meta-analyses, however, there is potential for bias from both known (e.g., selection of	692
trials based on their outcomes) and unknown (biases that cannot be identified from the data used	693
to conduct the meta-analysis) sources. Given this difficulty, standard measures of uncertainty,	694
such as significance levels, should be interpreted with caution.	695
One approach to account for the many potential sources of bias and error in a meta-analysis is to	696
replace the commonly used test size or alpha level for hypothesis testing, α = 0.05, with an	697
arbitrarily lower value (e.g., 0.01 or 0.001) in order for the results to be considered convincing.	698
The choice of a lower value would reflect the recommendation to compensate for known and	699
unknown sources of potential bias as well as to minimize the impact of multiplicity resulting	700
from multiple comparisons. Such an approach would be important if the meta-analysis is the	701
only basis for decision-making, as it will explicitly reflect the higher degree of uncertainty that	702
exists for meta-analysis results. At the same time, there are often other sources of safety	703
information so that the significance level for the meta-analysis is only one of many factors taken	704
into consideration. Consequently, no single test size (alpha level) and no single confidence level	705
can be recommended for deciding the level of statistical significance for results from a safety	706
meta-analysis to be relied upon. The potential for harm may be so serious that marginally	707
significant findings could prompt regulatory consideration. In this setting, however, the sources	708
of bias and error related to the meta-analysis should be identified and accounted for wherever	709
possible.	710
In addition to the magnitude of the observed effect and the level of uncertainty, the robustness of	711
the risk estimate to appropriate sensitivity analyses can also support the strength of the meta-	712
analysis findings. The importance of sensitivity analyses is described in section V, as their results	713
play an important role in determining the strength of evidence. Risk estimates that are reasonably	714
robust to the inclusion or exclusion of particular studies, or to changes in the statistical analysis	715
methods used and assumptions required for appropriate use of those methods, will carry a greater	716
weight of evidence than estimates that vary widely with such changes.	717
Similarly, risk estimates that are consistent across trials will also carry greater weight. In section	718
IV, the use of forest plots or other graphical display of the study-specific risk estimates and their	719
confidence intervals is advocated as a descriptive assessment of study-to-study heterogeneity.	720
Absent any known cross-study differences, a high degree of similarity among study-specific	721
results will strengthen the evidence provided by the meta-analytic summary risk estimate.	722
Conversely, a large amount of variability among studies would make a marginal risk estimate (in	723
terms of lack of statistical significance or small in magnitude) less persuasive.	724
B. 	Hierarchy of Evidence for Decision-Making	725
The factors described above for evaluating the strength of meta-analytic findings can be used to	726
define a hierarchy of evidence against which meta-analyses conducted or reviewed for regulatory	727
purposes should be evaluated.	728
• 	A top tier meta-analysis is one that is prospectively planned prior to the conduct of the	729
trials to be included, and where the component trials are designed with the meta-analysis	730
objectives in mind. The trials have well-ascertained outcomes and exposure periods, and	731

Contains Nonbinding Recommendations
Draft — Not for Implementation
20
subject-level data are available for analysis. This level represents a gold standard not	732
often realized in practice but useful as a benchmark in evaluating the quality of a meta-	733
analysis.	734
• 	The next level down is a prospectively planned meta-analysis based on existing trials that	735
were designed for other purposes but for which the quality of the data and the	736
ascertainment of outcomes and exposure are adequate to support the planned analysis.	737
Further, all meta-analytic study plans and trial inclusion decisions were made without	738
knowledge of the study outcomes for the safety events of interest.	739
• 	The lowest tier, representing the least useful evidence for regulatory decision-making,	740
corresponds to meta-analyses for which prospective planning did not occur, or is in	741
doubt, study outcomes and trial inclusion decisions were made with outcome data in	742
hand, and one or more of the important quality factors is in question, e.g., lack of rigor in	743
outcome ascertainment, lack of subject-level data for use in determining exposure, use of	744
inappropriate statistical methods such as simple pooling of trial data, or other issues.	745
Between the bottom and top tiers lies a broad range of meta-analyses for which an evaluation of	746
the strength of evidence provided should include careful consideration of the important factors	747
delineated in the previous subsections.	748
The level of evidence from a meta-analysis that is based solely on study level summary data,	749
either prospective or retrospective, is generally considered to be lower than one for which	750
subject-level data are available, as the party conducting the meta-analysis has little ability to	751
judge the quality or completeness of the data or the appropriateness of the analysis methods used.	752
On the other hand, if the outcome is relatively judgment-free and well-ascertained (e.g. mortality	753
or perhaps stroke rate), these meta-analyses may still play a role in regulatory decisions. A study-	754
level meta-analysis could be used as a first step to determine whether a more resource intensive	755
subject-level meta-analysis is needed, perhaps based on the same studies. A hybrid would be a	756
combination of studies for which subject level data are available for a subset; the mix would	757
determine where in the hierarchy such a meta-analysis should be placed. The recommendations	758
laid out in this guidance for producing high-quality meta-analyses apply regardless of the level	759
(subject- or trial-level) of analysis involved.	760
There are two categories of meta-analyses considered particularly problematic for the regulatory	761
framework and worth mentioning here. The first includes meta-analyses reported in the literature	762
with no prior publication or credible record of a protocol to guide the selection of studies or	763
prespecification of study objectives and analysis strategy. This type of meta-analysis is likely	764
insufficient for regulatory purposes, for the reasons outlined in section IV. Even if the studies	765
included in the meta-analysis represent a reasonable subset of those available (as opposed to only	766
published studies), without documentation of a prespecified plan for deciding which to include	767
and identifying outcomes of interest, it is usually not possible to determine what was known at	768
the time the studies were selected, what analysis methods were chosen, or how many different	769
analyses were conducted, in what sequence, and for which study populations or subgroups.	770
Evidence from such an analysis would generally be considered too weak to support regulatory	771
decision-making without further confirmation of the findings.	772

Contains Nonbinding Recommendations
Draft — Not for Implementation
21
The second category includes meta-analyses that are based solely on safety results appearing in	773
the literature. Limiting the meta-analysis to studies whose results appeared in publications about	774
the exposure-risk relationship can introduce publication bias. This well-known phenomenon	775
arises from the concern that studies failing to find a significant association between drug use and	776
risk are not published at the same frequency as studies that show an association, and even among	777
those published, bias may occur due to a failure to include certain safety outcomes in the	778
publication and failure to include studies that did not show the outcome sought (Chalmers, Levin	779
et al. 1987). Further, the information contained in the publication for each study may be lacking	780
in detail, and without access to subject-level data, it may not be possible to rule out bias or severe	781
heterogeneity in the results. For example, even if the results for the safety outcome of interest are	782
reported for each trial, the details of how events were defined, measured, or adjudicated in the	783
trials may not be clear, and it may not be possible to determine if the safety events of interest	784
occurred on or off drug. Subject-level data are typically not available in publications of	785
completed trials, limiting the ability to resolve these issues.	786
In summary, a number of important factors should be involved in determining the credibility of	787
evidence from a particular meta-analysis. These factors range from the knowledge about and	788
documentation of eligible studies, both published and unpublished; the quality and relevance of	789
the studies selected as well as the process and timing of selection; and the validity of the	790
statistical analysis that supports the inferential conclusions and the strength of the findings,	791
evaluated against sources of potential or real bias. Whether or not the findings of a meta-analysis	792
influence regulatory decision-making will generally depend, in part, on the strength of evidence	793
provided by the findings, as determined by a careful evaluation of the important factors	794
described in this guidance.	795
VII. 	EXAMPLES	796
A. 	Example 1: Antidepressant Use and Suicidal Events in Adults	797
This example illustrates the use of a meta-analysis to evaluate risks associated with a class of	798
drugs and represents a prospectively planned meta-analysis of retrospective data, falling into the	799
middle tier of the hierarchy of evidence discussed in section VI.B. The research hypotheses,	800
study inclusion criteria, outcome measures, and statistical analysis plan were all specified prior	801
to the conduct of the meta-analysis. Outcomes were uniformly adjudicated across studies,	802
pooling of study data was accomplished with stratification, and subject-level data were available	803
to explore subgroups as well as trends in risks across time. The interpretation of the findings,	804
which resulted in a boxed warning for labeling of drugs in the class, reflects appropriate	805
consideration given to the level of statistical significance, and to the consistency of findings.	806
In 2004, FDA completed a meta-analysis of studies of pediatric patients that showed an	807
association of antidepressant drugs and suicidal behavior and ideation (Hammad, Laughren et al.	808
(2006). Unsolicited information provided by drug sponsors to FDA and published articles	809
motivated this meta-analysis. Based on the meta-analysis findings and deliberations from a	810

Contains Nonbinding Recommendations
Draft — Not for Implementation
22
meeting in 2004 of FDA Advisory Committees (69 FR 471574) and further consideration by	811
FDA, a boxed warning was added to the labeling of all antidepressants concerning use in	812
pediatric patients. Other FDA Advisory Committees later asked FDA to explore the association	813
in adult patients (71 FR 665455.)	814
For this purpose, FDA planned and conducted a meta-analysis of randomized clinical trials of	815
antidepressants. FDA is uniquely positioned to address this research question, because of the	816
Agency’s knowledge of marketed products in the drug class in question. The meta-analysis had	817
several important features that supported its quality and utility for regulatory actions: (1)	818
hypotheses generated from previous and independent evidence provided the meta-analysis	819
objectives; (2) the meta-analysis was based on well-defined inclusion criteria and a complete set	820
of the trials that met the inclusion criteria; (3) the meta-analysis employed rigorous and	821
consistent outcome definitions across trials and patients; (4) the meta-analysis was based on a	822
prespecified plan; and subject-level data was available.	823
FDA requested from all manufacturers of antidepressants all available subject-level data from	824
randomized placebo-controlled trials of antidepressants. Basing the meta-analysis on data	825
available to sponsors, while not inclusive of all potentially available data, has some important	826
advantages. Because of regulatory requirements, trials from drug manufacturers typically contain	827
detailed subject-level data including medical history, baseline characteristics, subject	828
dispositions, patient outcomes, and adverse events. Focusing on the relatively small group of	829
drug manufacturers (nine) allowed for the timely acquisition of the large amounts of pertinent	830
data. Overall, the FDA obtained subject-level data considered usable for 372 trials.	831
The meta-analysis was prospectively planned but was based on previously collected data.	832
Because the specific outcomes of interest were not systematically collected and adjudicated	833
during the conduct of the trials, FDA provided specific instructions to the individual sponsors to	834
conduct a retrospective identification and adjudication of potential suicidal behavior and ideation	835
events from the subject-level data. The outcome definition required that the suicidal behavior and	836
ideation events occurred on randomized treatment or within one day of stopping the randomized	837
treatment. Based on adverse event reporting, potential events were identified with a specified	838
algorithm. Based on blinded narratives of the events, qualified personnel classified the events	839
into specific outcomes including: completed suicide, attempted suicide, preparatory actions	840
toward imminent suicidal behaviors, and suicidal ideation based on the Columbia Classification	841
Algorithm for Suicide Assessment (Posner, Oquendo et al. 2007) . The overall process resulted in	842
outcome measures that were consistently and rigorously defined across trials and subjects.	843
The meta-analysis employed a prespecified plan that included the trial inclusion criteria,	844
hypotheses, outcome definitions, analysis methods, sensitivity analyses, and subgroups. The	845
primary analysis method incorporated stratification by trial and accounted for the sparse nature	846
of the outcome events by using exact statistical methods for hypothesis testing. Sensitivity	847
4 Briefing package:
http://web.archive.org/web/20040911055410/https://www.fda.gov/ohrms/dockets/ac/04/briefing/2004-4065b1.htm
5 Briefing package: https://wayback.archive-
it.org/7993/20170405070114/https://www.fda.gov/ohrms/dockets/ac/06/briefing/2006-4272b1-01-FDA.pdf

Contains Nonbinding Recommendations
Draft — Not for Implementation
23
analyses were planned to examine the possibility and consequences of the following: differential	848
exposure time between the randomized drug arms; heterogeneity of the effect measure across the	849
trials; and trials with no events. The subject-level data allowed for the examination of important	850
subgroups, including subject age, and for the examination of changing risk over time.	851
FDA presented the meta-analysis findings to a 2006 meeting of FDA Advisory Committees	852
(FDA 2006) and sought advice on the interpretation and possible regulatory actions based on the	853
findings of the meta-analysis. The meta-analysis found that the overall association of	854
antidepressant drugs and suicidal behavior and ideation was not statistically significant in adult	855
subjects, in contrast to the FDA meta-analysis of pediatric subjects. However, the association	856
was nearly statistically significant for young adults, and a clear pattern emerged with respect to	857
patient age (see Figure 1). The result from the pediatric meta-analysis supported this trend.	858
Based on the totality of the evidence, including results from the meta-analysis, FDA requested	859
that manufacturers update the boxed warning on all antidepressants to include the risk of suicidal	860
behavior and ideation associated with antidepressants for young adult patients in addition to	861
pediatric patients. The warning states that the effect was not seen in adults over the age of 24,	862
and for adults aged 65 and older, there was a reduction in risk. It should be appreciated that the	863
clear pattern observed with respect to age and not just statistical significance led to the warning.	864
865
Figure 1: FDA Meta-Analysis of Antidepressants and Suicidal Behavior and Ideation.	866
Note: Pediatric results from previous FDA meta-analysis of pediatric patients (Hammad,	867
Laughren et al. 2006).	868
Odds Ratio
0.1 	0.3 	1 	3.2 	10
Adult Overall
65 and Up
31 to 64
25 to 30
18 to 24
Pediatric Data
0.84 (0.69, 1.02) [248/39729 196/27164]
0.39 (0.18, 0.78) [12/3227 24/2397]
0.77 (0.60, 1.00) [147/27086 124/18354]
1.00 (0.60, 1.69) [41/5558 27/3772]
1.55 (0.91, 2.70) [47/3810 21/2604]
2.22 (1.40, 3.60) [61/2414 26/2073]
Age Class 	OR (95% CI) [Sample Sizes]*
Suicidal Behavior and Ideation
Psychiatric Indications
*

Contains Nonbinding Recommendations
Draft — Not for Implementation
24
B. 	Example 2: Tiotropium and Cardiovascular Events	869
This example illustrates FDA’s consideration of the cardiovascular safety of the drug tiotropium	870
and shows how the relative strengths of a well-designed, large, long-term trial and a meta-	871
analysis based on published literature as well as other trial-level information (Michele, Pinheiro	872
et al. 2010) were factored into a regulatory decision. Tiotropium bromide inhalation powder is a	873
long-acting anticholinergic approved for use in treating bronchospasm associated with chronic	874
obstructive pulmonary disease (COPD) and for reducing COPD exacerbations. The potential	875
association of the drug with cardiovascular events was first reported to FDA based on an analysis	876
of adverse events from 29 placebo-controlled trials conducted by the drug manufacturer. In	877
particular, the simple pooled analysis of these studies showed that the drug had an excess number	878
of strokes associated with its use. The pooled analysis was intended to identify potential signals	879
for further evaluation and examine a range of adverse events. As is typical for such analyses,	880
findings from the pooled analysis were not adjusted for multiplicity associated with examining	881
multiple endpoints.	882
Because of the severity of the clinical outcomes, FDA issued a communication informing the	883
public of the potential safety signal and FDA’s efforts to investigate the findings. The	884
communication noted that data from a large, four-year study called UPLIFT (Tashkin, Celli et al.	885
2008) would soon be available and would provide additional long-term safety data on the drug.	886
Following the FDA communication, an article appeared on a meta-analysis of 17 randomized	887
trials reporting a statistically significant increase in a cardiovascular composite outcome	888
(cardiovascular death, myocardial infarction, and stroke) associated with inhaled anticholinergics	889
(consisting of tiotropium and ipratropium) (Singh, Loke et al. 2008). At the same time, the	890
results from UPLIFT had become available, and the initial review did not support a finding that	891
tiotropium was associated with an increased risk of stroke, heart attack, or cardiovascular death.	892
A comparison between the pooled analysis of 29 trials conducted by the manufacturer and the	893
UPLIFT trial highlights some important differences between the two sources of safety	894
information. Although the pooled analysis contained more than twice as many subjects as	895
UPLIFT (13,544 versus 5,992), the study duration of UPLIFT (4 years) was substantially longer	896
than the durations of the trials in the pooled analysis (1 – 12 months). Consequently, UPLIFT	897
provided more than twice as many person-years of follow-up (17,721 person-years) than the	898
pooled analysis (7,636 person-years). Additionally, UPLIFT prospectively collected data on	899
death and adjudicated cause of death for all subjects, including subjects who withdrew from the	900
study.	901
In 2009, FDA convened an advisory committee meeting to discuss the results of UPLIFT and the	902
published meta-analysis. The advisory committee concluded (11 votes to 1) that the results of	903
UPLIFT adequately addressed the cardiovascular safety concerns that had been raised for	904
tiotropium based on the initial pooling of 29 trials by the manufacturer and the published meta-	905
analysis. The committee noted methodological concerns with the published meta-analysis,	906
including lack of accounting for differential withdrawal rates between treatment groups and the	907
potential for publication bias due to including only studies reporting an increase in	908
cardiovascular events with use of tiotropium in the meta-analysis. The committee also noted	909
concerns about the heterogeneity of trial designs in the review, including differences in study	910

Contains Nonbinding Recommendations
Draft — Not for Implementation
25
drug, comparator drug, trial duration, and studied population. Based on the strength of the	911
UPLIFT study findings and the methodological concerns of the published meta-analysis, FDA	912
concluded that the available data did not support an association between the drug and adverse	913
cardiovascular events.	914
The tiotropium example shows that a meta-analysis based on trial-level summaries may not	915
agree with a large trial that is well designed specifically with a safety outcome as a primary	916
objective. However, the Agency’s position on the safety and effectiveness of a drug is based on	917
the best information available at the time. In the tiotropium example, FDA issued a series of	918
public communications to apprise the public of the latest safety information available and FDA’s	919
intended course of action. The example shows FDA’s intention to carefully evaluate potential	920
safety risks while balancing the need to not unnecessarily discourage or restrict the use of safe	921
and effective drugs. The example also shows FDA’s intention to act in a transparent manner, to	922
the extent possible, based on the available data to ensure the safe use of drugs.	923
VIII. 	REFERENCES	924
Bender, R., C. Bunce, M. Clarke, S. Gates, S. Lange, N. L. Pace and K. Thorlund (2008).	925
"Attention should be given to multiplicity issues in systematic reviews." J Clin Epidemiol 61(9):	926
857-865.	927
Borenstein, M., L. V. Hedges, J. P. Higgins and H. R. Rothstein (2010). "A basic introduction to	928
fixed-effect and random-effects models for meta-analysis." Res Synth Methods 1(2): 97-111.	929
Bradburn, M. J., J. J. Deeks, J. A. Berlin and A. Russell Localio (2007). "Much ado about	930
nothing: a comparison of the performance of meta-analytical methods with rare events." Stat	931
Med 26(1): 53-77.	932
Chalmers, T. C., H. Levin, H. S. Sacks, D. Reitman, J. Berrier and R. Nagalingam (1987).	933
"Meta-analysis of clinical trials as a scientific discipline. I: Control of bias and comparison with	934
large co-operative trials." Stat Med 6(3): 315-328.	935
Chien, P. F., K. S. Khan and D. Siassakos (2012). "Registration of systematic reviews:	936
PROSPERO." BJOG 119(8): 903-905.	937
Chuang-Stein, C. and M. Beltangady (2011). "Reporting cumulative proportion of subjects with	938
an adverse event based on data from multiple studies." Pharm Stat 10(1): 3-7.	939
Greenland, S. and J. M. Robins (1985). "Estimation of a common effect parameter from sparse	940
follow-up data." Biometrics 41(1): 55-68.	941
Hammad, T. A., T. Laughren and J. Racoosin (2006). "Suicidality in pediatric patients treated	942
with antidepressant drugs." Arch Gen Psychiatry 63(3): 332-339.	943
Higgins, J. P. T., Green, S., (editors). (2011). Cochrane Handbook for Systematic Reviews of	944
Interventions, Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011.	945
Available from www.handbook.cochrane.org.	946

Contains Nonbinding Recommendations
Draft — Not for Implementation
26
Michele, T. M., S. Pinheiro and S. Iyasu (2010). "The safety of tiotropium--the FDA's	947
conclusions." N Engl J Med 363(12): 1097-1099.	948
Moher, D., A. Liberati, J. Tetzlaff, D. G. Altman and P. Group (2009). "Preferred reporting items	949
for systematic reviews and meta-analyses: the PRISMA statement." PLoS Med 6(7): e1000097.	950
Muthukumarana, S. and R. C. Tiwari (2016). "Meta-analysis using Dirichlet process." Stat	951
Methods Med Res 25(1): 352-365.	952
Ohlssen, D., K. L. Price, H. A. Xia, H. Hong, J. Kerman, H. Fu, G. Quartey, C. R. Heilmann, H.	953
Ma and B. P. Carlin (2014). "Guidance on the implementation and reporting of a drug safety	954
Bayesian network meta-analysis." Pharm Stat 13(1): 55-70.	955
Permutt, T. (2003). "Probability models and computational models for ANOVA in multicenter	956
clinical trials." J Biopharm Stat 13(3): 495-505.	957
Posner, K., M. A. Oquendo, M. Gould, B. Stanley and M. Davies (2007). "Columbia	958
Classification Algorithm of Suicide Assessment (C-CASA): classification of suicidal events in	959
the FDA's pediatric suicidal risk analysis of antidepressants." Am J Psychiatry 164(7): 1035-	960
1043.	961
Senn, S. (2000). "The many modes of meta." Drug Information Journal 34(2): 535-549.	962
Singh, S., Y. K. Loke and C. D. Furberg (2008). "Inhaled anticholinergics and risk of major	963
adverse cardiovascular events in patients with chronic obstructive pulmonary disease: a	964
systematic review and meta-analysis." JAMA 300(12): 1439-1450.	965
Spiegelhalter, D. J., K. R. Abrams and J. P. Myles (2004). Bayesian approaches to clinical trials	966
and health-care evaluation. Chichester, Wiley.	967
Sutton, A. J. and K. R. Abrams (2001). "Bayesian methods in meta-analysis and evidence	968
synthesis." Stat Methods Med Res 10(4): 277-303.	969
Tashkin, D. P., B. Celli, S. Senn, D. Burkhart, S. Kesten, S. Menjoge, M. Decramer and U. S.	970
Investigators (2008). "A 4-year trial of tiotropium in chronic obstructive pulmonary disease." N	971
Engl J Med 359(15): 1543-1554.	972
Whitehead, A. (1997). "A prospectively planned cumulative meta-analysis applied to a series of	973
concurrent clinical trials." Stat Med 16(24): 2901-2913.	974