FTC Staff Comment Before the Postal Commission Concerning Documentation of Statistical Analyses

COMMISSION AUTHORIZED
\. ~-{\Lt s\~	BEFORE THE
UNITED STATES OF AMERICA POSTAL RATE COMMISSION
WASHINGTON, D.C. 20268-001
Documentation of Statistical Analyses, 1989 	-Docket No. RM89-3
COMMENTS OF
THE STAFF OF THE BUREAU OF ECONOMICS OF THE
FEDERAL TRADE COMMISSIONl
(submitted April 24, 1989)
I. 	INTRODUCTION
The staff of the Bureau of Economics of the Federal Trade Commission
(FTC) 	appreciate 	the 	opportunity 	to 	submit 	these 	comments 	to 	the 	Postal
Rate Commission (PRe) concerning its proposals to improve documentation
of 	statistical 	analyses 	that 	underlie 	its 	rate 	and 	service 	decisions. 2 	The
PRC's 	proposals 	would 	require identification of assumptions, 	data 	sources,
and techniques used in statistical analyses, and under specific circumstances
would 	make 	original 	data 	sets 	and 	updated 	data 	available 	to 	interested
parties. 	Our 	experience 	suggests 	that 	the 	PRC's 	proposals 	may 	offer
substantial 	benefits 	in return 	for 	relatively 	modest costs. 	We discuss 	this
below and we also suggest additions and modifications to the proposed rules
that may be desirable.
1 These comments represent 	the views of the staff of 	the 	Bureau of
Economics of the Federal Trade Commission. 	They are not necessarily the
views of the Commission or any individual Commissioner. 	Questions about
these 	comments 	may 	be 	addressed 	to 	John 	C. 	Hilke, 	Federal 	Trade
Commission, Bureau of Economics, 6th Street and Pennsylvania Avenue, N.W.,
Washington, D.C. 20580, telephone: (202) 326-3483.
2 The proposed rules and invitation to comment were published in the
Federal Register. Vol. 54, No. 44, on March 8, 1989 at pages 9848-9852.
1
.
.~,,,~~.t;.j

II. 	FEDERAL TRADE COMMISSION STAFF EXPERTISE
The 	FTC 	is 	an 	independent 	regulatory 	agency 	responsible 	for
maintaining competition and safeguarding the interests of consumers. 3 	The
staff 	of 	the 	FTC, 	upon 	request 	by 	federal, 	state, 	and 	local 	government
. bodies, 	often 	ana.lyze 	regulatory 	or 	legislative 	proposals 	that 	may 	affect
competition or the efficiency of the economy. 	In the course of this work,
as well as i~n antitrust and tonsumer protection research and litigation, the
staff 	frequently 	pres~t econometric 	and 	other 	statistical 	evidence 	and
evaluate similar evidence presented by other parties.
The FTC staff have commented previously on various issues before the
PRC, including: (1) use of a single set of rate hearings to establish a series
of 	United 	States 	Postal 	Service 	(USPS) 	rate 	changes;· 	(2) 	elaboration 	of
competition 	issues 	inherent 	in 	proposed 	rate 	and 	classification 	changes
related to electronic computer originated mail (ECOM);5 (3) d"rawbacks to a
proposed 	modification 	of 	the 	test 	period 	for 	cost 	recovery 	in 	ECOM;6 (4)
advantages of setting ECOM rates to cover full costs;7 (5) costs and benefits
3 15 U.S.C. Section 41 tl ~
• PRC Docket No. MR82-3, filed November 4, 1982.
5 PRC Docket No. R83-I, filed June I, 1983.
6 PRC Docket No. R83-1, filed June 16, 1983.
7 PRC Docket No. R84-1, filed December 23, 1983.
2

of 	current 	preferred 	mail 	rates;8 	(6) 	expedited 	procedures 	in 	reviewing
proposed rate changes for Express Mail;9 and, most recently, (7) a complaint
urging 	a 	study 	of 	the 	possibility 	of exempting addressed 	third 	class 	mail
from 	the 	private 	express 	statutes. 10 	Several 	of 	our 	commen ts 	have
referenced 	the PRC's difficulties 	in reaching informed 	decisions on 	postal
rates 	and 	services 	because 	of 	the 	lack 	of 	sufficient 	accessible 	data 	on
demand for and costs of different postal services.
III. 	THE PRESENTATION OF STATISTICAL ANALYSES
A. Introduction
In order for statistical analyses effectively and efficiently to 	provide
objective 	information 	for 	the 	PRC 	to 	use 	in 	making 	rate 	and 	service
decisions, 	participating 	statistical 	analysts 	must 	clearly 	explain 	what 	they
did and why, and the sources and characteristics of the data on which they
relied.ll 	Further, statistical 	results 	must 	be 	replicable, 	if 	the 	PRC 	is 	to
place substantial confidence in them. 12 	Documentation and replication are
8 PRC Docket No. SS86-1, filed April 20, 1986.
9 PRC Docket No. RM88-2, filed October 14, 1988.
10 PRC Docket No. C89-1, submitted February 28, 1989.
11 See McAleer, M., A. Pagan, and P. Volker, "What Will Take the Con
Out of Econometrics?" American Economic Review 75:3 (June 1985), pp. 293-
307.
12 Statistical 	results 	are 	replicable 	if 	other 	analysts 	obtain 	the 	same
results 	when 	they 	employ 	the 	same 	data 	and 	procedures. 	Dewald, 	W., 	J.
3
" 	,-.

central 	elements 	in 	the 	application 	of 	the scientific 	method, 	and 	no 	less
should be expected from any statistical analyses presented to the PRC that
purport to 	be objective. 	Consequently. we concur with -the PRC's proposal
that statistical analyses on which the parties urge the PRC to place serious
weight adhere to these criteria. 	At the same time, the PRC recognizes that
application 	of 	unnecessary 	restrictions 	and 	arbitrary 	requirements 	could
discourage 	useful 	analyses and cause a 	net decrease in 	the availability of
useful information to the PRC.lS 	We believe that the PRC's proposals make
appropriate allowance for this concern.
• 	B. Failure of Current Rules
The 	USPS 	and 	other 	parties 	who 	present 	statistical 	analyses 	to 	the
PRC often do not, in the PRC's view, provide the analyses in a ~orm that
can be readily understood. 'reviewed, or challenged. 14 	Current submissions
sometimes fail to define the variables used. identify sources of data. explain
the 	theoretical 	concepts 	employed, 	provide 	citations 	for 	statistical
Thursby, and R. Anderson, "Replication in Empirical Economics: 'The Journal
of Money, Credit, and 	Banking' Project," 	American 	Economic 	Review 	76:4
(September 1986), pp. 587-603.
13 For 	example, 	the 	PRC 	proposes 	that 	requests 	to 	the 	USPS 	from
interested parties for additional statistical analyses be restricted to plausible
alternative assumptions, rather than to all potential alternatives.
14 The 	implications 	of 	a 	statistical 	analysis 	may 	be 	unclear 	either
because the analyst has failed 	to provide clear descriptions and references
about 	what 	was 	done 	or 	because 	the 	results 	are 	inconclusive. 	There 	is
nothing that procedural changes can do about the latter and therefore the
PRe's proposals will have no effect in such instances. 	In the former case,
procedural 	changes, such as 	those 	proposed 	by 	the 	PRC, can 	increase 	the
amount of useful information obtained from each statistical test.
4
. 	.

techniques, 	indicate 	alternative 	assumptions 	considered, 	report 	relevant
statistics, or 	justify the exclusion 	of 	potential explanatory 	variables. 	The
PRC 	states 	that 	it 	is 	particularly 	concerned 	about 	statistical 	analyses 	of
demand 	for 	different 	classes 	of 	mail.15 	These 	analyses 	have 	become
essential elements in 	many of the PRC's rate and service decisions. 	As a
result of poor documentation, the PRC has sometimes been forced, even in
some of its most important decisions, to rely on statistical analyses on which
it 	cannot 	place 	great 	confidence. 16 	With 	better 	documentation 	and
descriptions, independent review and evaluation of statistical analyses by the
PRC's staff and other parties become possible, and this will help insure that
the PRC's decisions are based on sound and objective analyses.
The 	FTC 	staff's 	experience' in 	commenting 	on 	PRC 	matters 	,is
consistent 	with 	the 	PRC's 	observations. 	Poor 	documentation 	of 	studies
submitted to the PRC results in a lack of sufficient information to replicate
results and to test 	results for robustness.l1 	For example, in our comments
on 	procedures 	for 	adjusting 	Express 	Mail 	rates 18 	we 	observed 	that 	"one
rationale for the USPS's involvement in expedited mail could be economies of
15 See the Federal Register notice at p. 9849.
16 See the Federal Register notice at pp. 9848-9849. 	It reads in 	part:
" 	the 	principal obstacle 	to effective review 	by 	the Commission 	has 	been
the inadequate support offered for the numerous subjective judgments that
the Postal Service has employed in developing the econometric models upon
which its forecast is based. 	In the past the Commission often had been put
in 	the 	position 	of 	having 	to 	'take 	or 	leave' 	the 	Postal 	Service's 	volume
forecast, and some unsupported judgments that underlie it, because there has
been no viable alternative to the Postal Service's econometric demand models."
17 Results are 	robust 	if slight changes in 	the 	data 	sample, 	variables,
and/or statistical techniques do not substantially alter the results.
18 P~C Docket No. RM88-2, filed October 14, 1986, p. 8.
S

scope 	between 	Express 	Mail 	and 	statutorily 	protected 	classes 	of 	mail
service."19 	However, the empirical evidence 	presented 	to the PRC on 	this
important 	question 	was 	insufficient 	to 	make 	a 	detenilination.2o 	If 	the
procedural changes proposed by the PRC had been in force, they might have
helped 	the 	PRC resolve 	the 	question 	by pinpointing data 	deficiencies 	and
allowing interested parties to respecify the models so that the results could
more adequately address the question of economies of scope (and scale).
Similarly, when commenting on a proposed study of an exemption from
the 	private express statutes 	for 	addressed 	third 	class 	mail, 	the' FTC staff
concluded 	that there was insufficient statistical information 	in the 	USPS's
submissions to judge what effects such an exemption might have. 	If better
documented demand analyses and tests for robustness, as now contemplated
in 	the 	PRC's 	proposals, 	had 	been 	available 	at 	the 	time, 	they 	could 	have
provided very useful information on this issue,21
Finally, 	in 	our 	comment 	on 	ECOM 	demand 	forecasts, 	we 	expressed
.
concern that the lack of well-documented statistical information on demand
might make it difficult for the PRC to determine whether the USPS's demand
19 An "economy of scope" exists when the cost of providing a good or
service 	is 	lower 	if 	it 	is 	produced 	in 	conjunction 	with 	another 	good 	or
service than if it is produced alone. 	By comparison, an "economy of scale"
exists 	when the 	per item cost of providing a good or service decreases as
the 	quantity 	of 	production 	increases. 	Economies 	of scope 	and 	scale 	may
exist simultaneously.
20 PRC Docket No. C89-l, filed February 28, 1989, at pages 3-4.
21 PRC Docket No. C89-l, filed February 28, 1989. 	Data from earlier
rate 	making cases could 	be 	used 	if 	made 	available 	in 	a 	well-documented
fashion.
6

estimates for ECOM were unduly optimistic.22 	If the PRC's proposed rules
had 	been 	in 	effect 	at 	that 	time, 	interested 	parties 	could 	have 	requested
econometric 	forecasts 	from 	the 	USPS, 	or 	produced 	such 	forecasts
themselves, using alternative models and data that could have been compared
to the USPS's projections.
C. Potential Benefits
The 	potential 	benefits 	to 	consumers 	from 	improved 	statistical
information 	and 	the 	resulting 	improvements 	in 	decision-making 	may 	be
substantial, 	if 	for 	no 	other 	reason 	than 	that 	the 	volume 	of 	affected
commerce 	is 	in 	excess 	of 	$30 	billion 	annually 	and 	there 	appears 	to 	be
considerable 	room 	(or 	improvement 	in 	the 	analY,ses 	now 	submitted.23 	We
have 	found 	that 	well-documented 	statistical 	work 	can 	provide 	-useful
information in 	the FTC's antitrust and consumer protection work on those
occasions when suitable data are available, and we believe that the PRe may
experience a similar increase in useful information, if it adopts the proposed
(or similar) rules. 24
22 PRC Docket No.R83-1, filed June 16, 1983, pp. 9-10.
23 Revenues of the 	U.S. Postal Service alone exceed $30 billion. 	See
"Summary Financial and Operating Statements," various dates, Department of
the 	Controller, 	U.S. 	Postal 	Service. 	To 	the 	extent 	that 	rate 	and 	service
decisions of the PRC impact on private firms providing similar services, the
volume of affected commerce is greater than the revenues of the USPS.
The 	PRC's views on 	the shortcomings of previous statistical analyses
are discussed in the Federal Register notice at pages 9848-9850.
24 See for example, Barton, D., and R. Sherman, "The Price and Profit
Effects of Horizontal Merger: A Case Study," Journal of Industrial Economics
33:2 	(December 	1984), 	pp. 	165-177, 	and 	Cox, 	S., 	"Advertising 	Restrictions
Among Professionals: Bates v .. State Bar of Arizona," in Kwoka, J., and L.
7
I
'1I

Regardless 	of 	the 	reasons 	for 	the 	current shortcomings 	of 	statistical
analyses submitted 	to the 	PRC, 	the 	PRC's proposals are 	likely 	to 	produce
substantial 	improvements 	for 	three 	reasons. 	First, 	the 	procedures 	will
dictate minimum standards of presentation that will make the details of the
analyses 	more 	transparent 	and 	accessible 	to 	the 	PRC. 	When 	analyses 	are
more understandable because they are better explained and documented, the
insights 	from 	them 	are 	more 	likely 	to 	be 	incorporated 	into 	decisions.
Second, by permitting other parties to propose alternative specifications (or
to 	obtain 	the 	data 	directly), 	the 	PRC's 	proposals 	promote 	competition 	in
providing the "best model" to incorporate in rate and service decisions. 	In
this way, the PRC reduces its dependence on submissions from any single
party 	and 	removes 	a 	possible 	source 	of 	bias 	in 	its 	decisions. 	Third, 	by
establishing an ongoing data collection process for postal data, the proposed
regulations 	should 	I)elp 	all 	parties 	to 	update 	statistical analyses 	of 	postal
demand conditions in a timely manner. 	Further, regular data updates may
alert 	interested 	parties 	to 	changes 	in 	circumstances 	that 	may 	warrant 	the
attention 	of 	the 	PRC 	or 	the 	USPS. 	Increasing 	the 	understandability 	of
analyses, removing bias, and ensuring up-to-date data all have the potential
to improve the quantity of useful information available to the PRC and the
quality of the PRC's decisions based on it.
White, Editors, The Antitrust Revolution. Glenview, Ill.: Scott, Foresman and
Company, 	1989, 	pp. 	134-159. 	Data of sufficient quantity 	and 	quality 	for
econometric work are not available, however, in many cases.
We find that obtaining the original data is very helpful in evaluating
analyses 	submitted 	by 	other 	parties 	and 	in 	assessing 	the 	effects 	on 	the
results of using alternative explanatory variables, assumptions, or statistical
techniques.
8

submitted.
acceptable
The proposals also may reduce the administrative costs of the PRe and
private parties by reducing the amount of investigation~ otherwise required.
The 	proposals 	should 	reduce 	the 	amount 	of 	necessary 	investigation 	into
theoretical 	assumptions, 	data 	sources, 	and 	statistical 	techniques 	used 	in
analyses presented to the PRC by making this information available in initial
submissions.
D. Potential Costs of the Proposals
The 	primary 	costs 	associated 	with 	the 	PRC's 	proposals 	rela,te 	to
additional 	time 	and 	care 	in 	preparing 	the 	documents 	and 	statistics 'to 	be
i
: 	I
If the analysts now submitting work to the PRC are following
: ,
professional 	procedures, 	but 	are 	not 	rep:obing 	the 	work
•
thoroughly, 	then 	the 	primary costs 	to 	them would 	seem 	to stem 	from 	the
increased 	use 	of labor, 	materials, and equipment 	to document 	the progress
of 	the 	analyses. 	We 	do 	not 	expect 	these 	costs 	to 	be 	excessive. 	Once
parties 	know 	that 	such 	documentation 	is 	expected, 	they 	may 	develop and
implement low cost procedures to preserve the necessary information.25
25 If 	the 	USPS 	or 	other 	presenters 	are 	not 	following 	professionally
accepted procedures, the new requirements would force them to do so or to
disclose 	that 	such 	procedures 	were 	not 	followed. 	In 	this 	event, 	revising
procedures to comply with professional requirements could entail additional
costs (and in some cases perhaps significant additional costs), but should also
improve the accuracy of the analyses. 	However, analyses that do not follow
professional procedures may not be particularly useful to the PRC. 	It may
be preferaole that they not be submitted.
9

These rules could impose a different type of cost by burdening the use
of 	statistical 	analysis 	to 	such 	an 	extent 	that 	parties 	opt 	to 	abandon 	this
form of analysis -- a 	troubling possibility that could 	result 	in poorer rate
and service decisions. 	We do not believe 	that the PRC's proposals 	pose a
significant 	danger 	of 	driving 	out 	objective 	statistical 	analyses. 	This 	is
because 	the 	PRe's 	proposals 	primarily 	require 	disclosure 	of 	work 	that
presumably is routinely being done. but not reported or presented.
Finally. the 	rules could 	induce some parties to attempt to confuse or
hamper understanding of statistical analyses by using the new regulations as
an excuse for "overdocumenting" the data and the theoretical assumptions or
statistical 	techniques 	employed. 	Such 	efforts 	to 	"bury" 	the 	PRC 	in
documents 	could 	adversely 	affect 	the 	PRe's 	ability 	to 	evaluate 	materials
adequately. particularly when there are tight time constraints. 26 	However,
we believe this is unlikely to happen frequently. 	The PRC nevertheless may
wish 	to 	consider 	monitoring 	statistical 	submissions 	to 	detect 	and 	to
discourage this practice if the proposed rules are adopted.
26 Although 	overdocumentation 	is 	a 	possible 	problem. 	it 	may 	be 	of
limited concern to the PRC for three reasons. 	First. current rules appear to
permit overdocumentation. which apparently has not posed a problem for the
PRC 	to 	date, 	suggesting 	that 	it 	may 	continue 	to 	be 	a 	minor 	concern.
Second, where private parties, as well as the PRC, are reviewing an analysis,
overdocumentation may not be a serious obstacle to informed decision-making
to 	the 	extent 	that 	private 	parties 	have 	sufficient 	staffing 	flexibility 	to
advise 	the 	PRC 	of 	the 	critical 	information 	contained 	within 	voluminous
submissions. 	Third, provision k.2.iv.d., calling for a summary of the sources
of the input data, should help alleviate the effect of efforts to overdocument
the data.
10

IV. 	SPECIFIC PROVISIONS
The 	PRC 	may 	wish 	to 	consider 	the 	following 	modifications 	to 	its
proposals.
Sensitivity Analysis
The PRC proposals do not iinclude provisions requiring any sensitivity
tests. 	The literature on econometrics and the experience of the FTC staff
suggest 	that 	imposing 	mod;st 	sensitivity 	testing 	requirements 	might 	be
; 	II
useful.27 	Sensitivity tests illustrate how and to what degree the resul'ts of a. 	, 	~ I
statistical 	test 	change 	with Sligh:t! changes' i~ data' or 	assumptions. 	When
results 	are 	robust. 	increased 	confidence 	in 	t~e data 	and 	analysis 	is 	often
warranted. 	Requesting modest sensitivity testing should not be burdensome.
again 	because 	good 	econometric 	practice 	already 	includes 	such 	testing.
Sensitivity analysis may also assist in expediting proceedings 	because 	they
may forestall or narrow the scope of challenges from other parties.
1
27 Leamer. 	E.. "Sensitivity Analyses 	Would 	Help." 	American 	Economic
Review 75:3 (June 1985). pp. 308-313.
The PRC proposals permit other parties to request limited tests of this
nature. but .do not require that such tests be done routinely.
11

Ad Hoc Econometric Techniques
The PRC proposes to exclude use of ad hoc econometric techniques that
have not been reviewed in econometrics texts and journal articles.28 	While
we agree that utilization of such techniques should generally be avoided, the
PRC might wish to consider altering this provision to provide an exception
when 	there 	are 	no 	viable 	alternatives. 	Such 	an 	exception 	would 	avoid
placing 	the 	PRC 	in 	the 	position 	of 	choosing 	to 	learn 	nothing 	about 	a
particular question of fact rather than choosing to learn something based on
ad hoc techniques.29
Prior Forecasts
Section 	3001.54 	provides 	specific 	requirements 	for 	the 	forecasts 	of
future 	postal 	demand. 	Assessment 	of 	the 	reliability 	(or 	bias) 	of 	these
forecasting models may be facilitated by requiring that the USPS also submit
econometric analyses indicating (I) the accuracy of previous USPS models in
predicting 	demand, 	and 	(2) 	the 	accuracy 	of 	the 	current 	USPS 	model 	in
predicting past changes in demand using earlier data.
28 Subsection "e" of proposed rule 31.k.2.iii.
29 Alternatively, 	the 	PRC 	could 	disclose 	its 	intention 	to 	discount
analyses which use ad hoc statistical procedures needlessly. 	This could 	be
sufficient to discourage the practice, once the PRC is no longer dependent
on the analysis provided by a single party. 	Further, such a disclosure policy
could 	avoid 	creating 	disincentives 	for 	innovative 	econometric 	techniques
where there are no viable alternatives.
12
. 	.

v. 	CONCLUSION
The 	PRC 	has 	proposed 	rules 	designed 	to improve documentation 	and
clarity 	of statistical analyses presented to it. 	Our experience is consistent
with the PRC's view that there is substantial room for improvement in the
presentation of statistical studies to the PRC. 	Such improvements are likely
to improve the PRC's decisions, which affect a large volume of commerce.
I 	,
In 	contrast, 	the 	costs 	of 	the 	improvements 	are 	likely 	to 	be 	small--
consisting largely of, disclosing work already 	being done. 	We believe that
benefits from these clhanges/may be substantial and that they will outweigh
, 	I
any cost~ likely to be !attributable to them.
~ !
13