FTC Staff Comment Before the Postal Rate Commission Concerning Documentation of Statistical Analyses, 1989

V890077
COMMISSION AUTHORIZED
BEFORE THE
UNITED STATES OF AMERICA POSTAL RATE COMMISSION
WASHINGTON, D.C. 20268-001
Documentation of Statistical Analyses, 1989
In Response to the Second Notice of Proposed Rulemaking
COMMENTS OF
Docket No. RM89-3
THE STAFF OF THE BUREAU OF ECONOMICS OF THE
FEDERAL TRADE COMMISSIONl
(submitted June 13, 1989)
1 These comments represent 	the 	views of the staff 	of the 	Bureau of
Economics of the Federal Trade Commission. 	They are not necessarily the
views of the Commission or any individual Commissioner. 	Questions about
these 	comments 	may 	be 	addressed 	to 	John 	C. 	Hilke, 	Federal 	Trade
Commission, Bureau of Economics, 6th Street and Pennsylvania Avenue, N.W.,
Washington, D.C. 20580, telephone: (202) 326-3483.

BEFORE THE
UNITED STATES OF AMERICA POSTAL RATE COMMISSION
WASHINGTON, D.C. 20268-001
Documentation of Statistical Analyses, 1989
In Response the Second Notice of Proposed Rulemaking
Docket No. RM89-3
COMMENTS OF
THE STAFF OF THE BUREAU OF ECONOMICS OF THE
FEDERAL TRADE COMMISSION 1
(submitted June 13, 1989)
I. 	INTRODUCTION
The staff of the Bureau of Economics of the Federal Trade Commission
(FTC) appreciate the opportunity to submit these additional comments to the
Postal 	Rate 	Commission (PRC) concerning 	the 	PRC's 	proposals 	to 	improve
documentation 	of 	statistical 	analyses 	that 	underlie 	its 	rate 	and 	service
decisions.' 	We limit these supplemental comments to (l) sensitivity tests and
(2) the track record of Postal Service forecasting models in response to the
PRC's request for more specific proposals concerning our initial suggestions
submitted on April 24, 1989. 	In what follows, we now provide more specific
proposals 	designed 	to 	offer 	substantial 	benefits 	in 	return 	for 	relatively
modest costs.
1 These comments represent 	the 	views of the staff of 	the 	Bureau of
Economics of the Federal Trade Commission. 	They are not necessarily the
views of the Commission or any individual Commissioner. 	Questions about
these 	comments 	may 	be 	addressed 	to 	John 	C. 	Hilke, 	Federal 	Trade
Commission, Bureau of Economics, 6th Street and Pennsylvania Avenue, N.W.,
Washington, D.C. 20580, telephone: ~202) 326·3483.
, The initial noti.:e of pl,:\posed rulemaking and invitation to comment
were published in th~ Federal Register. Vol. 54, No. 44, on March 8, 1989 at
pages 9848-9852. 	The second notice of proposed rulemaking and invitation to
comment were published in the Federal Register. Vol. 54, No. 98, on May 23,
1989 at pages 22317-22324.

II. 	EXPERTISE OF THE STAFF OF THE 'FEDERAL TRADE COMMISSION
The 	FTC 	is 	an 	independent 	regulatory 	agency 	responsible 	for
maintaining competition and safeguarding the interests of consumers.s 	The
staff of the FTC Bureau of Economics,4 upon request by federal, state, and
local 	government 	bodies, 	often 	analyze 	regulatory 	or 	legislative 	proposals
that may affect competition or the efficiency of the economy. 	In the course
of this work, as well as in antitrust and consumer protection research and
litigation, the Bureau of Economics staff frequently present econometric and
other statistical evidence and evaluate similar evidence presented 	by other
parties.
We 	have 	commented 	previously 	on 	various 	issues 	before 	the 	PRC,
including: (1) use of a 	single set of rate 	hearings 	to establish a 	series of
United 	States 	Postal 	Service 	(USPS) 	rate 	changes;5 	(2) 	elaboration 	of
competition 	issues 	inherent 	in 	proposed 	rate 	and 	classification 	changes
related to electronic computer originated mail (ECOM);6 (3) drawbacks to a
proposed 	modification 	of 	the 	test 	period 	for 	cost 	recovery 	in 	ECOM;7 (4)
advantages of setting ECOM rates to cover full costs;' (5) costs and benefits
S See 15 U.S.C. Section 41 ~~.
4 Staffs of other FTC bureaus also file comments separately or jointly
with the staff of the Bureau of Economics.
5 PRC Docket No. MR82-3, filed November 4, 1982.
6 PRe Docket No. R83-1, filed June I, 1983.
7 PRe Docket No. R83-1, filed June 16, 1983.
, PRC Docket No. R84-1, filed December 23, 1983.
2

of 	current 	preferred 	mail 	rates;~ (6) 	expedited 	procedures 	in 	reviewing
proposed rate changes for Express Mail;10 (7) a complaint urging a study of
the 	possibility 	of 	exempting 	addressed 	third 	class 	mail 	from 	the 	private
express 	statutes;11 	and, 	most 	recently, 	(8) 	the 	PRC's 	original 	proposed
regulations 	to 	improve 	documentation 	of statistical 	analyses. 12 	Several 	of
our comments have referenced 	the PRC's difficulties in reaching informed
decisions 	on 	postal 	rates 	and 	services 	because 	of 	the 	lack 	of 	sufficient
accessible data on the demands for and costs of different postal services.
III. 	THE PRESENTATION OF STATISTICAL ANALYSES
A. Introduction
For 	statistical 	analyses 	to 	provide 	information 	objectively 	and
efficiently 	to 	the 	PRC 	for 	use 	in 	its 	rate 	and 	service 	decisions,
participating statistical 	analysts should 	explain 	clearly 	what 	they 	did 	and
why, 	and 	the sources and 	characteristics of 	the 	data 	on 	which 	they 	have
relied. IS 	Further, statistical results should be replicable, particularly if the
~ PRC Docket No. 5586·1, filed April 20, 1986.
10 PRC Docket No. RM88·2, filed October 14, 1988.
11 PRC Docket No. C89·1, submitted February 28, 1989.
12 PRC 	Docket 	No. 	RM89·3, 	filed 	April 	24, 	1989. 	The 	present
comments of the FTC staff follow up on the April 24th comments.
IS See 	Leamer, 	E., 	"Sensitivity 	Analyses 	Would 	Help," 	American
Economic 	Review 	75:3 (June 	1985), pp. 308·313. 	Also see McAleer, M, 	A.
Pagan, 	and 	P. 	Volker, 	"What 	Will 	Take 	the 	Con 	Out 	of 	Econometrics?,"
American Economic Review 75:3 (June 1985), pp. 293·307.
3

PRC 	is 	to 	place 	substantial 	confidence 	in 	them. 14 	Documentation 	and
replication 	are 	central 	to 	the 	scientific 	method, 	and 	no 	less 	should 	be
expected 	from 	the 	statistical 	analyses 	presented 	to 	the 	PRC. 	We 	thus
support the PRC's efforts to insure that the statistical analyses on which the
PRC 	is 	expected 	to 	rely 	adhere 	to 	these criteria. 	At 	the 	same 	time, 	the
PRC 	recognizes 	that 	unnecessary 	and 	arbitrary 	restrictions 	on 	the
presentation 	of statistical 	results could discourage 	analysis and 	reduce 	the
amount of useful information available to the PRC.16 	We believe that the
PRC's current proposal makes appropriate allowance for this concern.
In addition to adopting its current proposed rules, the PRC may wish to
consider 	new 	rules 	pertaining 	to 	two 	additional 	aspects 	of 	statistical
analysis, specifically, sensitivity tests and assessment of the track record of
postal service forecasting models. 	The PRC's notice indicates its agreement
in 	principle with our previous comments on 	these topics, and 	indicates an
interest in further comments. 16
14 Statistical 	results 	are 	replicable 	if 	other 	analysts 	obtain 	the 	same
results 	when 	they 	employ 	the 	same 	data 	and 	procedures. 	Dewald, 	W., 	J.
Thursby, and R. Anderson, "Replication in Empirical Economics: 'The Journal
of 	Money, Credit, and 	Banking' Project," 	American 	Economic 	Review 	76:4
(September 1986), pp. 587-603.
16 For 	example, 	the 	PRC 	proposes 	that 	requests 	to 	the 	USPS 	from
interested parties for additional statistical analyses be restricted to plausible
alternative assumptions, rather than to all potential alternatives.
16 Federal Register, Vol. 54, No. 98 	(May 23, 1989), pp. 22317-22324.
4

·.
B. Sensitivity Tests
There 	are 	many 	sensitivity 	tests 	that 	can 	aid 	in 	assessing 	the
robustness of econometric results and the PRC could require that results for
several of these be presented,17 	If the PRC wishes to restrict its sensitivity
test 	requirements 	to a 	few 	relatively simple 	tests 	that 	are 	likely 	to entail
modest 	cost 	and 	to 	be 	prepared 	in 	the 	normal 	course 	of 	standard
econometric practice, the PRC may wish to require the three types of tests
noted 	below. 	Interested 	parties 	in 	the 	rate 	hearings 	might 	reasonably 	be
expected 	to 	routinely 	request 	the 	results 	of 	these 	tests 	if 	they 	were 	not
required 	as 	part 	of 	the 	original 	submissions. 18 	All 	of 	these 	tests 	are
17 Results 	are 	robust 	if 	slight 	changes in 	the 	data 	sample, 	variables,
and/or 	statistical 	techniques 	do 	not 	substantially 	alter 	the 	results. 	These
changes may be termed 	sensitivity 	tests. 	The goal of sensitivity testing is
to show that "minor changes in the list of variable (and other similar minor
adjustments) 	do 	not 	alter 	fundamentally 	the 	conclusions 	... " 	Leamer, 	E.,
"Sensitivity 	Analyses 	Would 	Help," 	American 	Economic 	Review 	75:3 	(June
1985), p. 308. 	Unless results are robust, little confidence can 	be placed in
them.
A list of widely used tests of the adequacy of econometric models (in
addition 	to 	those described 	below) is shown, 	for 	example, on 	page 304 	in
McAleer, 	M, 	A. 	Pagan, 	and 	P. 	Volker, 	"What 	Will 	Take 	the 	Con 	Out 	of
Econometrics?," American Economic Review 75:3 (June 1985), pp. 293-307.
Since 	standard 	econometric 	practice 	may 	evolve 	with 	changes 	in
computer technology and advances in econometric techniques, the PRC may
wish to assure that 	its rules on sensitivity testing are periodically updated
by providing a "sunset" provision for 	these rules. 	This may help to avoid
persistent application of outmoded requirements in the future.
18 Interested 	parties 	would 	be 	able 	to 	make 	such 	requests 	under
proposed rule 31(k)(2)(iii)(d). 	We believe that the cost and duration of the
PRC examination of statistical analyses n:ay be reduced by initially requiring
these 	sensitivity tests 	rather 	than 	wahing 	for 	interested parties to 	initiate
procedures under rules 31(k)(2)(iE)(d) to s,cure the results of such tests.
We note that sensitivity trsts beyond the standard set discussed below
could 	be requested 	by interested 	parties, if such additional 	testing seemed
warranted by special circumstances.
5

frequently part of standard econometric practice and should not be costly to
perform.1Q
Deletion of variables: The PRC could request that econometric tests be
rerun deleting a different explanatory variable in each computer run 	until
each 	variable 	has 	been 	omitted 	in 	one 	run.
parameter estimates could then be reported. 20
The 	resulting 	range 	of
Deletion 	of 	data: 	The 	PRC 	could 	request 	that 	econometric 	tests 	be
rerun deleting a proportion of the observations randomly selected from the
entire 	set. 	The 	correct 	proportion 	of 	observations 	to 	be 	deleted 	is
indeterminate, 	but 	in 	general 	is typically a 	substantial 	part of 	the 	whole.
The PRC could consider a rule arbitrarily requiring that at least 10% of the
sample be deleted.21
Alternative 	forms 	of 	the 	variables: 	The 	PRC 	could 	request 	that
econometric 	tests 	be 	rerun 	using 	alternative 	forms 	of 	the 	variables.
Traditionally, 	alternative 	forms 	have 	included 	the 	natural 	logarithm 	and
squared transfor~ations of the variables. 	One approach would be to require
one regression run in which all variables are considered in natural logarithm
lQ These sensitivity 	tests 	can 	be 	readily 	and 	inexpensively 	conducted
with modern computer technology. 	The analyst typically makes the necessary
changes 	in 	the 	model, 	translates 	these 	changes 	into 	computer 	commands,
reruns 	the 	regression 	program, 	and 	makes 	the 	appropriate 	comparison
between the initial results and the results for the revised model.
20 This 	test 	represents 	a 	simplified 	approximation 	of 	the 	global
sensitivity 	approach 	described 	in 	Leamer, 	E., 	"Sensitivity 	Analyses 	Would
Help," American Ecooqnic Review 75:3 (June 1985), pp. 308·313.
21 A I!jore exa..-ting and complex procedure for such partitioning of a
data 	sample 	is 	termed 	the 	"jackknife" 	technique. 	See 	Efron, 	B., 	~
Jackknife. 	the 	Bootstrap. 	and 	Other 	Resampling 	Plans, 	Philadelphia, 	Pa.:
Society for Industrial and Applied Mathematics, 1982.
6

form and another run in which explanatory variables are included in both
their original and squared forms. 22
C. Irack Record for Postal Service Forecasting Models
Section 	3001.54 of 	the PRC's current 	proposed 	rules 	provides specific
requirements 	for 	forecasts 	of 	postal 	demand. 23 	Assessmen t 	of 	the
reliability 	(or 	bias) 	of 	these 	forecasting 	models 	may 	be 	facilitated 	by
requiring that the USPS also submit econometric analyses indicating (I) the
accuracy 	of 	previous 	USPS 	models 	in 	predicting 	actual 	demand, 	(2) 	the
accuracy of the current USPS models in predicting past changes in demand,
(3) forecasts for future demand using previous USPS models rather than the
current models.
Although 	we 	understand 	that 	the 	USPS 	may 	create 	its 	forecasting
models only for the PRC's omnibus rate hearings and that the models have
changed considerably from hearing to hearing,24 we do not believe that this
22 An exception to this rule must be available when the transformation
will 	result 	in 	undefined 	values. 	For 	example, 	a 	natural 	logarithm
transformation of a dummy (0 or 	I) variable would be undefined whenever
the original variable takes the value of zero.
U 	Although 	this 	section 	applies 	explicitly 	to 	the 	USPS's 	demand
forecasts, 	the 	PRC may 	wish 	to require similar information 	if 	forecasting
models are submitted by other parties.
24 The PRC notes discontinuities in adjustments to and application of
the USPS's demand forecasting 	models, observing that the demand forecast
models have -only been implemented three times .... to the best of the PRC's
knowledge. 	Notice at p. 22223. 	As additional omnibus rate 	hearings take
place, additional applications will 	be available for 	purposes of comparison.
This should 	make 	the 	assessment of 	the 	track 	record 	of 	the 	models 	more
informative over time.
7

obviates 	the 	potential 	usefulness 	of 	examining 	the 	track 	record 	of 	the
USPS's demand 	forecasting 	models.25 	In particular, 	the estimations ot the
models discussed below should help the PRe to identify persistent biases in
the demand forecasts, major effects on the demand forecasts of changes in
the 	specifications, 	improvements 	in 	the 	accuracy 	of 	the 	demand 	forecasts
over time, and explanatory variables that have been particularly critical to
the accuracy of the demand forecasts. 	Arguably, the importance of assessing
the 	track 	record 	of 	the 	USPS's 	demand 	models 	is 	increased 	rather 	than
diminished by the fact that the models are not maintained and applied on a
continuous basis.26
We do not believe that requiring econometric analyses sufficient for a
modest review of this track record would be particularly costly to the USPS
since 	most 	of 	the 	necessary 	data 	should 	be 	readily 	available 	and 	no 	new
econometric modeling would be required.
One 	informative 	approach 	to 	establishing 	the 	track 	record 	of 	USPS
demand forecasts might consist of comparisons between the USPS's forecasts
produced 	at 	previous 	hearings 	and 	the 	forecasts 	that 	would 	have 	been
26 Even 	if 	there 	were 	only 	two 	implementations 	of 	the 	forecasting
model, comparisons between the models could still be critical in identifying
and understanding the practical effects of changes in assumptions, variables,
and techniques.
26 Intermittently 	maintained 	and 	applied 	models 	may 	be 	more
susceptible 	to 	ad 	hoc 	and 	idiosyncratic 	adjustments 	that 	have 	little
predictive power, but which give the models explanatory power with respect
to one set of recent data. 	Long-term continuous modeling efforts may be
less 	susceptible 	to 	such 	questionable 	adjustments 	because 	forecasts 	are
regularly contrasted to actual events. 	When forecasts are regularly assessed,
ad 	hoc 	and 	idiosyncratic 	adjustments 	that 	add 	little 	to 	the 	long-term
accuracy of the models can be more effectively identified and deleted.
8

produced 	if 	the 	current 	specifications 	of 	the 	USPS's 	demand 	forecasting
models 	had 	been 	in 	place 	in 	each 	of 	the 	prior 	omnibus 	rate 	hearings.
Forecasts (based on the new and old specifications) could also be contrasted
in each of these instances with actual demand. 	During the 4th rate hearings
(as 	yet 	unscheduled), 	for 	example, 	the 	then 	current 	specifications 	of 	the
demand forecast models will be applied to then current data to produce the
new demand forecasts proposed by the USPS.27 	The following comparisons
could then be made:
a. Current specifications of the demand forecast models applied to data
from 	the 	previous 	omnibus 	hearings 	(3rd 	hearings). 	This 	would 	produce
forecasts to compare with the forecasts presented by the USPS at the third
hearings 	(which 	used 	the 	specifications 	and 	data 	from 	the 	3rd 	hearings).
The forecasts using the new specifications and 3rd hearing data also could
be compared with actual demand observations after the third hearings.
b. Current specifications of the demand forecast models applied to data
from 	the 2nd hearings. 	This would produce forecasts to compare with the
forecasts 	presented 	by 	the 	USPS 	at 	the 	second 	hearings 	(which 	used 	the
specifications and data from the second hearings). 	The forecasts using the
new specifications and 2nd hearing data also could be compared with actual
demand observations after the second hearings.
c. Current specifications of the demand forecast models applied to data
from 	the 	1st hearings. 	This would produce forecasts 	to compare with the
forecasts 	presented 	by 	the 	USPS 	at 	the 	first 	hearings 	(which 	used 	the
specifications and 	data 	from 	the 	first 	hearings). 	The 	forecasts 	using 	the
27 In all cases, the USPS and other interested parties could be invited
to 	supply 	evaluations 	of 	differences 	in 	results 	for 	different 	models 	and
different time periods.
9

new 	specifications and 	the 	1st 	hearing 	data 	also 	could 	be 	compared 	with
actual demand observations after the first hearings.
Another avenue of inquiry might be to compare the current forecasts
with forecasts 	produced by applying each of the previous specifications of
the 	USPS's 	forecasting 	models 	to 	current 	data. 	During 	the 	next 	omnibus
rate hearings the following comparisons could be made:
d. Specifications of the demand forecast models from the third hearings
applied 	to current data 	to 	produce demand 	forecasts 	to compare 	with 	the
USPS's new demand forecasts.
e. 	Specifications 	of 	the 	demand 	forecast 	models 	from 	the 	second
hearings 	applied 	to 	current 	data 	to 	produce demand 	forecasts 	to compare
with the USPS's new demand forecasts.
f. Specifications of the demand forecast models from the first hearings
applied 	to current data 	to 	produce demand 	forecasts 	to compare 	with 	the
USPS's new demand forecasts.
IV. CONCLUSION
The 	PRC's 	proposals 	to 	improve 	documentation 	of statistical analyses
appear 	to 	be 	well 	conceived 	and 	in 	our 	view 	may 	result 	in 	substantial
improvements at modest cost. 	In two areas, the PRC may wish to consider
certain additional proposals which 	we believe may also provide substantial
benefit at relativel" modest cost.
10

,-	..
In 	the area of sensitivity tests, we have 	identified a 	number of tests
that we believe could supply important information about the robustness of
statistical 	analyses 	presented 	to 	the 	PRC. 	In 	general, 	these 	tests 	are
relatively 	simple 	and 	inexpensive 	to 	perform. 	Further, 	we 	believe 	that
analysts 	are 	likely 	to 	have 	performed 	them 	in 	the 	normal 	course 	of
statistical investigation. 	Finally, costs and delays might be decreased since
other 	interested 	parties 	should 	now 	find 	it 	unnecessary 	to 	independently
request the results of these common tests for robustness.
In 	the 	area 	of 	assessing 	the 	track 	record 	of 	the 	USPS's 	demand
forecasts, the PRC may wish to require a relatively simple and low cost set
of 	comparisons 	that 	may 	provide 	important 	insights 	about 	features 	and
changes in the USPS's demand forecasting models.
11