Approaches to Analyzing Economic Impact: Yardstick/Benchmark Analysis and Before/During/After Analysis

When using data such as prices, wages, or sales to answer economic questions there are several dimensions on which comparisons can be made. The purpose of this article is to describe two of those. First, one group can be compared against another group. This is often referred to as yardstick or benchmark analysis. Second, one time period can be compared to another. This is often referred to as before/after, pre-post, or before/during/after analysis.

Yardstick/Benchmark Analysis

Benchmark analysis (also referred to as yardstick analysis) compares one group against another in terms of a measurable effect or outcome. Often this takes the form of comparing a group subject to a factor of interest to a group that is not subject to that factor. In a laboratory setting, this would equate to comparing a treatment group to a control group. One group received the treatment, the other did not. If the two groups were otherwise similar, such that the only difference between the two was receiving the treatment, then it may be reasonable to conclude that the difference between the groups was due to the treatment.

Ensuring that the two groups are similar will help isolate the effect that is being tested for. If the groups have too many different characteristics, this raises the possibility that differences in outcomes between the groups are due to the different underlying characteristics. These are often referred to as “confounding factors.”  Economists can use various tools to adjust or account for the presence of these confounding factors in order to isolate the effect of interest.

Sometimes two groups may have significant-looking differences that turn out to be unimportant regarding the question at hand. For example, even if one group of customers is significantly older, this wouldn’t matter if age does not impact purchase decisions or other market outcomes. Sometimes experts disagree about what factors need to be controlled for. As noted in Celebrity Cruises Inc. v. Essef Corp., 434 F. Supp. 2d 169, 189 (S.D.N.Y. 2006): “Arguments about what factors an expert should have controlled for in conducting a yardstick analysis generally go to the weight [given to an expert’s testimony], rather than the admissibility, of the expert’s testimony.”

Alternatively, some differences between groups can be important in terms of being able to isolate the effect of interest. Consider a case addressing whether advertising impacted purchase intentions. Such differences may be impactful if one compares a group consisting of those already likely to purchase the product against another group consisting of those who are not in the market for the product. Such a situation could result in misattributing underlying preferences to the advertising. The court in El Aguila Food Prods. Inc. v. Gruma Corp., 131 F. App’x 450, 453 (5th Cir. 2005) excluded an expert’s opinion because the expert “made no effort to demonstrate the reasonable similarity of the plaintiffs’ firms and the businesses whose earnings data he relied on as a benchmark.”

Before and After, Pre-Post, or Before/During/After Analysis

The before/during/after (sometimes referred to as before/after or pre-post) approach compares one (or more) time periods against another time period. For example, comparing a period when a company was allegedly breaking the law versus a period when it was not.

The “during” period is a period of time, e.g., days, weeks, months or years, during which an effect is alleged to be present. The “before” and “after” periods are before and/or after that allegedly affected period. There are several combinations of comparisons that can be made:

• Before/during, which compares an unaffected “before” period to the affected “during” period. This is sometimes referred to as “before and after” because it compares a period before the start of the effect and a period after the effect has started. Similarly, it may be referred to as pre-post given that it compares a period preceding an event (e.g., start of policy) to a period post-dating that event.
• During/after, which compares an unaffected “after” period to the affected “during” period; and
• Before/during/after, which compares unaffected “before” and “after” periods to the affected “during” period.

This approach differs from the yardstick (or benchmark) approach. The yardstick approach compares two different groups during the same time period. The before/during/after approach compares the same group over different time periods. Note that the term “benchmark period” may be used to describe the clean, unaffected period(s) in a before/during/after approach.

It is important to consider what other factors might be driving differences across time, analogous to the way the confounding factors noted above can influence the validity of tests. An economic expert can evaluate the relevance of these factors and run statistical and/or econometric tests to detect and isolate impacts specific to various factors. They key word here is “isolate” which allows an economist to identify the cause-and-effect of factors, rather than mere correlations.

Another important consideration is defining the beginning and end of each period. For example, when does an effect begin or end? If the affected period is defined too broadly, it may include points in time that were not affected or were affected by something else not of interest. In the former, the estimate of the average impact over this too-broad a time period will under-estimate the true impact (if any). In the latter, the average impact could over-estimate the true impact (if any).

The average impact may underestimate the true (full) impact if the alleged impact is also bleeding over into the before or after periods. Sometimes this is unavoidable. See, for example, In re Linerboard Antitrust Litigation, 497 F.Supp.2d 666, 683-684 (E.D.Pa. July 30, 2007):

“…if there was in fact collusion during the benchmark period, [the expert’s] but-for price estimate would be too high, causing his estimate of the overcharge (the difference between actual prices and but-for prices) to be too low. Accordingly, the Court rejects defendants’ argument that [the expert] incorrectly assumed that his benchmark period was free of collusion.”

This raises the question of how to treat periods when the presence of the challenged conduct is uncertain. These time periods have been referred to as “agnostic periods.” See, for example, In re Mushroom Direct Purchaser Antitrust Litigation, No. 06-0620 (E.D. Pa. July 29, 2015) (refusing to exclude an analysis where the expert “remained ‘agnostic’ about whether there was anticompetitive conduct” during a given period based on certain evidence).

Conclusion

Two methods of analyzing impact are yardstick (or benchmark) and before/during/after analysis. Yardstick analysis compares differences between affected and unaffected groups. Before/during/after analysis compares differences between affected and unaffected time periods. In both cases the impact is measured as the difference between an impacted observation and an unimpacted observation. For the reasons described above, it is important to evaluate and adjust for significant factors that would explain the difference between the impacted and unimpacted observations.

It is up to the judgment of the expert to determine which of these two methodologies best fit the facts and circumstances of the case. An experienced expert will be able to determine the best way to utilize available data to ensure these methodologies are implemented in a reliable and accurate manner.