**2. Suggestions for avoiding the problem of publishing overmatched results**

One way to conceptualize the problem of overmatching in conditional logistic regression is the preemptive removal of variance that should stay available for the hypothesized predic‐ tor variable to attempt to account for. The total variance in a data set can be defined using the average squared distance from the mean score for each participant: s2 = SS/df. A question related to overmatching concerns how much of the total variance is taken out beforehand (matched out)? How much is too much? 10%? 90%? 50%? We would propose that the per‐ cent removed before testing should normally be small compared to the total. Further, the re‐ moval of this variance should only occur when there is authentic need: when the potential matching variable is likely related to the outcome of interest via a path that is distinct from the risk variable of interest in a case-control design.

of vaccine safety is high stakes. There are concerns that a proper test of the full vaccine schedule has not been properly tested, and that the safety tests that exist have been de‐ signed by the vaccine industry itself. Such concerns about conflicts of interest may be preventing otherwise willing parents to adhere to the full vaccine schedule. Vaccines have been and will continue to be a huge benefit to humanity. But this paper is flawed.

**Figure 5.** Which manufacturer a given provider used for the vaccines varied by HMO. Manufacturers differed in their thimerosal use. For example, in 2002, Smith, Kline and Beecham were using thimerosal in their HepB vaccine, while Merck did not. While the data set is careful to note manufacturer and Hg in the associated batch and manufacturer,

The Price et al. research is an interesting case of overmatching that we think is of general interest in the field of epidemiology. To avoid misunderstanding, we wish to state that this research does not support the argument that vaccines or thimerosal in vaccines cause au‐

**2. Suggestions for avoiding the problem of publishing overmatched**

the average squared distance from the mean score for each participant: s2

One way to conceptualize the problem of overmatching in conditional logistic regression is the preemptive removal of variance that should stay available for the hypothesized predic‐ tor variable to attempt to account for. The total variance in a data set can be defined using

related to overmatching concerns how much of the total variance is taken out beforehand (matched out)? How much is too much? 10%? 90%? 50%? We would propose that the per‐

but CLR matching on HMO results in comparing cases to controls who had the same levels of exposure

tism. It is however, uninformative to the question.

**results**

Provider/HMO

**ASD Risk**

Amount of Exposure via vaccine manufacturer

= SS/df. A question

Unfortunately, there is not an analytic fix for overmatching: it is design flaw.

106 Recent Advances in Autism Spectrum Disorders - Volume I

Figure 5.

As elaborated above, matching is appropriate only if the matching variable is a strong pre‐ dictor of the outcome of interest, but it is not appropriate when the matching variable is strongly related to the exposure risk variable. We offer three suggestions to help objectively identify, and thus avoid, the problem of overmatching.

**Empirical Support.** Before matching, first and foremost, researchers should locate studies that suggest the potential match is likely correlated to the probability of the outcome occur‐ ing. These should be cited to support the need to match on that variable. If there is no reason to think the matching variable relates to the outcome, there is no reason to match it.

**Remaining Variance.** Next, once the participants have been selected as a matched data set, researchers can check to get an idea how much variance in the exposure variable is actually accounted for by the matching variable M. If only a small amount of the variance is left after the various matching, matching on the variable(s) cannot be justified and an unmatched or lesser matched set of participants is called for. Specifically, a check to see if too much of the total variance in the outcome of interest is matched out could be done by requesting Partial Eta Squared. Partial Eta Squared represents the proportion of the total variance that is ex‐ plained by the between factor when an ANOVA is performed. Specifically, one can take the extra step of analyzing the variance in the risk factor of interest (e.g., thimerosal exposure) as a function of the matched variable (e.g., HMO or BirthYear). In this example, using thi‐ merosal exposure as the dependent variable, the total SS is 23507522. The SS associated with the Birth Year is 1485471. This gives Partial Eta Squared =.456, meaning that about 46% of the total variance in thimerosal exposure is fully explainable based on Year of Birth. When one matches on this, only about half (54%) of the variance is left.

HMO, the other variable matched on, removed about 30% of the variance.

The percent that should be left would depend on the research question and causal assump‐ tions, but we suggest that if a matched variable is removing more than a fourth (25%) of the variance (corresponding to a large effect size, Cohen, 1977), matching is unlikely to be war‐ ranted for this reason alone and welcome commentary on this benchmark proposal.

**Relative relations.** Finally, there are times when it could be proper to match on a variable that accounts for variance in the risk factor being tested. A recent case coincidentally also related to vaccines helps to illustrate this more. It had been pointed out that the enormous benefits of the flu vaccine among the elderly appeared to far surpass even the effect that a total eradicating of flu from the vaccinated population could account for (Jefferson, 2006). After additional investigation, much of the original effect appears to be due to the tendency for seriously ill and/or less healthy elderly persons not to have the flu shot. To be clear, most of the flu vaccine effect on mortality was found to be due to health of the participants inde‐ pendent of the flu shot (Jackson et al., 2006). In this case, if this had been a case control de‐ sign, the risk factor would be flu vaccine and the probability outcome of interest would hospitalization or death. In such a case control study, it would be proper to match on preex‐ isting health, even though one would find that health accounts for some of the variability in getting or not getting the risk factor (flu vaccine). BUT: health would also relate to the mor‐ tality outcome, and even more strongly. It is this strong relationship that is key. If the varia‐ ble is more strongly related to the outcome – this serves to justify matching.

sure variable being tested. It is well known that matching on a variable that is associated on‐ ly with exposure, not with disease, reduces statistical efficiency in a case – control design (Zondervan et al, 2002; Rubenfeld et al., 1999; Day, Byar, & Green, 1980), and this in essence,

Vaccine Safety Study as an Interesting Case of "Over-Matching"

http://dx.doi.org/10.5772/53876

109

To sum, variables such as birth year, HMO, age, gender, address should first and foremost be matched if and only if there is a truly justifiable rationale to expect they have an inde‐ pendent causal pathway to the outcome; "matching will increase he efficiency of the study when the matching variable is a strong outcome determinant, but will actually reduce it when the matching variable is strongly related to the exposure variable (over-matching)," (Hanson & Khamis, 2008, p.595-596). Second, if the majority of the variance in the risk factor being tested is removed by matching, before the hypothesis is tested, extreme caution in re‐ porting a lack of effect is warranted. Finally, recalling that sample size will be held constant, testing the relationships of M to R and P and comparing the p values can be used to justify matching in the context of the matching variable removing variance relating to the risk fac‐ tor. We would propose that overmatching has and will continue to be a problem in matched case control designs, but suggest that employing the three checks above will serve to lessen

This work was partially funded by a small grant awarded to the second author, Robert T.

[1] Austin D. W., Shandley K. A., Palombo E. A. 2010. Mercury in vaccines from the Australian childhood immunization program schedule. *J. Toxicol. Environ Health A*,

[2] Agudo A. and González C. A. 1999. Secondary matching: A method for selecting controls in case-control studies on environmental risk factors. *Int. J. Epidemiol.*, 1999:

deleterious effects associated with publishing overmatched results.

Hitlan from Safeminds. We thank Safeminds for their support.

defines the problem of overmatching).

We welcome comments on these proposals.

M. Catherine DeSoto and Robert T. Hitlan

University of Northern Iowa, Cedar Falls, USA

**Acknowledgement**

**Author details**

**References**

73: 637-40.

1130-1133.

To objectively quantify this, one needs to know how strongly M is related to the Risk Factor R; and then how strongly M is related to the probability of response P. A problem is that different types of data can make precise comparisons of effect size hard to judge.

Assume that M would be HMO, R would be mcg Thimerosal exposure, and P would be ASD diagnosis. It would be desirable to compare the size of this relationship M to R with the relationship of M to P. It would be ideal if one could simple compute correla‐ tions for M and R and for M and P. However, in most cases this would not work: the scales are not all continual, and even if one were to employ a Spearman correlation, it would not be apparent how to code something like HMO to insure a linear relationship. What if HMO 2 was associated with an increase in thimerosal, and HMO 1 and 3 both had low levels? This would result in a low correlation due to the curvilinear relation‐ ship, even IF much of the variance were in fact associated with HMO. On the other hand, the relationship between HMO and thimerosal (M and R) can be checked via AN‐ OVA easily enough since R is continual and M is categorical. ANOVA would not work for testing association between M and P because both M and P are both categorical in this case. Chi – Square would be appropriate. However, regardless of the correct hypoth‐ esis test, all hypothesis tests are in fact unified by the p value.

The p value is a function of the size of the effect and the sample size. Different types of stat‐ istical tests have different probability distributions, but the total area under the curve has a constant meaning across tests. The percent of area covered means the same thing in any test, regardless of the precise shape of the curve associated with a particular statistical test (corre‐ lation, ANOVA, Chi-square). A small p value could be due to a large effect, or it could be due to a very small effect and a very large sample. It should be stated that when sample sizes are similar, it will not be unduly affected by sample size differences. Since the sample will be the same for testing M to P or testing M to R, we propose the p values are the most readily available means to index the comparison.

Compute a measure for the relationship of M and P and the associated p value. (e.g., HMO and ASD: X2 (2) = 1.59, p =.45 )

Compute a measure for the relationship of M and R and the associated p value. (e.g., HMO and Thimerosal exposure: F (2,1090) = 237, p <.0001).

The p value in all cases should be smaller for the M to R relationship, compared to the M to P relationship test. This will serve to demonstrate that even if the Matching variable does bear some relationship to the risk factor for the outcome probability, there is clearly a stron‐ ger relationship to the outcome itself, thus objectively justifying the matching. (e.g., the p value of.45 indicates no relationship exists between the matched variable and the outcome of interest, while the p value <.0001 indicates that matched variable is related to the expo‐ sure variable being tested. It is well known that matching on a variable that is associated on‐ ly with exposure, not with disease, reduces statistical efficiency in a case – control design (Zondervan et al, 2002; Rubenfeld et al., 1999; Day, Byar, & Green, 1980), and this in essence, defines the problem of overmatching).

To sum, variables such as birth year, HMO, age, gender, address should first and foremost be matched if and only if there is a truly justifiable rationale to expect they have an inde‐ pendent causal pathway to the outcome; "matching will increase he efficiency of the study when the matching variable is a strong outcome determinant, but will actually reduce it when the matching variable is strongly related to the exposure variable (over-matching)," (Hanson & Khamis, 2008, p.595-596). Second, if the majority of the variance in the risk factor being tested is removed by matching, before the hypothesis is tested, extreme caution in re‐ porting a lack of effect is warranted. Finally, recalling that sample size will be held constant, testing the relationships of M to R and P and comparing the p values can be used to justify matching in the context of the matching variable removing variance relating to the risk fac‐ tor. We would propose that overmatching has and will continue to be a problem in matched case control designs, but suggest that employing the three checks above will serve to lessen deleterious effects associated with publishing overmatched results.

We welcome comments on these proposals.
