Looking for Gender Effects-- Re-Examination of the Data in Budden et al.

First off a whole bunch of caveats:

1) I think double blind review is a fabulous idea.

2) I think that Budden et al. may be correct, but a more complex analysis of their data only supports part of their hypothesis.

3) This blog is not peer-reviewed, Trends in Ecology and Evolution is. So commenters, make sure I didn't screw up!

A brief rundown of the paper:

The paper is available here. Other blog posts about this article are here, here & here (and probably a few more that I missed).

The basic idea and hypothesis tested in this paper is that double blind review will impact the proportion of first authored papers published by females. I saw this yesterday and thought that this would be great. I was kind of disappointed when I really read the study. They compared articles published in 6 journals over two different time periods. In one of the journals, Behavioral Ecology, they instituted double blind reviewing and the data indicate that there is an increase in the % of female first authored papers in that journal. This leads to the issue I noticed and that Bob O'Hara has also done a nice post on. Bob, like myself noticed that the comparisons were not done among journals, but were rather based on if there were significant shifts from time point 1 to time point 2 within the journal. It is clear that the largest increase does occur in Behavioral Ecology (7.9%), but is this really different than the shift in authors seen in the other journals (Animal Behavior, Behavioral Ecology and Sociobiology, Biological Conservation, Journal of Biogeography, and Landscape Ecology).

My approach to this was to fit a generalized logit model using Proc Catmod in SAS v 9.1.3 (see here & here) where the first authors sex was the response (female, male, or unknown) and the predictors were journal (which one), year (before or after the introduction of double-blind reviewing in Behavioral Ecology), and the interaction between year and journal. The interaction will tell us if the response is different across years from different journals which one would predict based on the hypothesis above. That is, the proportion of female first authored papers should increase more in Behavioral Ecology than in the other journals. Of particular interest is comparing the change in female first authored papers in similar journals to Behavioral Ecology (e.g. Animal Behaviour, Behavioral Ecology and Sociobiology) since the comparison to Behavioral Ecology and Sociobiology was what was done in the original paper.

Here's the rundown of the results. The full analysis, data, and code are available on request from me if you'd like it (email:walkers2@mac.com). There's a significant impact of journal (P<0.0001), p="0.0018)," p="0.0009)." p="0.03)," p="0.0003)," p="0.0003).

Animal Behavior versus Biological Conservation-P=0.047

Animal Behavior versus Behavioral Ecology-P=0.5632

Animal Behavior versus Behavioral Ecology and Sociobiology-P=0.6044

Animal Behavior versus Journal of Biogeography-P=0.0019

Animal Behavior versus Landscape Ecology P=0.3148

Biological Conservation versus Behavioral Ecology-P=0.2356

Biological Conservation versus Behavioral Ecology & Sociobiology-P=0.0137

Biological Conservation versus Journal of Biogeography-P=0.0612

Biological Conservation versus Landscape Ecology-P=0.0273

Behavioral Ecology versus Behavioral Ecology and Sociobiology-P=0.3362

Behavioral Ecology versus Journal of Biogeography-P=0.0059

Behavioral Ecology versus Landscape Ecology-P=0.5323

Behavioral Ecology and Sociobiology versus Journal of Biogeography-P=0.0018

Behavioral Ecology and Sociobiology versus Landscape Ecology-P=0.5032

Journal of Biogeography versus Landscape Ecology-P=0.0013

Based on this analysis, which is done in a similar framework as the original analysis and does not add any random effects, there's no significant shift in the distribution of author types when comparing Behavioral Ecology to Animal Behavior or to Behavioral Ecology and Sociobiology (both of which are similar journals). Does this mean that double blind reviewing didn't have an impact on female first authored papers? I'm not sure, but it does suggest that the shifts scene in Behavioral Ecology aren't really that different from shifts observed in similar journals.

The way the data were analyzed in the original paper illustrates something that I've seen before. When separate analyses are done to see the impact of a factor on two different variables, I've seen the 'A significantly impacts B' but 'A does not significantly impact C' so 'B & C are responding differently to A' before. However, the third assertion is never tested when the analyses are not done in a factorial fashion and include interactions.

This is a good paper and I think double-blind reviews are a good thing and will result in a fairer reviewing process for scientists. However, I don't think that the data in this paper strongly support the hypothesis that double blind reviews increase the proportion of female first authored papers.

.