According to Val Henson, men outnumber women a bit over 2:1 in proprietary software, but about 65:1 in open source.
I think that you have actually hit the nail on the head better than any other commentators as to what we need to do to change this. I.e. we need to focus on creating architectures of participation where everyone can contribute according to his/her ability, and encouraging people to participate.
However, as in mathematics, there appears to be a fairly male-centered culture around open source. I personally think that it has to do with most great hackers being borderline OCD and coming from families where the object of one's obsession is technical (for example, I come from a family of scientists, and my grandmother was writing Astrophysics simulations in Fortran since before I was born). I will sometimes obsess about techical issues so that I am nearly always looking for a solution to them for *days.* Hence within the hacker culture there may be cultural issues beyond our immediate control. The challenge is hence to ensure that the hacker culture doesn't equal the FOSS culture, and that everyone is welcome provided that they are not obnoxious or harmful to the project.
The problem with these larger issues, however, is that they allow sexism in open source projects to go unnoticed, and for a very male-centric culture to thrive in some areas of FOSS.
In this spirit, I think that it would really help to build larger communities if there were groups out there that were rating open source projects as to how well women felt like they were encouraged to participate. I think we would find in this case that those projects which incouraged widespread contribution from large numbers of people would do better than those which tried to restrict contributions to a small number of core people. This would also allow us to look for other things that could be done to help build a more diverse contributor culture.