Open source vs. closed source is not really a question of management. If you have a free software project, that does not mean you have to do any more management than you would of a propietary project. You can still limit the number of committers, so you always have control over the code itself.
If you have lots of people sending you patches, it's hard to see how that's a bad thing. If you don't want to spend too much time reviewing them, then simply impose very strict standards (formatting, includes tests, log messages, etc.), which will let you reject a lot of the patches with little effort.
As to having a vision, I don't see any difference in closed vs. open here. The only problem is if you think open means you have to let 30 people all have a say in the project. But free software does not require consensus-based decision making! If this core team of four people wanted to make all the decisions, the worst they'd have to deal with is a fork, which by definition is not an additional burden on the core team, except perhaps for damaging their egos.