That's why we call it "planning for the unknown" and "allowing for extension". If you want to provide an extensible architecture, you can't be bound to what you know when you compile the code. If you stick with only what *you* know now, how can that be extended later by someone else?
In addition, you get the binding that matters; the compiled image binds to the methods of the interface directly, without any of that late-binding IUnknown-iness from good old COM, so you don't suffer a performance problem. The only thing you have to do is wrap the call to the factory in a try/catch block in case the reflective lookup fails, and you ought to be double checking any "new" calls anyway.
But, if you don't need future-proofing in your app, don't use the technique. Nothing fits all problems...