I hope the moderators of this site don't mind if I mention a product that I have recently developed, which solves the problem of Form creation and modeless Form display in a VB6 DLL, as I'm sure it will be of interest to readers of this article.
As various questioners above have found, an API dll written in VB6 can not reference any GUI elements - more specifically it can not access the VB6 runtime private class library, which in normally linked EXE's and ActiveX dll's acts as the class factory for the modules's VB Forms and Controls. Any attempt to do so will crash the process (i.e. your client's application).
This problem is actually not just restricted to Forms. It affects other 'Global' objects as well (e.g. the App and Err objects).
In a nutshell, the problem boils down to the fact that the private VB class library (class factory) initialisation is not performed when a VB dll is accessed via an API interface. This initialisation is buried away deep inside the dll's external COM interface.
I have found a way of getting this initialisation performed, thereby enabling a VB6 dll to escape these restrictions, and to create and show Forms when called via the API from a client application written in just about any language.
The only restriction is a fairly reasonable one - if you want to show modeless (asynchronous) Forms, the client application needs to have a window.
You may be familiar with the error that some ActiveX component providers can get if they try to show a modeless Form to a non-VB client: "Only modal Forms can be shown to a non-VB client".
That's because the only way that a Form in an ActiveX dll can receive its window messages is from the message pump in a VB6 EXE client, or the builtin message pump embedded in the "Show vbModal" method.
I demonstrated some years ago now that this was in fact a rather artificial restriction by VB6. Any client that has a window has a message pump and that pump will dispatch messages to any VB Forms in the process, as they are still normal top-level windows.
VB6's restriction appears to be based on the fact that the dll's Form in that case would not be part of the "managed Form" set of the application.
In any case, if an ActiveX DLL wants to show a modeless Form to a non-VB client, it just needs to use the ShowWindow API instead of the VB6 Show method.
And exactly the same thing applies to a VB6 API dll. If the client is not a VB6 exe, it should use ShowWindow instead of the Show method.
Happily this technique works with VB6 clients in any case, so the DLL does not really need to distinguish between them unless it wishes to.
Being able to show Forms, especially modeless ones, opens the way for VB6 developers to provide packaged services that incorporate a GUI through a simple API interface, avoiding the typical deployment hassles of the COM interface, such as the requirement to register the DLL, binary compatibility issues, etc.
I call the product "The VB6 SuperServer DLL Builder", and you can use it with any custom linker like the one described in this article.
Please email me if you'd like further details, I can provide a working demo.