It's a little of both. From an ISPs point of view, allowing any user to run programs on a web server, by request, is dangerous. Not only can the ISP not certify the security of these programs (without actually examining and understanding the code itself), but they also can't tell if the programs are well written - bad ones could cause infinite loops, take up lots of memory, and so forth.
From a user point of view, oftentimes, CGI scripts need to have special permissions on data files, so that the CGI script can read AND write to and from that file. These permissions often are 666 (which means "let everyone in the world read and write to this file"). If I hated you, for example, I could write a CGI script in my directory that could modify and mess with a data file in your directory, which of course, is not good.
Confining CGI scripts to a certain directory like /cgi-bin/ gives perk benefits - no one can view the contents of that directory from the web, for example, likewise making it harder to see secretive data contained within that directory. That protection doesn't happen if you can run scripts everywhere.