The Web 2.0 Jobs Boom

by Kurt Cagle

Three years ago, I was in the midst of both a personal and a financial depression. The aftermath of 9/11 had collided with the collapse of the Internet Bubble, leaving the tech sector in ruins. While the general population seemed to get by with remarkably little impact - especially if you were in housing or finance - being a programmer then was synonymous with being unemployed.

I started writing my first blog about then, though it was done primarily in the form of an email newsletter. I wrote long missives, often going on for eight to ten pages or more, about the fundamental unfairness being perpetrated upon the programming community. Much of it, in retrospect, involved working through a certain amount of anger and bitterness that I needed to get out of my system before I could start writing again professionally. Some of it, however, was warranted.

The tech sector in the late 1990s was poised for a fall. It was obvious, even before the NasDAQ came tumbling down in March of 2000, that there were too many people in the field, that there were too many companies trying to build killer apps in untenable niches using jury-rigged and unstable technologies, all chasing after the get-rich-quick IPO. Add into that a swell due to a significant number of semi-retired programmers coming back into the market to chase the Y2K bug, and the large number of Indian and Chinese contractors who were brought over because they were perceived as being cheaper labor than the native pool, and it was pretty much inevitable that when the bubble burst, it would be nasty.

In the last four years, there has been a dramatic sea change in the tech sector. The Y2K programmers have retired, likely for good. Many people on H1B visas had to go back home once the companies that had hired them collapsed, and after four years they are far more busy seeding their own economies with startups rather than making the long, arduous trek to the US. Moreover, as the dollar has weakened, its appeal to bring these workers over has consequently diminished.

Journeymen programmers found jobs as policemen, or lawyers, or accountants, or selling goods on eBay, and to a great extent they are not anxious to come back into the pressure cooker of the tech field (and are very nervous about Bubble 2.0). The ones that did well originally also retired young, and are now buying up real estate or gold or sailing on their yachts.

Finally, after ratcheting up in the 1990s, all too many colleges were left having developed programs and staffing them with fairly expensive professors, only to have the class count practically dry up after the NasDAQ collapsed. They shed faculty as fast as they could after that, and are likely to be very conservative in building up new tech departments ... which in turn has put a major damper on the number of incoming college trained programmers that acted as the shock troops in the last boom.

It's now 2006. The S&P has largely recovered its standing from its high, but the tech heavy NasDAQ is still nearly 60% below its maximum. The Google IPO was stratespheric, but I'm not so much convinced that Google is the harbinger of a new era so much as it is one of the few popping embers of a dying one - and its play in the advertising space makes me convinced that very senior people there know it.

There will be a few stellar companies out there in the next era, but my gut feeling at this stage is that the next wave is not playing out in the large spaces with companies coming to market with really cool technology. Rather, what is loosely called Web 2.0 is involved primarily with getting companies mapped to common, open standards, potentially (though not necessarily) on open software.

The fundamental problem that open standards has for the "software industry" is that it is in point of fact pure poison. This is an odd position for someone who advocates open standards to take, but I don't think its an unreasonable one. Most software companies in the pre-dot-com era were far more interested in technology differentials - niches in which the difference between two proprietary systems required a bridge to be built. Put a toll both on that bridge, and you can make money.

Open standards abolish those bridges. This means that companies either need to create artificial barriers faster than others can knock them down, or they need to provide a means to simplify the development of the bridges themselves and step out of the way. (The one area where this is absolutely not true is the computer game industry, but this is in fact a sector of entertainment, not information technologies, and has its own boom/bust cycle).

This is tremendously bad for the software industry as a whole, which has been limping towards SOA architectures more out of necessity to stay in the game than because it represents the next major frontier. Ironically, however, it is tremendously good for programmers, at least those that don't mind putting in some serious work.

Business requirements change - indeed, it is the accelerating phenomenon of the web that has shortened the business change cycle so much that business objectives change on a daily or even hourly basis. For this reason, traditional waterfall methodologies have largely been abandoned in favor of more agile techniques, with as one consequence the shift of programming away from large groups of hundreds of developers into much smaller groups of between one and maybe a dozen programmers, each group working largely in parallel in a loosely connected manner.

Commercial development (and adoption) has tended to stress a one-size-fits-all model (aka Turnkey software) that does in fact fit everyone ... badly. Customization of this software was, generally, seen as someone else's problem by the vendor. While this opened up the door for a lot of third party "consultants", most of whom were just barely trained in the software in question, it also meant that the customization base become highly balkanized, and usually led to customers with systems that either worked poorly or didn't work at all.

The community model of software development is emerging due more due to economic pressures than it is because of ideological evangelism on the part of a few open source evangelists. In essence, in the adoption of such software, the customer effectively gets the opportunity to utilize the software without the need to pay for the development costs of the core software -- only the customizations. These customizations, in turn, were typically rolled back into the core package either as compatible extensions to the product (because other people may also have similar needs and thus have a vested interest in that compatibility) or as the foundation for the next generation of the product itself.

The developers, ironically, still get paid ... and are still needed. However, what changes here is that what are needed tend to be specialists, not journeyman generalists. What's more, they are not in the specializations that were big even half a decade ago. Who would have known five years ago that having strong Javascript skills today would get you a better job than having strong C++ skills? There were any number of companies that bet the bank (and lost) thinking that SOAP/WSDL based web services would be the next big thing; within the boundaries of financial services that's been true, but in the consumer space - gee, even after half a decade, we're still buying milk by going to the grocery store, and the refrigerator is still not terribly bright about its contents.

What this means is simple. There aren't enough programmers to go around. Simple supply and demand ... wages are beginning to rise pretty dramatically again, top people in the field have to get social secretaries to handle all of the cold calls and sales pitches, and companies outside of the IT sector are relying increasingly upon community software to tap into the developer base.

We are entering, in fact, something that IT companies have been dreading for a while ... a tech crunch. For some companies, there's a certain amount of payback that's occuring ... hiring managers had become used to having their pick of programmers at comparatively low wages, and to being able to fire those programmers or treat them badly because there was an apparent glut. No wonder so many left. Now, those same companies can't get good talent because word does get around, and they will increasingly find it difficult to hire even the indifferently talented because there are comparatively few of those left.

However, beyond a cetain satisfactory karmic justice, the reality is that demographics are actually working against companies now in a big way. The US is fast losing its appeal as a technology center as other companies develop their own infrastructure (and offer competitive wages), though even in other countries the situation is far from favorable for hiring managers. The long term movement where women are making up the majority of college graduates plays against IT because programming is still seen as being too much of an alpha male geek sort of thing (which has more than a little truth to it). There's no reserve of "emergency coders" that can come out of retirement, and the echo effect of the GenXers is now fading as that group moves out of programming and into technical management.

On the other hand, the fundamentals in the industry are much more solid today than they were five years ago - standards that were still in flux half a decade ago have been field tested and either are in heavy use or aren't standards any more. The open source/open standard movement, which was considered radical in 1999, has in fact become the dominant means of writing much of the software in use today, contributing to a community model that again favors the developer over the vendor or the customer (and proprietary models are being increasingly seen as backwards or dinosaurs, much to the distress of those same vendors). Lightweight programming languages and agile metholodologies are replacing the over-engineered mega-project and twenty thousand class programming frameworks, turning programming into a cooperative venture where no one person or agency controls the whole network.

It's a good time to be a programmer.

Where do you thing the job market is going for programmers and IT professionals?