The Coming Tech Labor Crunch

by Kurt Cagle

This particular essay has been sitting as a draft on my laptop for some time. I've been intrigued lately by the bigger issues facing the IT market, and especially of the people who work in it. The news of late seems to be ominously similar in tone no matter where you go, that even in the face of softening employment elsewhere companies can't fill programmers fast enough. I don't think this is peculiar to this particular moment, but may in fact shadow a significant shift in the relationship of the IT workforce to the companies who employs it. My apologies for the length, but there's a lot to cover ...

An interesting paradox is occurring right now in the IT market, one that I predicted back in 2002 when jobs for programmers were as scarce as new IPOs about then. In a nutshell, even as the general economy is beginning to suffer from significant doldrums, the demand for skilled IT professionals has seldom been higher ... and many companies are having to shelve new production because they can't find the programmers to build their applications ... globally. From the United States and Canada to England, Germany, India, China, Japan, Australia ... there just don't seem to be enough skilled programmers to fill more than about fifty percent of all IT-related jobs.

9 Comments

len
2007-06-27 06:32:04
At least we have finally entered the golden age of drag and drop programming which shifts the talent burden from bit-twiddlery to requirements analysis and effective UI design. With the new generation of application-specific frameworks (the layer ABOVE say .Net), we'll have to invent a new term because this isn't programming as much as it is component assembly and "script kiddie" seems unkind.


Does that change the economic picture? It means tool and license costs figure more in those projects that don't rely on open source, that mixed open and closed source systems (say ASP and PostGreSQL) are a bigger part of the mix, and that programming as a profession becomes even more of a tradecraft job with less status and possibly less women in the trade.


It also means being multilingual at least to the ability to work well in a shop where accents and skills cause every third word to be lost, and it means increased emphasis on the documentation process.


Somehow we get through these crunching changes but it is ever more evident that tradecraft level programming will become a supporting skill that many professions will require in addition to the principal skill set. The number of to-the-metal programmers can wax and wane and that will slow down the framework churn (do we need another word processor?) which might be a good thing.

ASB
2007-07-01 08:38:41
Very good read. A little bit of over-generalization of developers, but all-in-all, this does address many of the issues that exist out there.


More importantly, however, is that the landscape for Network Engineers, Database Administrators and Systems Administrators is even more tight than for Developers, and these folks represent a critical part of the IT chain as well...


It's a good time to be in IT, and many of the organizations that have resorted to off-shoring simply for cost reasons will find that the costs have been rearranged, not eliminated.

Kurt Cagle
2007-07-01 09:50:29
ASB,


I'm familiar with the developer landscape, somewhat less so for the administrative side of things, though I would agree with you that if anything this sector is likely to be even more squeezed than the dev side will be. IT admins tend to be as much as a decade older, and usually have received considerable advanced training. Unlike the developer workforce, IT admins are much closer to retirement, and if the age of offshoring has affected developers, it has also meant that many organizations have chosen to have only one or at best two admins in a given slot, making them vulnerable to both retirement and poaching.


I've also wondered more than once whether the graphical approach that Microsoft has taken in its GUI admin tools may be backfiring. Such tools make it easier to administer sites, but at the cost typically of less understanding of what is in fact happening in the background. Admin positions especially tend to be trained on the job, and I suspect that this lack of understanding will propagate to the newer admins as a consequence.


-- Kurt

Kurt Cagle
2007-07-01 10:03:01
Len,


You know, it's funny, but I've been waiting for drag and drop programming for years, expecting with each subsequent generation of dev tools that we'll finally drop below the threshold at which programming becomes easy enough for the non-programmer.


Curiously, it never quite seems to come. JavaScript or Ruby are arguably simpler than Java or C++, but they're being used increasingly to handle distributed computing where the efficiencies of formal type declaration get lost and the costs of such become quite real. Indeed, what I find intriguing is that JavaScript as a language is evolving into something much more evocative of Lisp or Scheme than the fairly basic language it started out from.


I think programming has become more component oriented, which means that you can break down programming more into web developer vs. web designer roles (or app developer or app designer roles) but I don't necessarily think we're even close yet to the drag and drop level of programming, and I'm becoming increasingly convinced that we won't ever be. Programming is the art of dealing with complexity, and all that componentization does is expand the baseline of what can be done.

len
2007-07-02 06:26:09
Can be done faster and cheaper.


Drag and drop is pretty common in the ASP.NET world. It doesn't mean one doesn't program; just that the need for lots of to-the-metal programmers has been reduced significantly. I'm not saying one doesn't need component builders, just fewer of them. The size of a site that can be built and maintained by a handful of skilled and skill sets continues to increase without significant increases in cost.


I think the crunch is only a small crack to the back such as the teacher gives to the student to keep them from sleeping during meditation and to invite enlightenment. Otherwise the difference between component building and assembling components is a matter of where one is on the production line. I don't seen any evidence that we will run out of workers.

Kurt Cagle
2007-07-02 08:03:38
Len,


I'm not completely disagreeing with you on this - I think that componentization is significantly streamlining production, though given the number of software projects that continue to fail compared to the overall number, I'm not really sure that it's really making that much difference. Certainly, one of the key effects that componentization (and standardization, which is just componentization of process) has is that it does, as you say, shift software production towards a different model, though I'm not sure that the assembly line is the right metaphor here.


I think its worthwhile to differentiate here between the big software project and the small one. Componentization makes it possible for small software projects to be carried out by fewer people, but those people generally serve more to determine which components to use and then perform the necessary integration. While programmers may (and likely do) specialize, they typically are still experts of the pieces in their domain.


Admittedly, if you talk about a modern assembly line, I'd say the analogy is closer, because in those cases most repetitive operations have been automated, and the occasional person exists primarily as a decision maker, as a designer (which is a decision maker of a different sort), or as a maintenance worker to keep the machines humming. Significantly, all three of these sets of skills require a high degree of training and education, and all three represent a fairly significant degree of autonomy.


I also suspect that the H1B defeat in the Senate may very well end up hurting the software companies, though not IT in general. I do not buy the contention of the BSA or other software company alliances that the H1B visas are truly necessary - they represent cheap labor for these companies, who are now going to be forced to pay more for the home grown variety, but I suspect that global demand for software talent and services will only grow with time (because customization is, ultimately, the metier of programmers). Software companies exist to create "business components" - software that fulfills a certain requirement broadly enough that a large number of companies (or individuals) will buy that software.


Customization is a counterveiling force to commoditization - software companies want to create commodities, because they make less money when they sell to fewer customers (especially in the face of rising labor prices). Customization, on the other hand, is what the customers want - applications that are written specifically to their needs and wants, and while price is certainly a factor, it's not the only one. Open source works on a purely economic level because it makes the cost of the components cheap to non-existent, which in turn means that the primary costs become the integrational labor (the customization labor) - though it also has the effect of pushing down the price of proprietary components.


Note additionally that as the components themselves become more complex, the amount of work necessary to integrate those components to the specific needs of the company also increase. The complexity never completely disappears, because that complexity is driven largely by the customization aspect of the equation.


Some programmers will be needed to build the components in the face of changing technology elsewhere; other programmers will be needed to manage the complexity of integrating those components, to handle the customization. I don't care whether your language is C++ or .NET or Java or ___fill-in-the-blank___, that much will remain true. I think as the number of components continue to rise and the capabilities of the components themselves evolve, the focus will shift increasingly to the second group, but I'm not sure their job is necessary any easier, it just works at a larger scope - the requirements gathering and analysis phases, distributed computational systems, and so forth that you mentioned in your first comment. Those are engineering tasks, not manufacturing ones.

Kurt Cagle
2007-07-02 08:52:57
I love your comments, by the way - they tend to focus the mind wonderfully.


I did want to dissect one of your statements in greater detail:


I think the crunch is only a small crack to the back such as the teacher gives to the student to keep them from sleeping during meditation and to invite enlightenment. Otherwise the difference between component building and assembling components is a matter of where one is on the production line. I don't seen any evidence that we will run out of workers.


I'm not a big fan of large corporations, in general. They tend to resemble feudal fiefdoms far too closely for me to take a lot of comfort in their existence. Thus when I hear the talk about labor shortages by companies, I usually take a look at where they are getting there work done first, and by whom.


I do tend to believe that the current crunch is due in part to the dawning realization on the part of the hiring agencies at these corporations that global demand for engineering talent is on the rise, and their ability to bring in lower priced talent from outside is diminishing.


Moreover, demand tends to beget supply, though I suspect that, just as in the case of the oil companies not building up their infrastructure at times where demand is low (because they spent money on fancy buildings, extravagant parties, etc. when the money was there), their ability to move people through the hiring pipeline is currently fairly impaired.


Will the pendulum swing the other way? Certainly, though not for some years yet. There was a huge glut of people that entered IT in the mid-1990s, most driven by IPO madness, just as there was a huge glut of people that entered into real estate services in the early part of this decade. IT had a fairly significant shaking out during the tech winter (just as it's probably not a good time to be in mortgage banking right now). This glut coincided with the rise of the Internet, the rise of component oriented software development, the dramatic increase in the number of powerful PCs, and not coincidentally the entry of the GenXers into the marketplace.


We're now just off the peak of that population wave, heading down. There's a minor peak - the grandchildren of the boomers, on its way in about seven years, and the children of the GenXers, the Millenial babies, are now in second grade (my youngest was born in 2000, so she's at the very forefront of that next generation), so they'll enter the workforce in significant numbers in 2020.


Significantly, both of these peaks are actually smaller than the boomers (the millennial generation is significant in that it occurred at a time when the birth rate fell below the replacement rate), though because they overlap somewhat, you'll actually have two pulses spaced about five years apart with a plateau (and maybe a minor secondary peak) in between them. Thus, things ought to look pretty good for programmers in general for the next five to seven years on demographics alone.


Of course, by then, the interesting areas will likely be in bioinformatics and biocybernetics, nanotech derivative efforts in materials science, energy systems, robotics and aerospace, and pure information sciences will look positively primitive in comparison. If I was graduating from college now, I'd not be interested in information technologies, unless it was a minor to some other areas such as biology or space systems. You will still need programmers, of course, but the skilled engineers will be cross discipline, and the business programmer will look positively antiquated, which needs to be taken into account.


len
2007-07-03 06:19:33
We're not that far apart on this, but some other metrics.


1. Note Spolsky's article on hiring programmers that Elliotte pointed out. Note the need for non-programming skills beyond the need for ATL knowledge. English proficiency rates highly. These non-tech skills are becoming another limiting quality but also are a force pushing us to invent new ways to communicate. As the tech lead for a team that includes two non-native English speakers, it makes a difference but not one I can't overcome.


2. Note the articles starting with T. Bray's reference to the DevChix complaints about leering bullying males. Note the comments about declining numbers of women in the business. There are social status forces at work here too.


But componentization is not just a force working at the lower levels such as the frameworks, eg;, .Net. Microsoft and others are componentizing above that level in software fitted to a particular business so even the jobs of customization you refer to are being pushed up a level where it is becoming more of a form/fit/function analysis plus scripting. How far that will go depends a lot on the analysis and design of the classes. I'm fascinated by the ASP 2.0 changes to the framework and where the difficulty shifted. In the goodOldeDaze I could open the HTML and find everything I needed. Now I have to wade through layers of components designed to hide the complexity of the server from me just to figure out why a certain directory (APP_DATA) can't have a jpg in it. It turns out I had to find a blog commenting on that because MS never tells one this in a place one can find reasonably quickly. So a software design feature wastes a half day of production because a) it is counter intuitive to experience and b) no one bothered to document it transparently.


So even with the componentization, we lose time in complexity of the hidden couplers, technical, social, skills, and competence. Costs are hard to track but it is clearly evident that for the scale of software projects we can deploy, we need fewer workers. Unless of course one is taken over by a distant equity firm that demands absolute control AND cost cutting, thus increasing the numbers of bean counters (Process Design: One Person == One Fact. NeXt), slowing down production and ultimately failing to create value with a company except at the point of resale. Weird but it seems there is no lack of suckers for voodoo economics in the world today (Yes! Matilda Build Virtually And Cash Out Real Before the Pyramid Collapses!).


Does the economic picture change? We rely on computers in ways we never have in the past and this sea change is of such recent emergence that we actually don't know what will result based on experience other than as I said, things speed up. Will we develop more efficient communication, more efficient production based on more efficient tools, better social skills or just learn to accept that as the status of being a computer professional drops, some will go elsewhere to find status. If so, then salaries rise and perhaps in social terms go back to being what they were in the 70s relative to today's costs of living. As that happens, the status point shifts and the flow tips back in the other direction. We've seen this before in America. The Space Program cranked out a lot of engineers and practically created the modern software market. Then in 1971, it began to die at an incredible rate. Then it picked up when a combination of technology and economic forces put the money back into the market.


You enjoy computer science. For others, it is just a job, but when the big numbers are considered, only money matters. So it seems. What interests me more is where these tidal forces will impact software quality just at the point that the complexity can only be managed by the computers themselves and we are so reliant that we can't manage a work stoppage.

Kurt Cagle
2007-07-03 14:58:00
Again, I agree with you on just about everything you've said. Comments inline:


<snip>
Note the need for non-programming skills beyond the need for ATL knowledge. English proficiency rates highly.</snip>
One of the reasons I suspect that the Indian outsource market has outperformed the Chinese ... India has 100+ years of British colonialism to insure that English is still largely a near-native language for their technical workforce. More people worldwide speak Chinese than English, but most of those people are concentrated in China, whereas English is a second language almost everywhere its not a primary one.


<snip>Note the comments about declining numbers of women in the business. There are social status forces at work here too.</snip>
Absolutely. Programming has long been dominated by young (not terribly well socialized) males, as much for socioeconomic reasons as anything, and young men are notorious in using their work environment to establish their social position (my code's cleaner|more compact|more elegant|whatever than your code). That's a topic for another time, however.



<snip>But componentization is not just a force working at the lower levels such as the frameworks, eg;, .Net. Microsoft and others are componentizing above that level in software fitted to a particular business so even the jobs of customization you refer to are being pushed up a level where it is becoming more of a form/fit/function analysis plus scripting.</snip>
Yes, though this comes with the danger of biasing the business logic towards the Microsoft way of doing things. Still, t'is a good point.


<snip>In the goodOldeDaze I could open the HTML and find everything I needed. Now I have to wade through layers of components designed to hide the complexity of the server from me just to figure out why a certain directory (APP_DATA) can't have a jpg in it (...) So a software design feature wastes a half day of production because a) it is counter intuitive to experience and b) no one bothered to document it transparently.</snip>


Componentization is hard ... even after more than twenty years of doing it myself, I find the process of task decomposition to be a remarkably difficult and treacherous discipline, very much analogous to the designing of good schemas. The irony here is that the process of creating such components is in some respect becoming easier than the process of designing what the components should do in the first place (namespaces!! Aaaargghh!!!)


<snip>So even with the componentization, we lose time in complexity of the hidden couplers, technical, social, skills, and competence. Costs are hard to track but it is clearly evident that for the scale of software projects we can deploy, we need fewer workers.</snip>


Granted. However, we have fewer workers. The question that I think will shape the software development universe is whether the declining pool of technical competent workers is in fact still larger than the declining pool of programming and IT slots. If the problem is simple economics (software producers are not willing to pay for local talent so long as less expensive remote talent exists) then the solution will be market driven - if the value proposition of the product exceeds the cost of hiring the programmer by some currently unknown factor, then salaries will go up until that is no longer true. If the problem, on the other hand, is fewer trained programmers and a collapsed training pipeline, then production will slow until stasis is reached.


<snip>Unless of course one is taken over by a distant equity firm that demands absolute control AND cost cutting, thus increasing the numbers of bean counters.</snip>


For every project that has failed because of insufficient software, I'd wager that ten have failed due to building too many layers of bureaucracy, each bureaucrat of whom justified their existence by imposing more and more oversight on software projects. Some oversight is necessary, a great deal of it is highly counterproductive



<snip>You enjoy computer science. For others, it is just a job, but when the big numbers are considered, only money matters. So it seems. What interests me more is where these tidal forces will impact software quality just at the point that the complexity can only be managed by the computers themselves and we are so reliant that we can't manage a work stoppage.</snip>


It's coming soon. The biggest constraint on it happening now is that we require a level of accountability and legibility in our software that computers generally cannot produce (computers are poor at creating abstractions at this stage, which is one of the keys to complexity management). As applications become more complex (thinking in the distributed network realm here) I suspect we're reaching a point where the level of computatonal complexity is so high that we may have no choice but to let computers work at their own level of abstraction and constrain the results, especially in the realm of distributed computing.


I'm not sure that will signal the end of programming as a human discipline, but it will certainly alter the balance of power between man and machine in ways that aren't even remotely known.