From the essay No Silver Bullet written in 1986. Many of the principles described in this essay apply to the last 60+ years of Software development from the days of the ENIAC. For example the concepts for constructing software are based on a comment from 1958!
Warning, What follows is a fairly long discussion of things I have learned while developing software for the last 30 years. I started writing software when I was a senior in high school in 1978. I am still writing software and designing systems today. Most of this blog entry is based on an essay written by Fredrick P. Brooks in 1986. He also wrote the The Mythical Month in 1975 and it is still a great read. Believe me most of what Brooks says in the Mythical Man Month applied to my first 8 years of programming as much as it applies today. Brooks original Mythical Man Month is based on developing an OS at IBM in the 1960's but ignore that and look at the principals. There have been other authors who have expanded on what Brook's had to say in the original 15 chapters of this book but Brooks said it first. Why do you think the primary law of project management is called Brooks's Law?
Two additional essays from Brooks, No Silver Bullet and Silver Bullet Re-fired were added to the 25th Anniversary addition of the Mythical Man Month. No silver Bullet was written in 1986 (over 20 years ago and 8 years after I started writing software). This is the essay I want to talk about now. All quotes from No Silver Bullet are in Bold Itallics.,
...the anomaly is not that software progress is so slow, but that computer hardware progress is so fast.
Essential Properties of Software
The essence of a software entity is a construct of interlocking concepts,
I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.
If this is true, building software will always be hard. There is inherently no silver bullet.
Software is also essentially complex. When a software architect models a system they must take this into account so that the Software Models do not obscure this essential property.
The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence. For three centuries, mathematics and the physical sciences made great strides by constructing simplified models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.
So what are some of the essential properties of software?
First software must conform to business processes and institutions. This adds to the complexity of software because the requirements of conformity are based on Business processes that do not always make sense or are not as clearly defined as the principles of other disciplines (such as physics). So much of the complexity a software designer faces is a arbitrary complexity...
Much of the complexity that he must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform.
...in all cases, much complexity comes from conformation to other interfaces; this complexity cannot be simplified out by any redesign of the software alone.
Many large Enterprise Systems are just now beginning to try to tackle this. Business Process reengineering targets the business process itself, not the software that implements the business process. Essentiality though this is just an attempt to do better requirements analysis and help to reduce the arbitrary complexity of software. This complexity is not addressed by re-factoring code. It is a essential property of the software.
Another essential property of successful software is that it changes:
All successful software gets changed. Two processes are at work. First, as a software product is found to be useful, people try it in new cases at the edge of or beyond the original domain. The pressures for extended function come chiefly from users who like the basic function and invent new uses for it.
Brooks goes onto describe another essential property of software. It is invisible
As soon as we attempt to diagram software structure, we find it to constitute not one, but several, general directed graphs superimposed one upon another. The several graphs may represent the flow of control, the flow of data, patterns of dependency, time sequence, name-space relationships.
Many Enterprise Architecture tools demonstrate this truth. In fact the entire premise of UML demonstrates this. We have Class Diagrams, Component Diagrams, Sequence diagrams, etc.. all designed to help us try to visualize software. These all succeed and fail to some degree because invisibility is a essential property of software.
Accidental Properties of Software
Brooks goes on to describe technology advances that attack the accidental part of software development
If we examine the three steps in software technology development that have been most fruitful in the past, we discover that each attacked a different major difficulty in building software, but that those difficulties have been accidental, not essential, difficulties.
The three that he describes include:
- High Level Languages. Now this would include java and C#.
- Time Sharing. Now this would be the entire concept or personal computers, laptops, PDA's etc... essentially ways to get the answer immediately.
- Unified frameworks. Here he talks about UNIX, but you can also include in this the .NET framework and the Java class libraries.
I find this section really amusing
Operating systems, loudly decried in the 1960's for their memory and cycle costs, have proved to be an excellent form in which to use some of the MIPS and cheap memory bytes of the past hardware surge.
How often do you hear the cry that windows is bloated and overgrown but UNIX is lean and mean. This battle has been going on for over 45 years with no end in site. For all of you who continue in these debates know that the next 45 years will probably continue in the same way. We will simply substitute a different OS name and battle on...
Hopes for the Silver Bullet
This is a interesting section. See how many of these concepts seem familiar
- High Level Languages advances. We have all blogged about the neat new features in C#. So 25 years ago it was the neat new features of Ada...
Ada not only reflects evolutionary improvements in language concepts, but indeed embodies features to encourage modern design and modularization. Perhaps the Ada philosophy is more of an advance than the Ada language, for it is the philosophy of modularization, of abstract data types, of hierarchical structuring.
- Object Oriented Design. Have you seen an order of magnitude change by adopting C# or .Net in general?
An order-of-magnitude gain can be made by object-oriented programming only if the unnecessary type-specification underbrush still in our programming language is itself nine-tenths of the work involved in designing a program product. I doubt it.
- Artificial Intelligence. It is interesting that Brooks breaks out this section into two separate sections. His comments on speech recognition and image recognition are very true today. One of the essential problems in speech recognition is not recording what you say, but understanding what you mean.
The techniques used for speech recognition seem to have little in common with those used for image recognition, and both are different from those used in expert systems. I have a hard time seeing how image recognition, for example, will make any appreciable difference in programming practice. The same problem is true of speech recognition. The hard thing about building software is deciding what one wants to say, not saying it. No facilitation of expression can give more than marginal gains.
- Expert systems. The most advanced part of the artificial intelligence art, and the most widely applied, is the technology for building expert systems. The next section discusses why mentoring is so important for Architects and Senior Software Developers today since we have yet to build an expert system to do this for us.
The essential prerequisite for building an expert system is to have an expert. The most powerful contribution by expert systems will surely be to put at the service of the inexperienced programmer the experience and accumulated wisdom of the best programmers. This is no small contribution. The gap between the best software engineering practice and the average practice is very wide_perhaps wider than in any other engineering discipline.
- Automatic Programming. What we could term as code generation today still has not achieved the goal of being able to state the problem and have the computer just solve it for you. Business Process Execution Languages are the latest attempt at this. But I still agree with Brooks when he says this about code generators
It is hard to see how such techniques generalize to the wider world of the ordinary software system, where cases with such neat properties are the exception. It is hard even to imagine how this breakthrough in generalization could occur.
- Graphical Programming. Brooks has some interesting comments on software diagrams (Read UML and generating code from UML diagrams). I believe many of these apply just as easily to software developed today. How man times have you generated the class diagram from the existing code instead of the other way around?
Nothing even convincing, much less exciting, has yet emerged from such efforts. I am persuaded that nothing will.
Brooks goes on to say:
In the pitiful, multipage, connection-boxed form to which the flowchart has today been elaborated, it has proved to be useless as a design tool--programmers draw flowcharts after, not before, writing the programs they describe
- Program Verification. In the simplest form this is a way to have the tests match the requirements (as David put it). In a broader form it is to verify program program correction overall. To many times we take this as Unit testing and think if we have 100% code coverage on our Unit Tests we are golden. Scott Hanselman has a two great podcasts that address this myth here and here. The root of this is as Brooks says,
More seriously, even perfect program verification can only establish that a program meets its specification. The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification.
- Environments and tools. Surely integrated development environments are the answer to better programs, right? Although I would not want to give up Visual Studio, the interesting thing is Brooks saw those developments coming 20 years ago and still did not see them as that important even today integrated debuggers are now decried as less useful then unit tests by agile developers.
One's instinctive reaction is that the big-payoff problems--hierarchical file systems, uniform file formats to make possible uniform program interfaces, and generalized tools--were the first attacked, and have been solved. Language-specific smart editors are developments not yet widely used in practice, but the most they promise is freedom from syntactic errors and simple semantic errors.
- So what about the latest and greatest in workstations. That new laptop with 4 gig of memory, solid state disks, fast networking, 64 bit development environments...
What gains are to be expected for the software art from the certain and rapid increase in the power and memory capacity of the individual workstation? Well, how many MIPS can one use fruitfully? The composition and editing of programs and documents is fully supported by today's speeds. Compiling could stand a boost, but a factor of 10 in machine speed would surely leave thinktime the dominant activity in the programmer's day. Indeed, it appears to be so now.
More powerful workstations we surely welcome. Magical enhancements from them we cannot expect.
Attacks on the Essence of the Problem
So in the end we are left with an early form of the discussion on Team Velocity
If, as I believe, the conceptual components of the task are now taking most of the time, then no amount of activity on the task components that are merely the expression of the concepts can give large productivity gains.
Hence we must consider those attacks that address the essence of the software problem, the formulation of these complex conceptual structures. Fortunately, some of these attacks are very promising.
Here is a shocker for how to swiftly deploy systems.
Buy versus build. The most radical possible solution for constructing software is not to construct it at all.
...Any such product is cheaper to buy than to build afresh. Even at a cost of one hundred thousand dollars, a purchased piece of software is costing only about as much as one programmer year. And delivery is immediate!
The development of the mass market is, I believe, the most profound long-run trend in software engineering. The cost of software has always been development cost, not replication cost. Sharing that cost among even a few users radically cuts the per-user cost. Another way of looking at it is that the use of n copies of a software system effectively multiplies the productivity of its developers by n. That is an enhancement of the productivity of the discipline and of the nation.
And finally a initial discussion of the Knowledge worker of today
I believe the single most powerful software-productivity strategy for many organizations today is to equip the computer-naive intellectual workers who are on the firing line with personal computers and good generalized writing, drawing, file, and spreadsheet programs and then to turn them loose.
How about a early discussion on Iterative development. Software construction concepts based on a comment in 1958?
The hardest single part of building a software system is deciding precisely what to build.
...Therefore, the most important function that the software builder performs for the client is the iterative extraction and refinement of the product requirements. For the truth is, the client does not know what he wants. The client usually does not know what questions must be answered, and he has almost never thought of the problem in the detail necessary for specification.
...Much of present-day software-acquisition procedure rests upon the assumption that one can specify a satisfactory system in advance, get bids for its construction, have it built, and install it. I think this assumption is fundamentally wrong, and that many software-acquisition problems spring from that fallacy. Hence, they cannot be fixed without fundamental revision--revision that provides for iterative development and specification of prototypes and products.
...Incremental development--grow, don't build, software. I still remember the jolt I felt in 1958 when I first heard a friend talk about building a program, as opposed to writing one. In a flash he broadened my whole view of the software process. The metaphor shift was powerful, and accurate. Today we understand how like other building processes the construction of software is, and we freely use other elements of the metaphor, such as specifications, assembly of components, and scaffolding.
So instead we should grow software systems organically
Let us turn nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and self renewing. The secret is that it is grown, not built.
So it must be with our software-systems. Some years ago Harlan Mills proposed that any software system should be grown by incremental development. [10] That is, the system should first be made to run, even if it does nothing useful except call the proper set of dummy subprograms. Then, bit by bit, it should be fleshed out, with the subprograms in turn being developed--into actions or calls to empty stubs in the level below.
I have seen most dramatic results since I began urging this technique on the project builders in my Software Engineering Laboratory class. Nothing in the past decade has so radically changed my own practice, or its effectiveness. The approach necessitates top-down design, for it is a top-down growing of the software. It allows easy backtracking. It lends itself to early prototypes. Each added function and new provision for more complex data or circumstances grows organically out of what is already there.
The morale effects are startling. Enthusiasm jumps when there is a running system, even a simple one. Efforts redouble when the first picture from a new graphics software system appears on the screen, even if it is only a rectangle. One always has, at every stage in the process, a working system. I find that teams can grow much more complex entities in four months than they can build.
As someone who has been developing software for over 30 years and never gone the management track, the next comment is very telling. I became a Fellow at Micron in less then 5 years and still did not see the dedication to the technical team he mentions here.
Hence, although I strongly support the technology-transfer and curriculum development efforts now under way, I think the most important single effort we can mount is to develop ways to grow great designers.
No software organization can ignore this challenge. Good managers, scarce though they be, are no scarcer than good designers. Great designers and great managers are both very rare. Most organizations spend considerable effort in finding and cultivating the management prospects; I know of none that spends equal effort in finding and developing the great designers upon whom the technical excellence of the products will ultimately depend.
My first proposal is that each software organization must determine and proclaim that great designers are as important to its success as great managers are, and that they can be expected to be similarly nurtured and rewarded. Not only salary, but the perquisites of recognition--office size, furnishings, personal technical equipment, travel funds, staff support--must be fully equivalent.
Conclusion
So in the last 30 years of software development we have only begun to addresses the essence of software development.
- Software is Complex
- Successful Software is ever changing
- Software is invisible
So as you continue to develop software look at what you are doing. Attack the essence of the problem, not the accident.
- Buy vs. Build where appropriate
- Empower your users to allow them to solve problems themselves
- Use Iterative Development Processes
- Build strong technical teams and designers
.gif)
