[FoRK] Software's Dirty Little Secret ZapFlash

Stephen D. Williams sdw at lig.net
Tue Sep 7 06:55:58 PDT 2004


This sentiment is the culmination of a long history of experience 
showing that it is usually better to build very flexible meta-solutions:

"Here is the critical point: virtually all software created up to this 
point is fundamentally of low quality, since it's all built to meet 
pre-defined requirements, and those pre-defined requirements are rarely 
the true requirements for the software. The only way software quality 
will ever improve is for software to become high tech. The key insight 
here is that today's common understanding of what software quality means 
is lacking, because those definitions apply to purpose-built software. 
We must take a step back, so that we can judge software quality based 
upon its flexibility and agility, rather than how few defects it has, or 
how well-documented the process of creating it might be."

http://www.zapthink.com/report.html?id=ZAPFLASH-09012004
Software's Dirty Little Secret ZapFlash

By Jason Bloomberg
Document ID: ZAPFLASH-09012004 | Document Type: ZapFlash

Since the days of Bletchley Park's Colossus, the world's first 
programmable computer, the creation of software has been separate from 
the production of the machines that run that software. For the first 
time, a machine was programmable instead of purpose-built. In other 
words, people did not determine the specific requirements for computers 
before they built them. Instead, programmers wrote software later to 
meet those requirements. No longer did hardware have purpose-built 
requirements; instead, people had the meta-requirement of 
programmability -- the requirement that they be flexible enough to meet 
as-yet undefined requirements at some point in the future.

Today, we use the term high technology to refer in part to programmable 
machines -- whether they be computers or any number of other devices 
with the computer chips that enable them to meet the meta-requirement of 
programmability. While computers are high tech, the surprising fact is 
that software itself is almost always low tech, because software is 
generally built for specific purposes. Programmers rarely, if ever, 
build software applications for the sort of general purpose uses that a 
computer is suitable for. We have done an extraordinary job of building 
computer chips that can run everything from weather forecasters to video 
games -- but the software itself is only good for one kind of activity 
or the other.

Software has been low-tech for so long that we're all used to it that 
way. After all, software production is still a craft-based enterprise, 
with experienced practitioners turning out custom software in small 
teams. Just as when cobblers produced shoes and seamstresses fashioned 
clothing, today's software is either entirely custom or appallingly 
ill-fitting. Custom software is enormously expensive, often requiring 
teams of developers, and only meets the particular needs of the moment. 
Meanwhile, commercial off-the-shelf (COTS) software is rarely flexible 
enough to meet all the needs of its users, requiring users to work 
within the constraints of the software, rather than the other way 
around. In any case, regardless of whether it's custom or COTS, today's 
software is of appallingly low quality -- and that's software's dirty 
little secret.

The challenge facing us today is whether new, Service-oriented (SO) 
approaches to software will help to resolve this quality issue. Clearly, 
there is no guarantee that implementing a Service-Oriented Architecture 
(SOA) will improve quality -- people can still implement SOAs poorly, 
after all. The real questions are first, whether SOA done right will 
raise the level of software quality overall, and second, what will it 
take to do SOA right?

Revisiting Software Quality
Bring up the word "quality" in a business context, and people think of 
one of two things -- either lowering the defect rate as approaches like 
Six Sigma and ISO 9001 advocate, or formalizing the manufacturing and 
development processes, as Total Quality Management espouses. Both 
approaches can apply to the production of software, as well as a wide 
range of manufactured goods. Developers and testers are the most 
familiar with the bug hunt and fix approach to software quality, and 
engineers and IT management within the enterprise often look to the 
Capability Maturity Model as the established approach to formalizing the 
processes of creating and managing software.

Unfortunately, both perspectives on software quality fall within the 
low-tech, craft approach to software production we follow today. Neither 
perspective addresses the more important and fundamental definition of 
software quality, which is how well software meets the requirements set 
out for it. The unfortunate fact is that when people build software to 
meet specific, pre-defined requirements, it almost always falls short of 
meeting the true requirements for that software, primarily because the 
actual requirements for software are typically in a state of constant 
flux. In other words, our true requirements for software is for it to be 
high tech -- that is, for software to be flexible enough to meet as yet 
undefined requirements at some point in the future, just as high-tech 
hardware does today.

Here is the critical point: virtually all software created up to this 
point is fundamentally of low quality, since it's all built to meet 
pre-defined requirements, and those pre-defined requirements are rarely 
the true requirements for the software. The only way software quality 
will ever improve is for software to become high tech. The key insight 
here is that today's common understanding of what software quality means 
is lacking, because those definitions apply to purpose-built software. 
We must take a step back, so that we can judge software quality based 
upon its flexibility and agility, rather than how few defects it has, or 
how well-documented the process of creating it might be.

Will Service Orientation Lead to Higher Software Quality?
Business agility is the most fundamental benefit of SOA, where business 
agility means being able to meet as-yet undefined, changing business 
requirements, and leverage such change for competitive advantage. 
Business agility is thus the linchpin of high-tech software. Businesses 
require software that responds quickly and efficiently to a changing 
business environment, and SOAs promise this level of agility.

However, it is crucial that we temper this long-term vision with a 
short-term reality check. It will take many years to achieve the 
high-tech vision of agile software, and there will be many roadblocks 
along the way. It won't be easy, by any means; such a transition will 
require substantial change, both in technology and human behavior. 
Nevertheless, today's SO approaches can provide a good measure of 
business agility.

What is it about SO software approaches, then, that provide such 
business agility? What technology advances are we making today that are 
precursors of the vision of high-tech software? The vision of high-tech 
software paints a long-term goal, and the following achievements 
illustrate the progress we're already making:

    * Metadata-driven functionality. An increasing number of products 
are moving business logic from the code into the metadata, as metadata 
management becomes increasingly important to building and running agile 
SOAs.

    * Declarative "programming." Instead of true programming, business 
users are increasingly able to determine the functionality of the 
software they use through configuration, rather than compilation. Such 
declarative approaches to determining software functionality move the 
power over business logic into the hands of business users.

    * The maturation of the discipline of software architecture. Today, 
architects learn most of their craft on the job. As the discipline of 
architecture matures, what it means to be an architect will stabilize, 
enabling companies to plan, design, and produce high-tech software in an 
industry standard manner. Enterprise architects will increasingly be 
able to architect their enterprises with business agility in mind.

    * The rise of standards. We are in the early chapters of what will 
likely be the long story of IT standards, but we're on our way 
nevertheless. Standards are one important element to moving beyond craft 
production of software. Just as standardization in manufacturing led to 
the industrialization of production, standardization in software will 
lead to software that is built to be agile.

    * The Service Orientation abstraction layer. Software grows by the 
addition of abstraction layers that simplify complexity. SO abstracts 
application functionality, giving businesses unprecedented control and 
flexibility over the software they use. Such an abstraction layer is 
thus a necessary first step to building high-tech software.

The ZapThink Take
ZapThink believes that Service Orientation is one of the key mechanisms 
for leaving the world of low-tech, low-quality software behind, and 
achieving the goals of high-tech software. Clearly, much work remains to 
be done, and SO alone does not guarantee quality. However, the trend 
toward high-tech software is apparent in the progress vendors and IT 
end-users are making today. ZapThink predicts that we will see a gradual 
decline in custom programming in the enterprise, as users become able to 
configure business applications to meet changing requirements. At the 
same time, companies will need fewer IT developers, as the software they 
purchase becomes more agile. After all, software vendors should be the 
companies that produce the software, while enterprises in other lines of 
business should concentrate on what they do best instead.

Finally, ZapThink predicts the end of craft software, as standards 
mature and vendors learn how to build high-tech software. Software 
production will achieve benefits analogous to those that mass production 
achieved -- consistent quality at substantially lower cost, leading to a 
new era of software quality. As the industry matures and becomes 
high-tech, software will no longer be judged on how well it can meet 
today's requirements, but rather how flexibly it can meet tomorrow's 
challenges. Will Service Orientation solve the software industry's 
"dirty little secret" of appalling quality? Only time will tell, of 
course. There's no question, however, that if vendors become able to 
deliver software that can truly meet as-yet undefined business 
requirements, then enterprises will demand such software.

sdw

-- 
swilliams at hpti.com http://www.hpti.com Per: sdw at lig.net http://sdw.st
Stephen D. Williams 703-724-0118W 703-995-0407Fax 20147-4622 AIM: sdw



More information about the FoRK mailing list