I've
been following some of the writings and actions of the Massachusetts
State Executive Office for Administration and Finance as it deals with
its Information Technology needs. It was through listening to Secretary
Kriss and reading the writings he and other Massachusetts government
officials have produced that I've come to look at software development
from a whole new perspective. This essay tries to present that
perspective and examine some of its implications.
Many things in society are long-term
In
many human endeavors, we create infrastructure to support our lives
which we then rely upon for a long period of time. We have always built
shelter. Throughout most of recorded history, building or buying a home
was a major starting step to growing up. This building would be
maintained and used after that, often for the remainder of the builder's
life span and in many instances beyond. Components would be replaced as
they wore out, and the design often took the wear and tear of normal
living into account. As needs changed, the house might be modified. In
general, though, you thought of a house as having changes measured in
decades.
Likewise,
human societies also create infrastructure that are built once, then
used and trusted for a long period of time. Such infrastructure includes
roads, bridges, water and power distribution systems, sewers, seaports
and airports, and public recreational areas. These also would be used
and maintained without major modifications after they were built, often
for many decades or even centuries.
Software has been short-term
By
contrast, software has historically been built assuming that it will be
replaced in the near future (remember the Y2K problem). Most developers
observe the constant upgrading and replacement of software written
before them and follow in those footsteps with their creations. In the
early days of computer software, the software was intimately connected
to the hardware on which it ran, and as that hardware was replaced by
new, better hardware, new software was built to go with it. In the early
days, many uses of computing power were new -- they were the first
application of software to problems that were previously done manually
or not at all. The world got used to the fact that the computer version
was an alternative and the special features and cost savings were what
was special.
Today,
hardware is capable enough that software can be written that will
continue to run unmodified as hardware is changed. Computers are no
longer new alternatives to other applications -- they are the only
alternative. Despite this, old thinking and methodologies have remained.
Computers
and computer software have been viewed as being valuable for no longer
than common short-term durable goods like an automobile or sometimes
even tires. In accounting, common depreciation terms for software are 3
to 5 years; 10 at most. Contrast this to residential rental property
which is depreciated over 27.5 years and water mains and brick walls
which are depreciated over 60 years or more.
Records
Another
aspect of human society is the keeping of records. Common records kept
by governments include property ownership, citizenship and census
information, and laws. Personal records include images (such as
portraits) and birth, death, and genealogical information. General
information kept by society includes knowledge and expression, and
artifacts representative of culture. Again, the time frame for keeping
such records is measured in decades or centuries. I can go to city hall
and find out the details of ownership relating to my house going back to
when it was built in the late 1800's. "Family bible" records go back
generations. The Boston Public Library, like many city libraries, has
newspapers from over 200 years ago available on microfilm, and many from
the last 150 years in the original paper form.
Most
of these societal records have been kept on paper. When computers were
first introduced, they were an adjunct to the "real" paper records, and
paper printouts were made. Computer-readable "backups" and transaction
logs were produced and stored on removable media such as magnetic tapes,
or even paper printouts. These were usually written and then rarely
accessed, and even then accessed in a manner akin to the newspaper
stacks of a library. Only the recent, working copies of data were
actually available to the computers on an instantaneous basis. Much of
the use of computers was for "transactions", and only the totals at the
end of the time period of interest needed to be carried forward except
in rare circumstances, such as disaster recovery or audits. Switching to
a new computer system meant copying the totals and then switching the
processing of new transactions to the new system instead of the old.
When
it comes to moving ahead, most new software and hardware can only access
the most recent or most popular old data. Old manuscripts created with
old word processors, often archived on obsolete disk cartridges in
obsolete backup formats, are almost impossible to retrieve, even though
they are less than 25 years old. The companies that built the software
and hardware are often long gone and the specifications lost. (If you
are older than 30, contrast this to your own grade school compositions
saved by your parents, or letters from their parents, still readable
years later.)
Today's world and Societal Infrastructure Software
The
world is different now than it was even just a decade or two ago. In
more and more cases, there are no paper records. People expect all
information to be available at all times and for new uses, just as they
expect to drive the latest vehicle over an old bridge, or fill a new
high-tech water bottle from an old well's pump. Applications need to
have access to all of the records, not just summaries or the most
recent. Computers are involved in, or even control, all aspects of the
running society, business, and much of our lives. What were once only
bricks, pipes, and wires, now include silicon chips, disk drives, and
software. The recent acquisition and operating cost and other advantages
of computer-controlled systems over the manual, mechanical, or
electrical designs of the past century and millennia have caused this
switch.
I will call this software that forms a basis on which society and individuals build and run their lives "Societal Infrastructure Software".
This is the software that keeps our societal records, controls and
monitors our physical infrastructure (from traffic lights to generating
plants), and directly provides necessary non-physical aspects of society
such as connectivity.
We need to start thinking about software in a way more like how we think about building bridges, dams, and sewers.
What we build must last for generations without total rebuilding. This
requires new thinking and new ways of organizing development. This is
especially important for governments of all sizes as well as for
established, ongoing businesses and institutions.
There
is so much to be built and maintained. The number of applications for
software is endless and continue to grow with every advance in hardware
for sensors, actuators, communications, storage, and speed. Outages and
switchovers are very disruptive. Having every part of society need to be
upgraded on a yearly or even tri-yearly basis is not feasible. Imagine
if every traffic light and city hall record of deeds and permits needed
to be upgraded or "patched" like today's browsers or email programs.
Needing every application to have a self-sustaining company with
long-term management is not practical. How many of the software
companies of 20 years ago are still around and maintaining their
original products?
Software development culture
Traditional
software development falls into two general categories: Prepackaged and
Custom. Prepackaged software is written by Application Software
Companies (often called Independent Software Vendors, ISVs) who produce a
program and then sell the same product to multiple customers. Custom
software is written either by an independent company under contract or
by in-house developers for a specific user. Common elements may be
reused from project to project, but the overall program is unique.
Prepackaged
software has the advantage of using the leverage one gets by spreading
development costs over multiple users. Custom software has the advantage
of being able to be tuned to very specific needs and circumstances of
each user. A challenge when developing prepackaged software is
developing a product that appeals to a wide audience. A challenge when
developing custom software is to take advantage of "generic" prepackaged
components to lower development costs.
The
most successful prepackaged software applications have been those that
may be inexpensively customized to meet the needs of users by developers
with less and less computer skills, most desirably by the users
themselves, or that form a base on which other prepackaged or custom
software are built. Examples of such software are the common
"productivity" applications like word processors and spreadsheets and
"plumbing" software like operating systems, database engines, and web
servers. The developers of prepackaged software are driven by a need to
make their products appeal to today's potential users (and buyers),
usually through features that distinguish them from competition.
A
traditional prepackaged software company is organized as an ongoing
enterprise, usually with a desire and plans for growth. An initial core
of technical and product design people build the first version of the
product. Marketing and sales people are added to sell the product and
bring in revenues. Development continues and new, better versions are
produced. New revenue comes from selling to existing customers, with
each new version needing to give existing users a reason to replace the
old product. The mentality, and the resulting major investments in
corporate marketing, sales, and research activities, is focused on
obsolescence and "upgrading" -- but only upgrading to products from that
company. The potential for new customers and upgrade revenue is often a
requirement to procure initial funding.
There
are prepackaged software companies that are structured to make their
profits from services and activities separate from the actual delivery
of software code. The software itself may be available with no or little
charge, but the organization is set up so that support of various sorts
is provided by the company which has special knowledge of, and access
to, the product. Again, there is a culture of obsolescence, to keep
customers upgrading to new versions and paying for maintenance.
The needs of Societal Infrastructure Software
Let us look at the needs for societal infrastructure software. They include the following:
Meet the functional requirements of the task.Robustness and long-term stability and security.Transparency to determine when changes are needed and that undesired functions are not being performed.Verifiable trustworthiness of all three of the above.Ease and low cost of training for effective use.Ease and low cost of maintenance.Minimization of maintenance.Ease and low cost of modification.Ease of replacement.Compatibility and ease of integration with other applications.Long-term availability of individuals able to train, maintain, modify, determine need for changes, etc.
The
structure and culture of a typical prepackaged software company is not
attuned to the needs of societal infrastructure software. The
"ongoing business entity" and "new version" mentality downplay the value
of the needs of societal infrastructure software and are sometimes at
odds.
By
contrast, custom software development can be tuned better to the needs
of societal infrastructure software. The mentality is more around the
one-time project leaving an ongoing result, and the cost structures are
sometimes such that low maintenance is encouraged. The drivers of custom
software are often the eventual users themselves, paying upfront for
development.
Some
of the problems with custom development with regards to societal
infrastructure software are the inability to spread the development and
maintenance costs among a large number of customers and the narrow focus
on the current requirements of the particular customer and their
current stage of need (which often may change in ways visible to other
customers but not yet to them).
A new style of development
What
is needed is some hybrid combination of custom and prepackaged
development that better meets the requirements of societal
infrastructure software.
How should such development look? What is the "ecosystem" of entities that are needed to support it? Here are some thoughts:
Funding for initial development should come from the users.
Bridges and water systems are usually funded by governments, not by
private entities that will run them for generations. The long-term needs
of the funders must be more inline with the project requirements than
the investment return needs of most private sources of capital.The projects need to be viewed as for more than one customer.
A system for tracking parking tickets is needed by many municipalities.
There is little need to have a different one for each. As a result, the
funding should also be able to come from a combination of multiple
sources. Funding or cost-sharing "cooperatives" need to exist.The requirements for the project must be set by the users, not the developers. The long-term aspects of the life of the results must be very explicit. Best-practices must be established, tracked, and revisited.There is the whole issue of data storage and interchange standards
that is critical to the long-term success and ability to do migration.
Impediments such as intellectual property restrictions and "digital
rights management" chokepoints must be avoided. (Lawmakers today must
realize how important data interchange and migration is to the basic
needs of society. They must be careful not to pollute the waters in an
attempt to deal with perceived threats to a minor part of the economy.)Another critical issue is platform (hardware and software) independence.
All development of long-term software needs to be created with the
possibility of new hardware, operating systems, and other "computer
infrastructure" in mind.The actual development
may be done by business entities which are built around implementing
such projects, and not around long-term upgrade revenue. Other
entities are needed for providing the ongoing services with a mentality
of keeping existing systems running. (The two entities may or may not be
related.) Many such companies already exist.The attributes of open source software need to be exploited.
This includes the transparency of the source code and the availability
for modification and customization. Much has been written with regards
to open source and its value for bug finding, security checking, etc.,
which is why this is needed. The added benefit here is that society as a
whole may benefit in unforeseen ways as new applications are found for
programs, be they in the private or public sector. The availability of
the source code, as well as the multi-customer targeting and other
aspects, enables a market for the various services needed for support,
maintenance, and training as well as connected and adjunct products.The
development may be done in-house if that is appropriate, but in many
cases there are legal advantages as well as structural for using
independent entities. Some governmental agencies may be precluded from
licensing their results under licenses that are most appropriate for the
long-term health of the projects. For example, they may be required to
release the program code into the public domain where it may then be
improved by others (and re-released under restrictive licenses) without a
return benefit to the original funders.Unlike much of the discussion about open source, serendipitous volunteer labor must not be a major required element.
A very purposeful ecosystem of workers, doing their normal scheduled
work, needs to be established to ensure quality, compatibility,
modifications, testing, security, etc. Educational and other
institutions may be employed with the appearance of volunteer labor as
students and other interested parties are used, much as courts and other
governmental agencies have used interns and volunteers for other
activities. The health of the applications being performed by the
software must not be dependent upon the hope that someone will be
interested in it; like garbage collecting, sewer cleaning, and probate
court judging, people must be paid.
The ecosystem of software development this envisions is different than that most common today. The details must be worked out. Certain entities that do not now exist need to be bootstrapped and perhaps subsidized. There must be a complete ecosystem, and as many aspects of a market economy as possible must be present.
Learning from civil engineering
My
friend Peter Levin pointed out to me that the analogy between software
engineering and civil engineering (the building of bridges, dams, and
sewers) should be used to help flesh out a potential structure of the
ecosystem. Here are some more thoughts inspired by that:
Architects,
civil engineers, and contractors as part of their training learn a set
curriculum, pass tests, and are often licensed. They are supposed to
share a body of knowledge and experience and demonstrate competence. What thrust should be part of the training of software engineers?
For years we emphasized execution speed, memory constraints, data
organization, flashy graphics, and algorithms for accomplishing this
all. Curricula needs to also emphasize robustness, testing,
maintainability, ease of replacement, security, and verifiability.Standards bodies publish best practices
(how high should a railing be above the stair tread, how thick should a
concrete footing be under a supporting pillar, etc.). Even though a
project might be novel (such as a new bridge or Boston's Big Dig), there
are many standards that can (and must) be applied. By standards here we
mean a conservative approach that is intended to minimize error,
increase security, and lower maintenance costs, not just facilitate data
interchange. Like all engineering, new software, as we know, commits old errors. We need to teach the right "war stories". Physical
projects are subject to inspection by standards bodies. When you have
electrical or plumbing work done, the town inspector comes to check the
work before the job can be considered finished. Transparent societal infrastructural software needs inspection.
This will raise the role of independent testing entities. There is much
talk about such roles in the discussion about electronic voting and
gambling machines, but it is also important for the software we are
covering here. These jobs -- part QA, part auditor, part private
investigator -- can be very high status because of the range and depth
of knowledge and experience needed. For public projects, the
transparency of open source is needed to allow multiple, independent
inspections. There are also different inspection specialties, including
standards compliance, security and other stressing, maintainability, and
functionality.When physical projects fail (a suspension bridge twists in heavy winds, an elevated freeway falls down in an earthquake, an airplane crashes) public inquiries are performed,
reports are published, and fixes are designed and retrofitted to
existing projects. What we learn from failures enter the standards
lexicon and is used for training and new design. We don't do this yet in
the world of software. Access to the source code, the right to discuss
it in detail, and the ability to search for similar code elsewhere is
crucial to many such studies.
Further thoughts
The
heart of this is some sort of open source software. The exact license
requirements are not yet clear, and will probably vary depending upon
the project. The depth of thinking that went into the GNU General Public
License is needed, and it is a good start.
The role of open source software scares many traditional software developers.
There is an image of a need for volunteer labor, and developers not
getting paid. This is far from the case. Developers and companies are
still needed, and the best will be in high demand and well paid.
Criteria of what is "good" may change -- the ability to write clear,
robust, maintainable code with an eye to the future, or do clean
modifications, or explain how to use old software in new contexts, will
become even more important. Documentation, training, servicing, testing,
and more will still be paid for. In fact, the knowledge that such work
has long-term consequences and may be amortized over longer periods of
time raises their value. What does go away is the effort spent on making
upgrading and replacement a desirable thing, both in development time
and marketing dollars.
What about competition? There is nothing that says that there should only be one product for each application. Competition is very helpful
for bringing out the best in product development. With that knowledge,
funders should consider funding more than one project and keep all
promising ones alive even if, as is the tendency with software, one
comes to dominate in deployments.
What
does this say about the size of the development entities? There is no
special requirement. Some may be very big, some may be very small.
Smaller entities (and projects) have a better chance than today, because
in such an ecosystem they would not be evaluated based on their own
ability to provide long-term support (a major impediment today), but
rather on their products' characteristics for fitting into the
ecosystem.
The
structure of the development may be concentrated mainly all in one
entity, much as with a product like MySQL or Adobe Photoshop.
Alternatively, it may instead be coordinated by a strong center, but
distributed among many players, such as with Linux. The key skills will
include the ability to manage such projects in the ecosystem. There is
probably a separation between managing the initial development and the
long-term maintenance and monitoring.
Is
this "socialized software", with the government making all decisions?
No. While funding and management cooperatives seem a likely part of the
ecosystem, there is no need for a single such entity; in fact, that
would be bad. Developers with promising ideas can still use risk capital
for initial development, and would still be able to find single
customers to provide the funding. Also, some projects may be worth
funding solely because they are synergistic with existing products that
are being supported by existing entities. So, for example, a training
and support company may help fund a product that lowers maintenance
costs and that will need training.
Remember, this is only for one part of the software world
-- that of social infrastructure software. There are many other uses of
software, each with their own preferred ecosystem for development and
support.
As part of this, buyers must get used to funding projects in advance.
This is already the case in many areas, and the addition of cooperative
funding with others can lower the costs or increase the scope of
potential projects. Buyer funding lowers the requirement for potential
"big hits" to incentivize development.
There
is much talk about open source software in relation to existing
software firms and lowering costs. What we are discussing here is opening up new types of firms, with huge potentials for revenues stemming from valuable services.
Open
source essays often revolve around cost savings of acquisition and the
use of volunteer labor for testing and maintenance. That is not the
thrust here. In fact, the acquisition costs may actually be higher, and
paid labor is assumed. The key is a model for long-term use, with a
lowering of total cost of ownership, less disruption, and better
integration. Open source discussion for government and business is often
just in regards to existing open source applications, such as Linux and
hoped-for desktop applications. There needs to be more discussion about
projects of less general interest to the common software developer,
such as EPA compliance monitoring systems, government record keeping,
court workflow systems, and e-government components. Open source
software discussion should be about keeping the trains running on time
and not just saying it should run on Linux. The discussions should be
about funding the companies needed in such an ecosystem and assuring
their sources of healthy revenue. The code is not the only part of the
equation, and leadership for all aspects of the ecosystem need to be
addressed.
I hope that this essay is helpful to people that need to be involved in bringing about this needed ecosystem.
-Dan Bricklin, 14 July 2004
As a
continuation of examining the area of long-term software, I've written
another essay as part of a process of looking for design principles to
follow. See "Learning From Accidents and a Terrorist Attack".
-Dan Bricklin, 7 September 2004
Related material:
Massachusetts Secretary of Administration and Finance Eric Kriss: "Open Mind on Open Source". History and rationale for sharing source in government.
Dan Bricklin's Log reports of meetings with Secretary Kriss: October 8, 2003 and January 12, 2004.
GNU Project: "Philosophy of the GNU Project". Links to essays about Free Software and free software licenses such as the GPL.
Peruvian letter about Open Source in government: Reactions from a Peruvian lawmaker to statements about open source submitted by a Microsoft representative.
Government Open Code Collaborative Repository:
"...a voluntary collaboration between public sector entities and
non-profit academic institutions created for the purpose of encouraging
the sharing, at no cost, of computer code developed for and by
government entities where the redistribution of this code is allowed."
Includes Massachusetts, Rhode Island, Pennsylvania, Utah, Kansas,
Missouri, West Virginia, and a variety of cities.
Avalanche Corporate Technology Cooperative:
"...a private exchange that enables it's members to contribute,
collaborate and legally distribute intellectual property to other
members." Founded by Best Buy, Cargill, and others. Shares code and
procedures among members. It is not open source.
Leopard Project of the Open Government Interoperability Project of the Open Source Software Institute:
"The Open-Source Software Institute (OSSI) is a non-profit (501 c 6)
organization comprised of corporate, government and academic
representatives whose mission is to promote the development and
implementation of open-source software solutions within U.S. Federal and
State government agencies and academic entities." The Leopard Project
is an eGovernment web services platform based on LAMP (Linux, Apache,
MySQL, PHP/Perl/Python open source components).
Copy Protection Robs The Future: Dan Bricklin's essay about the long-term dangers of Digital Rights Management.
The New York Times article on why slot machines are more trustworthy than voting machines because of testing and enforcement: "Gambling on Voting".
Books about the role of failure in engineering:
Henry Petroski's "To Engineer is Human:
The Role of Failure in Successful Design". This book, which has a
picture of the Tacoma Narrows bridge collapsing and the Challenger in
flight on its cover, discusses several well-known engineering failures.
It goes into detail about how the failures were analyzed and what we can
learn from them.Another Petroski book: "Design Paradigms:
Case Histories of Error and Judgment in Engineering". Petroski presents
several general paradigms of error, such as errors in conceptual
design, errors related to scale in size, errors in logic, success
masking failure, and others. To quote from the Preface: "This book
argues for a more pervasive use of historical case studies in the
engineering curriculum."Charles Perrow's "Normal Accidents:
Living with High-Risk Technologies". This book emphasizes learning
about failures through detailed study of many "accidents" and especially
"near-misses" and the systems around them. Don't say "Whew!" and ignore
"almosts", or say "well, it was an accident" -- learn from them both.
There are about 5,000 people a year killed in U.S. industry. This book
is covered in great depth in my "Learning From Accidents and a Terrorist Attack" essay.
