RSP&&A;; Web Site: Commentary

What a Tangled Web We'll Weave

A version of this commentary was published in IEEE Software, January, 2000.

Yogi Berra, a professional baseball player, manager, and erstwhile philosopher, said it best, “It’s deja vu all over again.”

Of course, Yogi wasn’t talking about Internet-based applications and e-commerce sites (what I’ll call WebApps), but I digress. Maybe we should start at the beginning.

About every ten years or so, a major new software-related technology captures the consciousness of the industry. In some cases, the technology is more hype than substance, but it becomes the prevailing fashion. Avant garde software folks claim it as their own, and in so doing, become the darlings of the technological scene. High salaries, considerable prestige, and no small amount of hubris are sure to follow. The corps of avant garde adherents argue that the new technology is truly different, requiring a new ‘paradigm.’ The ways of the past simply don’t apply. In fact, the old ways can’t possibly be adapted to a new set of business rules and technological realities. As a result, the avant garde rejects the disciplines of the generation that preceded them, but ironically, adopts approaches that failed miserably a few generations back.

The Internet and the vast array of applications that it has spawned are undoubtedly a major new software-related technology. I won’t bore you with the obvious cliches—suffice to say that the Internet and the WebApps that populate it are big—very big, and that their impact is profound.

What worries me is that this major new technology has become a breeding ground for important WebApps that are hacked in much the same way as important application software was hacked a few generations back—in the 1960s and 1970s.

WebApps have to be hacked, argue the avant garde (who, of course, would never use the word “hacked”) because:

1. WebApps must be developed in days or weeks—time frames that don’t allow for anything but a rush to the finish line.

2. WebApps are constantly evolving—so why spend the time specifying what’s needed and designing how to build it when everything will change anyway.

3. WebApps are inherently different than application software—content (e.g., text, graphics, images, audio, video) is integrated inextricably with procedural processing.

4. The people who use WebApps are more tolerant of errors. What users really want are cool Web sites that are up and running in days, and besides, it’s almost impossible to know what WebApp users really want, because the demographics of Web visitors are so hard to predict.

5. The people who build WebApps are different—free-thinkers all—who certainly would feel unduly constrained by the old ways. In fact, talk of a disciplined approach—other than build it, test it to death (if time permits) and then put it on-line—usually results in grimaces all around.

Of course, I’m overstating my point a bit, but you get the picture. Too many WebApp developers make these statements in an attempt to erect barricades against an “old school” view that suggests a disciplined, engineering approach to the creation of business critical Web-based systems.

Can WebApps be Engineered?

In a virtual round table published in IEEE Software in September, 1998, I staked out my position with regard to Web engineering:

“It seems to me that just about any important product or system is worth engineering. Before you start building it, you’d better understand the problem, design a workable solution, implement it in a solid way, and test it thoroughly. You should probably also control changes to it as you work and have some mechanism for ensuring the end result’s quality. Many Web developers don’t argue with this; they just think their world is really different and that conventional software engineering approaches simply don’t apply.”

This leads us to a pivotal question: Can software engineering principles, concepts, and methods be applied to WebApp development? I believe that many of them can, but their application may require a somewhat different spin.

But what if I’m wrong? What if the current ad hoc approach to Web development persists? In the absence of disciplined process for developing Web-based systems, there is increasing concern that we may face serious problems in their successful development, deployment, and maintenance over the long term. In essence, the application infrastructure that we are creating today may lead to something that might be called a “tangled Web” as we move further into this new century. I use this phrase to connote a morass of poorly developed Web-based applications that have too high a probability of failure. As Web-based systems grow more complex, a failure in one can and will propagate broad-based problems across many. When this happens, confidence in the entire Internet may be shaken.

Of course, there is another view. Some argue that the application of software engineering principles (even if they are adapted to accommodate the unique characteristics of WebApps) is simply wrong-headed; that software engineering doesn’t really work very well in the rapid fire world of WebApp development.

I contend that software engineering principles _always_ work. It's never inappropriate to stress solid problem solving, good design, and thorough testing (not to mention, the control of change, an emphasis on quality, yadda, yadda). A specific software process may fail because it is overkill, or the work products it requires are unnecessary or burdensome, or a person or team becomes overly dogmatic in the application of the process. But history is on the side of a solid engineering approach.

What is Web Engineering?

Web Engineering is an adaptable, incremental (evolutionary) process populated by a set of framework activities that occur for all business-critical WebApp projects, regardless of the size or complexity. The following framework activities might be considered for Web engineering work: formulation, planning, analysis, modeling, page generation and testing, and customer evaluation. These activities are applied iteratively as a Web-based system evolves.

Formulation identifies the goals and objectives of the WebApp and establishes the scope for the first increment. Planning estimates overall project cost, evaluates risks, and defines a development schedule for the initial WebApp increment. Analysis establishes requirements for the WebApp and identifies the content items that will be incorporated.

The modeling activity incorporates two parallel tasks sequences: content design and production that results in the creation or acquisition of content within the WebApp, and architecture, navigation, and interface design which establish the structure of the WebApp and the flow of the user’s interaction with it.

Page generation_ and _testing are requisite construction activities. Finally, each increment produced as part of the Web engineering process is reviewed during _customer evaluation_. It is at this point that changes are requested (scope extensions occur). These changes are integrated into the next path through the incremental process flow.

I’m the first to admit that these framework activities are nothing more than common sense. But in our business, common sense is sometimes in short supply. Too many WebApp efforts just happen—with results that may work today, but are suspect over the long term.

Managing Web Engineering

Project management for Web engineering is governed by the unique characteristics of WebApp projects. These characteristics precipitate questions whose answers can make or break a project.

First, a substantial percentage of WebApps are outsourced. How is the competence of a WebApp vendor determined? How does one know whether a price quote is reasonable? What degree of planning, scheduling, risk assessment can be expected as an organization (and its contractor) embarks on a major WebApp development effort?

Second, WebApp development is a relatively new application area. There is little historical data and fewer project-oriented metrics to use for estimation. Therefore, how are reliable estimates derived? What degree of assurance can be given that defined schedules will be met?

Third, estimation, risk analysis, and scheduling are all predicated on a clear understanding of project scope. But WebApp scope is fluid. How can the contracting organization and the outsourcing vendor control costs and schedule when requirements are likely to change dramatically as a project progresses? How can scope creep be controlled, and more important, should it be controlled, given the unique nature of Web-based systems and applications?

At this stage in the history of project management for WebApps, the questions noted above are not always easy to answer. However, a few guidelines are worth considering.

Initiating a project. Even if outsourcing is your strategy, a number of Web engineering tasks should be performed internally _before_ outsourcing the work. You’ll have to identify the audience for the WebApp and pinpoint the internal stakeholders who may have an interest in the WebApp. The overall goals for the WebApp should be defined and reviewed, the information and services to be delivered by the WebApp should be specified, competing Web sites should be evaluated, and qualitative and quantitative ‘measures’ of a successful WebApp should be defined. Sound familiar? It should—you’re doing requirements analysis.

An expert developer should create a complete design for the WebApp, but time and cost can be saved if you do a rough design before turning the project over to the vendor. The design should include an indication of the type and volume of content to be presented by the WebApp, the types of interactive processing to be performed and any interoperability requirements for existing systems. In essence, the general look and feel of the WebApp is identified (this can always be modified during later stages of the project).

Finally, think about the degree of oversight and interaction you’ll require during the project. It’s not a bad idea to identify appropriate liaison personnel (by name) and to define review points as development proceeds.
Once you’ve completed these project initiation activities, you’re ready to generate a request for quote that is transmitted to candidate vendors. If WebApp development work is to be conducted by an internal group, nothing changes! The project is initiated in the same manner.

Selecting a Vendor. In order to select candidate Web developers, you must perform due diligence: (1) interview past clients to determine the Web vendor’s professionalism, ability to meet schedule and cost commitments, and ability to communicate effectively: (2) determine the name of the vendor’s chief Web engineer(s) for successful past projects (and later, be certain that this person is contractually obligated to be involved in your project), and (3) carefully examine samples of the vendor’s work that are similar in look and feel (and business area) to the WebApp that is to be contracted. Even before a request for quote is offered, a face to face meeting may provide substantial insight into the fit between contractor and vendor.

Assessing an Estimate. Because relatively little historical data exist and the scope of WebApps is notoriously fluid, estimation is inherently risky. For this reason, some vendors will embed substantial safety margins into the cost quoted for a project. The question are not “Have we found the lowest cost option?” or “Have we gotten the best bang for our buck?” Rather, the questions should be:

• Does the quoted cost of the WebApp provide an acceptable direct or indirect return-on-investment?
• Does the vendor exhibit the professionalism and experience we require?

If the answers to these two questions are ‘yes,’ the price quote should be given serious consideration.

Assessing the Development Schedule. Because WebApp development schedules span a relatively short period of time (often less than one or two months), the development schedule should have a high degree of granularity. That is, work tasks and minor milestones should be scheduled on a daily timeline. This fine granularity allows both the contractor and the vendor to recognize schedule slippage before it threatens the final completion date.

Managing Scope. Because it is highly likely that scope will change as a WebApp project moves forward, the Web engineering process model is incremental. This allows the development team to ‘freeze’ scope for one increment so that an operational WebApp release can be created. The next increment may address scope changes suggested by a review of the preceding increment, but once the second increment commences, scope is again ‘frozen’ temporarily. This approach enables the WebApp team to work without having to accommodate a continual stream of changes, but still recognizes that most WebApps evolve continuously.

There is much that managers can learn from the new generation of WebApp developers—their enthusiasm, creativity, technical competence, and innate understanding of what makes a good WebApp must not be ignored. But there are also things (such as the guidelines noted above) that the new generation can learn from those of us who have been around the block a few times. The question, really, is whether any meaningful learning will occur (in either direction).

Yogi Berra was an accidental philosopher when he talked about deja vu. George Santayana thought hard when he made the comment: “Those who cannot remember the past are doomed to repeat it.” But maybe that’s our karma in the software biz. There’s absolutely nothing that I’ve seen over the past 30 years to make me think otherwise.

Home About us Products Product Models SE Resources Commentary Contact us
Web site and all contents © 2006, R.S. Pressman & Associates, Inc.
All rights reserved.

Free website templates


	*What a Tangled Web We'll Weave* A version of this commentary was published in IEEE Software, January, 2000. Yogi Berra, a professional baseball player, manager, and erstwhile philosopher, said it best, “It’s deja vu all over again.” Of course, Yogi wasn’t talking about Internet-based applications and e-commerce sites (what I’ll call WebApps), but I digress. Maybe we should start at the beginning. About every ten years or so, a major new software-related technology captures the consciousness of the industry. In some cases, the technology is more hype than substance, but it becomes the prevailing fashion. Avant garde software folks claim it as their own, and in so doing, become the darlings of the technological scene. High salaries, considerable prestige, and no small amount of hubris are sure to follow. The corps of avant garde adherents argue that the new technology is truly different, requiring a new ‘paradigm.’ The ways of the past simply don’t apply. In fact, the old ways can’t possibly be adapted to a new set of business rules and technological realities. As a result, the avant garde rejects the disciplines of the generation that preceded them, but ironically, adopts approaches that failed miserably a few generations back. The Internet and the vast array of applications that it has spawned are undoubtedly a major new software-related technology. I won’t bore you with the obvious cliches—suffice to say that the Internet and the WebApps that populate it are big—very big, and that their impact is profound. What worries me is that this major new technology has become a breeding ground for important WebApps that are hacked in much the same way as important application software was hacked a few generations back—in the 1960s and 1970s. WebApps have to be hacked, argue the avant garde (who, of course, would never use the word “hacked”) because: 1. WebApps must be developed in days or weeks—time frames that don’t allow for anything but a rush to the finish line. 2. WebApps are constantly evolving—so why spend the time specifying what’s needed and designing how to build it when everything will change anyway. 3. WebApps are inherently different than application software—content (e.g., text, graphics, images, audio, video) is integrated inextricably with procedural processing. 4. The people who use WebApps are more tolerant of errors. What users really want are cool Web sites that are up and running in days, and besides, it’s almost impossible to know what WebApp users really want, because the demographics of Web visitors are so hard to predict. 5. The people who build WebApps are different—free-thinkers all—who certainly would feel unduly constrained by the old ways. In fact, talk of a disciplined approach—other than build it, test it to death (if time permits) and then put it on-line—usually results in grimaces all around. Of course, I’m overstating my point a bit, but you get the picture. Too many WebApp developers make these statements in an attempt to erect barricades against an “old school” view that suggests a disciplined, engineering approach to the creation of business critical Web-based systems. Can WebApps be Engineered? In a virtual round table published in IEEE Software in September, 1998, I staked out my position with regard to Web engineering: “It seems to me that just about any important product or system is worth engineering. Before you start building it, you’d better understand the problem, design a workable solution, implement it in a solid way, and test it thoroughly. You should probably also control changes to it as you work and have some mechanism for ensuring the end result’s quality. Many Web developers don’t argue with this; they just think their world is really different and that conventional software engineering approaches simply don’t apply.” This leads us to a pivotal question: Can software engineering principles, concepts, and methods be applied to WebApp development? I believe that many of them can, but their application may require a somewhat different spin. But what if I’m wrong? What if the current ad hoc approach to Web development persists? In the absence of disciplined process for developing Web-based systems, there is increasing concern that we may face serious problems in their successful development, deployment, and maintenance over the long term. In essence, the application infrastructure that we are creating today may lead to something that might be called a “tangled Web” as we move further into this new century. I use this phrase to connote a morass of poorly developed Web-based applications that have too high a probability of failure. As Web-based systems grow more complex, a failure in one can and will propagate broad-based problems across many. When this happens, confidence in the entire Internet may be shaken. Of course, there is another view. Some argue that the application of software engineering principles (even if they are adapted to accommodate the unique characteristics of WebApps) is simply wrong-headed; that software engineering doesn’t really work very well in the rapid fire world of WebApp development. I contend that software engineering principles _always_ work. It's never inappropriate to stress solid problem solving, good design, and thorough testing (not to mention, the control of change, an emphasis on quality, yadda, yadda). A specific software process may fail because it is overkill, or the work products it requires are unnecessary or burdensome, or a person or team becomes overly dogmatic in the application of the process. But history is on the side of a solid engineering approach. What is Web Engineering? Web Engineering is an adaptable, incremental (evolutionary) process populated by a set of framework activities that occur for all business-critical WebApp projects, regardless of the size or complexity. The following framework activities might be considered for Web engineering work: formulation, planning, analysis, modeling, page generation and testing, and customer evaluation. These activities are applied iteratively as a Web-based system evolves. Formulation identifies the goals and objectives of the WebApp and establishes the scope for the first increment. Planning estimates overall project cost, evaluates risks, and defines a development schedule for the initial WebApp increment. Analysis establishes requirements for the WebApp and identifies the content items that will be incorporated. The modeling activity incorporates two parallel tasks sequences: content design and production that results in the creation or acquisition of content within the WebApp, and architecture, navigation, and interface design which establish the structure of the WebApp and the flow of the user’s interaction with it. Page generation_ and _testing are requisite construction activities. Finally, each increment produced as part of the Web engineering process is reviewed during _customer evaluation_. It is at this point that changes are requested (scope extensions occur). These changes are integrated into the next path through the incremental process flow. I’m the first to admit that these framework activities are nothing more than common sense. But in our business, common sense is sometimes in short supply. Too many WebApp efforts just happen—with results that may work today, but are suspect over the long term. Managing Web Engineering Project management for Web engineering is governed by the unique characteristics of WebApp projects. These characteristics precipitate questions whose answers can make or break a project. First, a substantial percentage of WebApps are outsourced. How is the competence of a WebApp vendor determined? How does one know whether a price quote is reasonable? What degree of planning, scheduling, risk assessment can be expected as an organization (and its contractor) embarks on a major WebApp development effort? Second, WebApp development is a relatively new application area. There is little historical data and fewer project-oriented metrics to use for estimation. Therefore, how are reliable estimates derived? What degree of assurance can be given that defined schedules will be met? Third, estimation, risk analysis, and scheduling are all predicated on a clear understanding of project scope. But WebApp scope is fluid. How can the contracting organization and the outsourcing vendor control costs and schedule when requirements are likely to change dramatically as a project progresses? How can scope creep be controlled, and more important, should it be controlled, given the unique nature of Web-based systems and applications? At this stage in the history of project management for WebApps, the questions noted above are not always easy to answer. However, a few guidelines are worth considering. Initiating a project. Even if outsourcing is your strategy, a number of Web engineering tasks should be performed internally _before_ outsourcing the work. You’ll have to identify the audience for the WebApp and pinpoint the internal stakeholders who may have an interest in the WebApp. The overall goals for the WebApp should be defined and reviewed, the information and services to be delivered by the WebApp should be specified, competing Web sites should be evaluated, and qualitative and quantitative ‘measures’ of a successful WebApp should be defined. Sound familiar? It should—you’re doing requirements analysis. An expert developer should create a complete design for the WebApp, but time and cost can be saved if you do a rough design before turning the project over to the vendor. The design should include an indication of the type and volume of content to be presented by the WebApp, the types of interactive processing to be performed and any interoperability requirements for existing systems. In essence, the general look and feel of the WebApp is identified (this can always be modified during later stages of the project). Finally, think about the degree of oversight and interaction you’ll require during the project. It’s not a bad idea to identify appropriate liaison personnel (by name) and to define review points as development proceeds. Once you’ve completed these project initiation activities, you’re ready to generate a request for quote that is transmitted to candidate vendors. If WebApp development work is to be conducted by an internal group, nothing changes! The project is initiated in the same manner. Selecting a Vendor. In order to select candidate Web developers, you must perform due diligence: (1) interview past clients to determine the Web vendor’s professionalism, ability to meet schedule and cost commitments, and ability to communicate effectively: (2) determine the name of the vendor’s chief Web engineer(s) for successful past projects (and later, be certain that this person is contractually obligated to be involved in your project), and (3) carefully examine samples of the vendor’s work that are similar in look and feel (and business area) to the WebApp that is to be contracted. Even before a request for quote is offered, a face to face meeting may provide substantial insight into the fit between contractor and vendor. Assessing an Estimate. Because relatively little historical data exist and the scope of WebApps is notoriously fluid, estimation is inherently risky. For this reason, some vendors will embed substantial safety margins into the cost quoted for a project. The question are not “Have we found the lowest cost option?” or “Have we gotten the best bang for our buck?” Rather, the questions should be: • Does the quoted cost of the WebApp provide an acceptable direct or indirect return-on-investment? • Does the vendor exhibit the professionalism and experience we require? If the answers to these two questions are ‘yes,’ the price quote should be given serious consideration. Assessing the Development Schedule. Because WebApp development schedules span a relatively short period of time (often less than one or two months), the development schedule should have a high degree of granularity. That is, work tasks and minor milestones should be scheduled on a daily timeline. This fine granularity allows both the contractor and the vendor to recognize schedule slippage before it threatens the final completion date. Managing Scope. Because it is highly likely that scope will change as a WebApp project moves forward, the Web engineering process model is incremental. This allows the development team to ‘freeze’ scope for one increment so that an operational WebApp release can be created. The next increment may address scope changes suggested by a review of the preceding increment, but once the second increment commences, scope is again ‘frozen’ temporarily. This approach enables the WebApp team to work without having to accommodate a continual stream of changes, but still recognizes that most WebApps evolve continuously. There is much that managers can learn from the new generation of WebApp developers—their enthusiasm, creativity, technical competence, and innate understanding of what makes a good WebApp must not be ignored. But there are also things (such as the guidelines noted above) that the new generation can learn from those of us who have been around the block a few times. The question, really, is whether any meaningful learning will occur (in either direction). Yogi Berra was an accidental philosopher when he talked about deja vu. George Santayana thought hard when he made the comment: “Those who cannot remember the past are doomed to repeat it.” But maybe that’s our karma in the software biz. There’s absolutely nothing that I’ve seen over the past 30 years to make me think otherwise.