Alistair's top ten ways to know you are not doing agile

I stumbled upon the following not-so-new blog entry by Alistair Cockburn, a nice short list of "bad smells" for lack of agile project management. These days companies are still frequently trying to engage project managers asking for arcane list of certifications which are mostly theoretical and paperwork-oriented. Alistair, who has written one of the best book ever about Use-Cases, instead insists on agile methodologies and expert leadership as the only effective way to deliver projects.

In my experience pragmatism, agility and the ability to deeply dig for real business requirements are the most important factors for successful integration projects. For effective consulting is mandatory to keep open a constant, daily communication link with business experts and project stakeholders, relationships with just customer's technical people almost never provide for the necessary unbiased information.

Too many developers are just trying to experiment with the latest "cool" technologies, only because they want to fill a row in their CV, regardless of the real business needs of their employers. In my experience these days two of the biggest CV-fillers are SOAP Webservices and BPEL, as I rarely see those technologies used properly.
Not that they couldn't be useful, but:

"When all you have is a hammer, everything looks like a nail."

SOAP WS and BPEL are the nowadays' hammers.

Integration Projects Usual Madness

Joel Spolsky wrote a classical article about writing better code. The 12 Joel's rules are below summarized:

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

EAI projects pose additional serious challenges in terms of project management and quality assurance. Even if the term "EAI" seems to be out of fashion nowadays, I think the majority of projects which have been labelled as "SOA efforts" are, in facts, still plain, old EAI.

EAI Quality Assurance and Testing

Especially when doing EAI, unit testing is not enough. Joel is in favour of developer's unit testing and personally I think using a tool such as JUnit can prevent a huge numbers of bugs, but unit-testing integration flows is often much harder than unit-testing regular applications. This is due to the fact that when implementing EAI it is necessary to deal with a multitude of external systems which are out of our domain of control. Mocking those systems is often very hard or even impossible, it could lead to write and maintain more mocks than actually manageable and, additionally, the quality and realism of those mocks could be at least questionable.

Testing activities for EAI teams is a matter of organization and operations, because often it could be a cross-department issue. For example, testing SAP BAPI interfaces requires some commitment and lot of assistance from the SAP team, which in the meanwhile could have different priorities. The same for other legacy technologies. This clearly scales to a management issue, the EAI team has to be allowed to orchestrate the testing and validation process with all the involved external parties. Cross-department activities are usually difficult for organizational reasons, not much for technical ones.

Critical Success Factors

While at SeeBeyond we had some specifically integration-related rules, which were useful to establish what we called "Critical Success Factors":

• The executive sponsorship must be established up front to facilitate timely decision making, appropriate resource allocation and removal of business barriers.
• Establish an integration team with a balanced mix of IT, end business, management and consultants who will work together toward the main goal of a successful implementation
• Provide just-in-time training and coaching to the appropriate people on operation of the product components, keeping them ahead of the learning curve,
• Field level buy-in to the implementation is critical to the success of the project
• Executive management must communicate their commitment to the new technology and illustrate this commitment through their actions.
• Availability of key stakeholders to facilitate the decision making process and to keep the implementation moving forward.
• A complete test and validation plan should prepared in advance. It is not up to the single developer to validate his own work, QA is an overall function.
• As this is an integration project, all interfaces must be defined before starting the implementation.
• Changes have to be accommodated into a change management process which takes care of the deadline. All the interfaces should be unit-testable and end-to-end testable as being part of a higher level process.
• Implementation will proceed by very short iterations of analysis, design, development, test and deployment. At the end of each iteration a working prototype should be ready to run functional validation tests.

Regarding SOA, reports say that most SOA initiatives are failing because of inadequate business involvement and commitment. EAI is a better understood discipline (really?), but still projects tend to fail for exactly the same reason as newer SOA efforts.

Missing Deployment Plan

Just after the functional specs and scope, it is important to define a Deployment Plan and a Deployment Architecture:

"The deployment project plan establishes a baseline reference plan. It is your road map for the deployment project. The roles and responsibilities of each organization, the schedule of tasks, and any estimates must be defined in this plan.
[...]
Deployment Architecture (DA), together with Requirements Analysis, is the most important component of your plan, and should be documented carefully. Deployment begins when you plan and schedule how, in view of your analysis and allocated resources, you want to implement your environment. During this phase, you set up the operation procedure and schedule for the entire deployment project. DA provides guidance to the successive design and implementation phase, so DA creation always must precede it. In fact, developers needs to use the DA document as a guide for implementing their flows, adapting services communication to the DA. Of course, sometime it is necessary to adjust the DA to accommodate design needs emerging after some development, but if this happens frequently then it is a sign of a bad original DA, so it is time for a deep review.
Poor analysis and planning can cause major problems during the later phases. Thorough, comprehensive analysis and planning can make the deployment project easier, more efficient, and less costly."

The above is a mix of statements from the Java CAPS documentation plus some comments of mine.

Organizational Issues

In my experience most of the issue I have encountered during Integration projects are organizational and operational:

• Lack of organizational information
• No complete specifications
• Not enough back-end information to fill the specifications
• Lack of coordination
• Frequent changes in project organization (document store moving, too many documents everywhere, anonymous doc.)
• Huge integrations are thrown to single developers
• Lack of commitment
• Sometime is difficult to understand who is in charge
• Developers are frustrated by changing deadlines
• Bad or totally missing test data!!!

Missing Domain Model

You wouldn't write SQL queries before modelling any database relationship, would you?
An usual requirement for Services is to build first a Domain Model as a common, business oriented standard communication device. This is very close to good Object-Oriented programming practices, but Service-Oriented is focused on exchanging documents instead of objects. A domain model can be designed by normal entity-relationship diagrams and, for example, implemented through XML-Schemas.

Common Misunderstandings

• Integration projects are like software development projects

>> No! they are very different!
>> Integration projects are cross-department projects
>> Need more involvement
>> Need more cross-management

• Bugs in software are all the evil

>> No! Half of bugs are not real platform's bugs but behaviour due to bad design/assumption.

Integration project with any vendor will fail if there are no clear specifications...

Inter-team communication madness

• Every team has its own life
• Not coordinated at all
• No motivations to help
• Mostly Anonymous functional and technical documents !?!

Finally... Some Recommendations

• Long term strategy but short delivery cycles
• Top-down, business-driven analysis
• Short iterations, deliver value within few weeks
• Break big projects into smaller pieces
• Stick to deadlines! Avoid changing a deadline!
• Instead, what features can fit into the deadline?
• Adopt a Project Management methodology!

Some of the above statements could seem controversial. What about deadlines? there is nothing more frustrating for a developer than working 80 hours per week, weekends also, just to discover that a couple of days before deployment some manager decides that the feature is not really that important and can be delayed. Boy, you are not going to catch me anymore! So my advice is: fix the deadline and instead move in and out features, keep the pace, sustain the mood.

Always ask ourself why? "Why?" is extremely powerful: before start fighting to solve a technical issue one should ask why I'm doing this? Is there any better, cleaner design which could alleviate or even cancel the technical issue? Do I really have all the necessary information? Am I answering a question nobody asked for?

Conclusions

Integration projects are somehow quite different from application development projects:

• Interfaces and meta-data must be clearly defined
• Service contracts must be defined (SLAs)
• Test data must be readily available!
• External systems experts must be available!
• An architectural board in charge of taking decisions needs to be established.

It seems that the above list is exactly what we need in case of any project, application development or integration. But badly-managed dependencies with the external world can sink any integration project regardless of how good is the team and its internal process. Integration is about connecting existing interfaces: what can I do if interfaces are fuzzy and undefined? Frustration, that is....

Thanks goes to my friend Juraj Kazda of Sun, for discussing and writing down these issues while working together in several projects.

Services Virtualization and the Need for Lifecycle Management

Introduction

What I personally call "Natural Requirements" (NR) are requirements not coming from end users, analysts or marketing people. Natural Requirements are often more operational than functional or technical, let me call the latter "static requirements", and I mean the one we usually capture in the early phases of the requirements gathering process. NR instead derive from the natural and time-dependant evolution of a system well after the systems has been built and put into production. NR are naturally imposed by the environment, the context and by the fact that systems are always evolving and requires a lot of maintenance, and there is nothing we can do to eliminate them: instead we need to deal with them from a strategic standpoint, looking at the long run.

A SOA is made of a collection of interacting business services, each one with its own history and evolution. Starting from domain and data models, services defines an atomic and well identified piece of data, algorithmic result or a combination of both. Services creates a public API, they expose internal technical artefacts as integrated business, higher-level information. We (almost) all know what the SOA objectives are, so it is not interesting to repeat them here.



Natural Requirements for Service Virtualization

Services anyway pose a maintenance problem: they evolve, need to be monitored and policies have to be enforced. A public API, once published, can't be changed easily, as it is a public contract and we cannot control how many clients and how they make use of the API. When we change a public API we know that we are going to break contracts with clients, and clients need to be changed to adapt to the new interfaces.

A first Natural Requirement is that services have to be versioned to reach a level of backward compatibility. For this purpose, modern SOA Governance solutions offer not only a Registry but also a Repository - ideally a combination of both - where service interfaces and messages can be versioned to avoid breaking contracts when things change. Each client can point to the proper version of the same service and related messages.

Services implements business logic atomic features, but in their lifecycle they also deal with other orthogonal concerns, which are not functional in their nature: Policies, End point definitions, messages structure manipulation and message formats are all non-business aspects.
So Separation of Concerns is one of the main and most frequent Natural Requirement we could ever encounter in our daily routine as SOA architects. Orthogonal concerns must be consistently keep divided from business logic and other aspects. For example, one of the most common orthogonal concerns we must split from business logic is Security. For the sake of this discussion, we could identify a couple of concrete technical ways to implement services: SOAP messages over HTTP and REST Web Services. The first are usually secured by using some complex WS-Security implementation, the latter by simpler HTTP usual methods, such as basic authentication + SSL. Policy enforcement and core services must be kept separated, business logic developers shouldn't have to know that such a thing like WS-Security even exists.




"Composite applications must have the business logic defined separately from each application's infrastructure. For example, instead of baking security semantics into the code, use an external, general-purpose security framework. This will be your template for enforcing authentication and authorization rules based on declarative policies that specify the required security semantics for the service within the context of a specific application."

[ Anne Thomas Manes - The Elephant Has Left The Building ]

Services Virtualization is usually defined as a virtual view (much like a database view, at least conceptually) of a real serviced managed through a policy. A Policy is a declarative (not-programmatic) set of administrative tasks. For example you can have authentication, authorization and generally speaking access management policies. Or you can define a set of minimal performances level policies in terms of SLAs. Or then you can have billing policies for your services (think at Amazon's web services) based on volumes or number of calls.




Natural Requirements for PLM

SOA implies that business services and Composite Applications (CA) have to be handled like real products, so PLM (Product Lifecycle Management) issues and practices apply. Many vendors now offers (almost) comprehensive governance solutions which embrace policies + registry + artifacts repository and all the related versioning, but that's not enough. Governance solutions are platforms, but PLM is about the product and the process. Services needs to be created as the building blocks for CA, so they are part of the product lifecycle: the final product is the CA, while services are the components, which can be re-assembled and mixed to create other CA.
The main steps are:

Phase 1: Conceive
Imagine, Specify, Plan, Innovate

Phase 2: Design
Describe, Define, Develop, Test, Analyse and Validate

Phase 3: Realize
Manufacture, Make, Build, Procure, Produce, Sell and Deliver

Phase 4: Service
Use, Operate, Maintain, Support, Sustain, Phase-out, Retire, Recycle and Disposal

Why we should talk about PLM and not about ALM (Application lifecycle management)? ALM is just about software, while PLM is about manufacturing, so the first acronym sounds more natural. Because applications are monolithic, they are not composite in the same terms we use when building a product in manufacture. The roads to assemble a Composite Application are somehow different from Application ones, actually they are closer to manufacturing processes. The reason is that we want to create a predictable and repeatable assembly chain for services. ALM is really a subset of PLM: the Use, Operate, Maintain, Support, Sustain, Phase-out, Retire, Recycle and Disposal activities of Phase 4 needs to be operated at the level of each single Service and not at the Application level only. When we talk about SOA, services (the building blocks) are way more crucial than the resulting Composite Applications, like reliable foundations are for a house.

The Model T's body is joined to its chassis at the Highland Park plant

When we think again about the manufacturing metaphor, and specifically about car manufactures, we could assimilate the car as the final product of an assembly of other products. Let's say car wheels and tires has a clearly defined interface and the car maker just buys the product from another supplier. Tires producer knows its product can be assembled by a number of different clients, so it cannot make too many assumptions on the external world: they just have to respect the contract.

In conclusions, to be effective Services construction, assembly and maintenance require us to build our own assembly chain, following best PLM practices. But still best practices in this field are in their infancy.

The description of SOA I enjoy most is still Dan North's A Classic Introduction to SOA, which is additionally a very easy reading.

When all are responsible, nobody is

Democracy doesn't fit well with software development, committees don't work for software development. Individual accountability works. Ask few good individuals for commitment and give them the power to act.
I'm trying to recall what the context usually is when things go wrong and what are the conditions when a project succeed. If I look back to my experience most of the successful software development projects I've been involved succeeded exactly when the decision chain was very short and decision taken promptly, especially the unpopular ones.

I think it was Fred Allen who said, "A committee is a group of men who individually can do nothing, but as a group decide that nothing can be done."

This is known also as a variant of the bystander effect. "The bystander effect (also known as bystander apathy, Genovese syndrome, diffused responsibility or bystander intervention) is a psychological phenomenon in which someone is less likely to intervene in an emergency situation when other people are present and able to help than when he or she is alone."

Really big distributed projects, say Linux and the Python language, have clear leadership expressed by few or even one person. Linux has Linus, Python has Guido. Are really our projects any bigger or more complex?

Now integration is organizationally harder than application development because integration is cross-department and under control of different ownership domains. If you leave the execution control to a committee than you can be pretty sure it is not going to deliver: distributed responsibility means almost no responsibility.

"A good plan violently executed now is better than a perfect plan executed next week."
- George S. Patton

If the team is big the Scrum agile project management process is something which could help splitting the team into smaller teams, each one with a local leader who reports to the top team leader (or top Scrum Master). The team leaders (local and top) are responsible to deliver the solution, the Time & Resource Manager is in charge to check time and budget, while the QA Manager is responsible to provide testing and validation. That's it.
If you think you can deliver any SOA or EAI project by creating a committee made of members of all IT department involved in the integration effort then you are doomed, you'll end into analysis-paralisys very soon because everybody want to say something. The next common mistake is adding developers to a late project: as Tom DeMarco taught, adding manpower to a late project actually makes it later.

“Craziness is doing the same thing and expecting a different result.”
- Tom DeMarco

We are convinced that software development management is hard, but too many times I'm experiencing self-handicapping management, when consultants or end customers are scared of taking any decision and just start blaming the tools they are using and start opening support incident without relying on what, if they, know. Then a two-months integration flow takes three years and twenty times the estimated budget.
If you have a problem, fix it now. When a better solution comes, refactor to integrate it to your fix.

"Anticipate the difficult by managing the easy."
- Lao Tzu

Project Collaboration with Basecamp

Recently we started a small project and decided to try Basecamp, a collaboration tools from 37signals, authors of the famous Ruby on Rails framework.

Basecamp is a unique project collaboration tool. Projects don't fail from a lack of charts, graphs, or reports, they fail from a lack of communication and collaboration. Basecamp makes it simple to communicate and collaborate on projects.

Give this web application a try, it is really smart: it allows you to be productive in few minutes and incredibly facilitates teams collaboration. Each single operation is clean , simple and intuitive, making a wise use of Ajax widgets. We are testing the free edition, which comes with some limitations but it is good enough to start. Then it is possible to upgrade to a professional edition if you realize you need more horse-power. However, pricing looks very competitive.