The Growing of REST

Here I try to give my point of view on why REST is gaining a lot of traction against SOAP in the Webservices world.

First, I have to explain that REST is an architectural style while SOAP-based WS are just a way to implement RPC, which is the most basic inter-communication style. Even Corba one day went to the point of being very mature an stable, but how many new projects are now started with it? A technology should demonstrate itself within a reasonable time window, otherwise people get tired. My first SOAP implementation, I think it was 2000, was with Axis's ancestor Apache SOAP: I still feel the headache.

My opinion is that most self-proclaimed RESTful implementations are not really so, because REST is not just XML over HTTP (it can use JSON, plain text, Atom, binary streams, or a combination of all) but it is a way to implement an Object-Oriented architectural approach on top of Web resources (the data is the resource itself, the methods are the HTTP operations get, put, post delete, etc...). SOAP WS instead, as said, is plain, old RPC and it does not mandate to use HTTP only, but also works with other protocols such as JMS, etc.

Now I hope we agree programming languages evolved from procedural to OO, so I'd say the REST paradigm is more modern than RPC one, but we know that good OO is actually harder than procedural (and I mean good OO). So, IMHO, the traction REST is gaining is not due to the fact that it is conceptually simpler (as OO is not, if compared to procedural programming) but instead to the fact that REST is based on a simpler implementation, robust protocol (HTTP), few consistent operations (HTTP methods), a proven architecture (The World Wide Web, not a tiny one really...), a set of extremely tested technologies (Web servers, Web proxy and caches) and a long track of true interoperability and security (again, the WWW and SSL). Honestly I am very happy that I still access my on-line banking through HTTP + SSL and not any WS-Security......

Resource Oriented Architecture, implemented through REST, is actually conceptually harder than RPC style for the same reasons that proper Object Oriented modelling is conceptually harder than plain procedural programming. But a lot more rewarding, if we compare the final outcomes. And that's why, by the way, also good stuff like Spring is gaining a lot of traction: it allows to model a real domain through POJOs.

Maybe I am too old, because in this procedural world of SOAP and EJB (yes, EJB specs before version 3 forced a procedural programming style) I still persist thinking that Object Orientation should be the way to model business domains properly: I learned that many years ago and still nothing has been able to change my mind. Of course, integration often demands for a simpler paradigm than OO: probably scripting will happily rule here (IFL and Fuji anyone?).

My opinion is that the WSDL + WS-* combination, regardless of all the efforts, is still a way too complex, real interoperability is still questionable and security is largely unproven, if compared with plain, old HTTPS and its some billions of daily Web transactions. However, I agree things are improving, but while I'm at customer site I'd love not to spend days resolving WSDL and WS-* quirks and instead focus my attention to more added-value activities....

More simplicity quite usually equates to more security.

Tom Barrett's Open ESB and Mural Tutorials

Sun's Tom Barrett has produced a set of very well written and easy to follow tutorials, which can gently lead the reader toward a step by step introduction with many of the newest OpenESB related technologies, together with Netbeans and Glassfish. Good job Tom!

Tom Barrett's OpenESB Tutorials

Dump Glassfish HTTP conversations to debug security

In Glassfish is possible to dump the HTTP conversation to debug Web Services, by adding the following two JVM options to domain.xml:

<jvm-options>-Dcom.sun.xml.ws.transport.http.HttpAdapter.dump=true</jvm-options>
<jvm-options>-Dcom.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true</jvm-options>

Then when the web service is called the server.log will show complete SOAP envelopes for both request and reply:

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
---[HTTP request]---|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
Host: localhost:8080|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
Content-type: text/xml;charset=UTF-8|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
Content-length: 866|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
User-agent: Jakarta Commons-HttpClient/3.1|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
Soapaction: ""|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:svc="http://xml.netbeans.org/schema/svcMessages">
   <soapenv:Header>
<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><wsse:UsernameToken wsu:Id="UsernameToken-27540698" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"><wsse:Username>wsitUser</wsse:Username><wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordDigest">xhM3irVV+4ROPhzEjwlEeL7IViI=</wsse:Password><wsse:Nonce>oE1wwvm+ZLeBuvgDJvaLvw==</wsse:Nonce><wsu:Created>2009-01-21T02:13:32.238Z</wsu:Created></wsse:UsernameToken></wsse:Security>
</soapenv:Header>
   <soapenv:Body>
      <svc:payload>Maurizio</svc:payload>
   </soapenv:Body>
</soapenv:Envelope&gt;|#]

[#|2009-01-21T03:13:33.783+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|--------------------|#]

[#|2009-01-21T03:13:34.828+0100|SEVERE|sun-appserver9.1|com.sun.xml.wss.logging.impl.filter|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=9ffc6c88-9210-451f-ad64-08ec2bc5ca74;|WSS1408: UsernameToken Authentication Failed|#]

[#|2009-01-21T03:13:35.015+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
---[HTTP response 500]---|#]

[#|2009-01-21T03:13:35.015+0100|INFO|sun-appserver9.1|javax.enterprise.system.stream.out|_ThreadID=25;_ThreadName=httpSSLWorkerThread-8080-0;|
<?xml version="1.0" ?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns3="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">wsse:FailedAuthentication</faultcode><faultstring>Authentication of Username Password Token Failed</faultstring></S:Fault></S:Body></S:Envelope>|#]

Alistair's top ten ways to know you are not doing agile

I stumbled upon the following not-so-new blog entry by Alistair Cockburn, a nice short list of "bad smells" for lack of agile project management. These days companies are still frequently trying to engage project managers asking for arcane list of certifications which are mostly theoretical and paperwork-oriented. Alistair, who has written one of the best book ever about Use-Cases, instead insists on agile methodologies and expert leadership as the only effective way to deliver projects.

In my experience pragmatism, agility and the ability to deeply dig for real business requirements are the most important factors for successful integration projects. For effective consulting is mandatory to keep open a constant, daily communication link with business experts and project stakeholders, relationships with just customer's technical people almost never provide for the necessary unbiased information.

Too many developers are just trying to experiment with the latest "cool" technologies, only because they want to fill a row in their CV, regardless of the real business needs of their employers. In my experience these days two of the biggest CV-fillers are SOAP Webservices and BPEL, as I rarely see those technologies used properly.
Not that they couldn't be useful, but:

"When all you have is a hammer, everything looks like a nail."

SOAP WS and BPEL are the nowadays' hammers.

Buzzwords demystification

Words like "SOA", "ESB", "EDA" are common and, unfortunately, extremely abused. In summary, below a one page description of the main buzzwords floating around.

Architectural Styles

SOA is an architectural style aimed to create a conceptual framework and a collection of practices to improve the alignment of IT to core business needs. SOA is not about technologies. At its core there is the concept of Service, which is an abstract interface to a concrete business function, for example: logistics management, HR & recruiting, stock control, order management, invoicing, ... A Service implementation is governed by a policy, exposes an end-point, implements a defined contract and sends / receives messages. A service is transactionally atomic by definition, otherwise it's a process, which actually is about atomic services orchestration, so a very different and higher-level concept than service itself.

ESB is an architectural style aimed to highly decouple clients form service implementations, by creating an uniquely identifiable space where service policies can be applied uniformly and managed consistently.

SOI Service Oriented Integration is an evolution of classical EAI concepts, introducing an higher level of decoupling among applications by connecting abstract interfaces defined by service contracts instead of mere point to point connectivity usually happening within EAI traditional solutions. SOI = EAI + SOA.

EDA Event Driven Architecture is an architectural style which is the evolution of previous MOM techniques toward SOA concepts, by using one-way asynchronous message exchange patterns. It is a way to model the typical real-time, asynchronous system interactions happening in the real world. Routing, filtering, pipe-lining of messages are the typical tasks of an EDA, activities like CEP (Complex Event Processing) are sometimes necessary.

In Conclusions

Architectural styles and practices are the "what", "who", and "why" of a solution, they shouldn't be confused with technologies, which are the mere "how". Still we read around or listen to people who think that to make his/her IT infrastructure better and more fashionable they need to implement a bunch of SOAP Web-Services + some BPEL orchestration... That is deadly wrong and has driven the acronym SOA to be abused and inflationed and a lot of projects to fail.

Updates: yesterday Paul forwarded me this very interesting article about how Bechtel Corporation is implementing a huge SOA effort to create an internal SaaS infratructure.

The word “SOA” is dead

As usual Anne Thomas Manes wrote a brilliant and controversial article about SOA:

"Although the word “SOA” is dead, the requirement for service-oriented architecture is stronger than ever.

But perhaps that’s the challenge: The acronym got in the way. People forgot what SOA stands for. They were too wrapped up in silly technology debates (e.g., “what’s the best ESB?” or “WS-* vs. REST”), and they missed the important stuff: architecture and services."

I humbly wrote something similar before, I was discussing about governance needs but the problems are almost the same:

"IT is broken: we clearly see this simple and true fact daily, but I hardly find organizations able to recognize the fact that they need to reconcile their IT operations with their business needs first through organizational changes. They too often just try to buy the next magical box and hope that plugging it into their IT department it will fix problems.

Again and as usual: lot of money, few successes and SOA is becoming another abused bad word."

However the concrete requirement, as Anne wrote, is still here and unresolved.

Updates

Anne's article is creating a lot of discussions on blogs, here time by time I will collect some of the most interesting reactions.

Steve Jones's In a recession its even more about the services:

"Reading between the lines of what Anne writes I think I'd say that her statement is that vendors are moving away from SOA as they've flogged you enough stuff and now they want to flog you its offspring: mashups, BPM, SaaS, Cloud Computing
[...]
the marketing fury of T-SOA has moved on as their just aren't that many more ESBs and Web Service tools that you can be sold.
[...]
If you adopt the new technologies without having a services mentality then you will create a degree of mess that will make the one that consultants and vendors got fat on with EAI look like a trivial problem. Doing Spaghetti inside your firewall in big applications is one thing, doing it over the internet and with thousands of small ones is a completely different scale of problem.

So in a recession you need to Identify your services, understand the business value that they deliver, understand the cost model to deliver that value and then decide on the right technology approach.

If that isn't SOA then I don't know what is. So in reality its the "other" SOA that is dead, not the SOA of today."

Yes, that's the point: vendors are looking for the next cash cow. But if you don't want to be treat like a cow you shouldn't behave like a cow and stop eating all the hypes vendors are always throwing around. SOA is actually a good thing, but it is not something new and does not require special tools unless you have all your requirements clear. Is really SOA that different from old object-oriented domain modelling? Isn't that about clean separation of public interfaces from their implementations?

A deep and savy analysis is presented by Gary Barnett:

Essentially - this is what I have to say -

• Saying “SOA is Dead” is as meaningful as saying “Art is dead”
• It is either misleading or foolish (or both) to state that SOA is dead
• Giving up on architecture just because it’s a “chore” is the road to failure…
• If SOA is dead, then Web 2.0 is dead
• The good news, though, is that SOA is as fit as a fiddle
• The bad news is that if you want to succeed, you have to be willing to make an effort
• We have a simple choice: get it right, or do it all over again

Gary is not really distant from what Anne wrote, to me he just thinks her message can be misunderstood. I personally already realized in 2007 that the acronym "SOA" was seen with growing suspicion by customers, so I was suggesting my colleagues to start discussing more concrete aspects. As Gary wrote, in enterprises the need for Architecture is stronger than ever, it is not time to give up but maybe to just re-tune our vocabulary.

Gary writes:

"Architecture is pivotal to the success of service based IT. It’s not a “nice to have”, it’s a fundamental necessity. Unless we as an industry make a real commitment to architecture we’re doomed to continue to make the same mistakes we’ve been making for decades."
[...]
"Take mash-ups, as an example – Sure, you can quickly assemble some fantastic apps by simply grabbing some javascript and coding away. But there are big, big, disappointments in store for mash-up builders who assume that because it’s cool and new they don’t need to think about all those boring questions that a mature approach to architecture helps you to answer (Reliability, security, availability, scalability, re-use etc etc)."
[...]
"As growing numbers of people succeed, we’re learning (or re-learning) a host of lessons about how to deliver the agility, re-use, scalability and general goodness that a service oriented approach to software architecture, design and development can provide."

Read his complete blog article.

Integration Projects Usual Madness

Joel Spolsky wrote a classical article about writing better code. The 12 Joel's rules are below summarized:

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

EAI projects pose additional serious challenges in terms of project management and quality assurance. Even if the term "EAI" seems to be out of fashion nowadays, I think the majority of projects which have been labelled as "SOA efforts" are, in facts, still plain, old EAI.

EAI Quality Assurance and Testing

Especially when doing EAI, unit testing is not enough. Joel is in favour of developer's unit testing and personally I think using a tool such as JUnit can prevent a huge numbers of bugs, but unit-testing integration flows is often much harder than unit-testing regular applications. This is due to the fact that when implementing EAI it is necessary to deal with a multitude of external systems which are out of our domain of control. Mocking those systems is often very hard or even impossible, it could lead to write and maintain more mocks than actually manageable and, additionally, the quality and realism of those mocks could be at least questionable.

Testing activities for EAI teams is a matter of organization and operations, because often it could be a cross-department issue. For example, testing SAP BAPI interfaces requires some commitment and lot of assistance from the SAP team, which in the meanwhile could have different priorities. The same for other legacy technologies. This clearly scales to a management issue, the EAI team has to be allowed to orchestrate the testing and validation process with all the involved external parties. Cross-department activities are usually difficult for organizational reasons, not much for technical ones.

Critical Success Factors

While at SeeBeyond we had some specifically integration-related rules, which were useful to establish what we called "Critical Success Factors":

• The executive sponsorship must be established up front to facilitate timely decision making, appropriate resource allocation and removal of business barriers.
• Establish an integration team with a balanced mix of IT, end business, management and consultants who will work together toward the main goal of a successful implementation
• Provide just-in-time training and coaching to the appropriate people on operation of the product components, keeping them ahead of the learning curve,
• Field level buy-in to the implementation is critical to the success of the project
• Executive management must communicate their commitment to the new technology and illustrate this commitment through their actions.
• Availability of key stakeholders to facilitate the decision making process and to keep the implementation moving forward.
• A complete test and validation plan should prepared in advance. It is not up to the single developer to validate his own work, QA is an overall function.
• As this is an integration project, all interfaces must be defined before starting the implementation.
• Changes have to be accommodated into a change management process which takes care of the deadline. All the interfaces should be unit-testable and end-to-end testable as being part of a higher level process.
• Implementation will proceed by very short iterations of analysis, design, development, test and deployment. At the end of each iteration a working prototype should be ready to run functional validation tests.

Regarding SOA, reports say that most SOA initiatives are failing because of inadequate business involvement and commitment. EAI is a better understood discipline (really?), but still projects tend to fail for exactly the same reason as newer SOA efforts.

Missing Deployment Plan

Just after the functional specs and scope, it is important to define a Deployment Plan and a Deployment Architecture:

"The deployment project plan establishes a baseline reference plan. It is your road map for the deployment project. The roles and responsibilities of each organization, the schedule of tasks, and any estimates must be defined in this plan.
[...]
Deployment Architecture (DA), together with Requirements Analysis, is the most important component of your plan, and should be documented carefully. Deployment begins when you plan and schedule how, in view of your analysis and allocated resources, you want to implement your environment. During this phase, you set up the operation procedure and schedule for the entire deployment project. DA provides guidance to the successive design and implementation phase, so DA creation always must precede it. In fact, developers needs to use the DA document as a guide for implementing their flows, adapting services communication to the DA. Of course, sometime it is necessary to adjust the DA to accommodate design needs emerging after some development, but if this happens frequently then it is a sign of a bad original DA, so it is time for a deep review.
Poor analysis and planning can cause major problems during the later phases. Thorough, comprehensive analysis and planning can make the deployment project easier, more efficient, and less costly."

The above is a mix of statements from the Java CAPS documentation plus some comments of mine.

Organizational Issues

In my experience most of the issue I have encountered during Integration projects are organizational and operational:

• Lack of organizational information
• No complete specifications
• Not enough back-end information to fill the specifications
• Lack of coordination
• Frequent changes in project organization (document store moving, too many documents everywhere, anonymous doc.)
• Huge integrations are thrown to single developers
• Lack of commitment
• Sometime is difficult to understand who is in charge
• Developers are frustrated by changing deadlines
• Bad or totally missing test data!!!

Missing Domain Model

You wouldn't write SQL queries before modelling any database relationship, would you?
An usual requirement for Services is to build first a Domain Model as a common, business oriented standard communication device. This is very close to good Object-Oriented programming practices, but Service-Oriented is focused on exchanging documents instead of objects. A domain model can be designed by normal entity-relationship diagrams and, for example, implemented through XML-Schemas.

Common Misunderstandings

• Integration projects are like software development projects

>> No! they are very different!
>> Integration projects are cross-department projects
>> Need more involvement
>> Need more cross-management

• Bugs in software are all the evil

>> No! Half of bugs are not real platform's bugs but behaviour due to bad design/assumption.

Integration project with any vendor will fail if there are no clear specifications...

Inter-team communication madness

• Every team has its own life
• Not coordinated at all
• No motivations to help
• Mostly Anonymous functional and technical documents !?!

Finally... Some Recommendations

• Long term strategy but short delivery cycles
• Top-down, business-driven analysis
• Short iterations, deliver value within few weeks
• Break big projects into smaller pieces
• Stick to deadlines! Avoid changing a deadline!
• Instead, what features can fit into the deadline?
• Adopt a Project Management methodology!

Some of the above statements could seem controversial. What about deadlines? there is nothing more frustrating for a developer than working 80 hours per week, weekends also, just to discover that a couple of days before deployment some manager decides that the feature is not really that important and can be delayed. Boy, you are not going to catch me anymore! So my advice is: fix the deadline and instead move in and out features, keep the pace, sustain the mood.

Always ask ourself why? "Why?" is extremely powerful: before start fighting to solve a technical issue one should ask why I'm doing this? Is there any better, cleaner design which could alleviate or even cancel the technical issue? Do I really have all the necessary information? Am I answering a question nobody asked for?

Conclusions

Integration projects are somehow quite different from application development projects:

• Interfaces and meta-data must be clearly defined
• Service contracts must be defined (SLAs)
• Test data must be readily available!
• External systems experts must be available!
• An architectural board in charge of taking decisions needs to be established.

It seems that the above list is exactly what we need in case of any project, application development or integration. But badly-managed dependencies with the external world can sink any integration project regardless of how good is the team and its internal process. Integration is about connecting existing interfaces: what can I do if interfaces are fuzzy and undefined? Frustration, that is....

Thanks goes to my friend Juraj Kazda of Sun, for discussing and writing down these issues while working together in several projects.