Tuesday, November 13, 2012

RESTful API memo: PUT and POST differences

Before start designing a RESTful API, have a look at Hypertext Transfer Protocol -- HTTP/1.1, section 9

"The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line.".

In other terms, POST is meant to handle appends to existing resources or incremental creations of subordinate resources:

"The actual function performed by the POST method is determined by the server and is usually dependent on the Request-URI. The posted entity is subordinate to that URI in the same way that a file is subordinate to a directory containing it, a news article is subordinate to a newsgroup to which it is posted, or a record is subordinate to a database."

PUT instead seems is more appropriate to handle one-shot creations, creating or replacing an entire resource in one single transaction:

"The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI."

Differences between PUT and POST:

"The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server MUST NOT attempt to apply the request to some other resource."

Another remarkable difference is that PUT requests are required to be idempotent, while POST are not:

"Methods can also have the property of 'idempotence' in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property. Also, the methods OPTIONS and TRACE SHOULD NOT have side effects, and so are inherently idempotent."

Saturday, November 10, 2012

Creating your private Git repository on Dropbox in less than 5 minutes

Github is the tool I use daily to manage my public software projects, I love it. But sometimes I have to quickly and temporarily share private projects with colleagues or maybe even in a mixed environment, with customers and consultants from other companies. When there is no time / money to buy private remote repos from Github or even install a local Git repo on some server,  and for privacy constraints it is not possibile to publish the code on a public Github repo, then Dropbox comes to the rescue.

In this example I'm working on a simple Web application in Flask, which is a cool Python micro-framework. I created a "flask_sample" folder which contains the code I want to version with Git and share with other colleagues.

I promised it will take less than 5 minutes, so let's start.

Move to your Dropbox folder (in my case it's in /Users/mturatti/Dropbox/) and create a folder to host all your remote git repositories:

$ cd /Users/mturatti/Dropbox/
$ mkdir git

Then create here the folder to host this remote repository:

$ cd git
$ mkdir flask_sample.git
$ cd flask_sample.git

It's time to create a bare Git repository:

$ git init --bare

You'll see it creates a structure similar to the following:

mturatti:~/Dropbox/git/flask_sample.git$ ls -l
total 24
-rw-r--r--   1 mturatti  staff   23  9 Nov 18:38 HEAD
-rw-r--r--   1 mturatti  staff  112  9 Nov 18:38 config
-rw-r--r--   1 mturatti  staff   73  9 Nov 18:38 description
drwxr-xr-x  10 mturatti  staff  340  9 Nov 18:38 hooks
drwxr-xr-x   3 mturatti  staff  102  9 Nov 18:38 info
drwxr-xr-x  11 mturatti  staff  374  9 Nov 19:09 objects
drwxr-xr-x   4 mturatti  staff  136  9 Nov 18:38 refs

Now you have in place a git structure which can act as a shareable remote repository, even if in practice it's local to your hard disk. Being a Dropbox folder will do the magic in terms of backups, sharing and synchronization.

Initialize Git in your software project as usual (in my case the local project stays in /Users/mturatti/src/flask_sample)

$ git init

This creates the usual hidden .git folder.
The last configuration step is to add locally the previously created remote Git repository:

$ git remote add origin file:///Users/mturatti/Dropbox/git/flask_sample.git

Note we are using the file:// protocol for the remote Git repository here.
If you check the content of .git/config file you'll see the new origin (in bold below):

mturatti:~/src/flask_sample$ cat .git/config 

repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
precomposeunicode = false
[remote "origin"]
url = file:///Users/mturatti/Dropbox/git/flask_sample.git
fetch = +refs/heads/*:refs/remotes/origin/*

At this point you can start the usual Git lifecycle. For example, after you have added and committed all your files locally, you can "push to origin", which will push your code to your remote Git repository saved on Dropbox:

$ git push origin master

The last step will be to share the Dropbox folder with your colleagues, so that they can also add this as a remote repository and start cloning / pulling / pushing from this origin.

Saturday, September 29, 2012

Time, Cost, Quality and Agile Consulting

Types of Consulting Engagements

“Sometimes it's a little better to travel than to arrive” 
― Robert M. PirsigZen and the Art of Motorcycle Maintenance: An Inquiry Into Values
In my software consulting experience I have been engaged in many different kind of projects, but in all cases they fall into two main categories:

  1. Time and Material (T&M)
  2. Fixed Price (FP)
In most situations it happens than to decide that option #2 is achievable, then a quantity of T&M analysis must be performed in advance, in order to define the context and the scope for a possible, successive FP engagement. That's not always possible: for example, when the project is part of a public tender, you have to bid for the lowest possible price, trying to balance the need for adding a good amount of contingency, staying into the safe path, without self-sabotaging the possibility of winning the tender.

What marks the difference between T&M and fixed price? In T&M a customer is basically paying for your time, because deliverables and scope can't be clearly set in advance, or because it's already established that requirements are going to change in a way that a Fixed Price engagement is out of question, because it's too risky. A Fixed Price project is based on a set of much more stringent assumptions, in terms of context, requirements, functional and technical, which (hopefully) allows for a very accurate estimate of deliverables.

Usually customers are more keen on FP because, of course, they think it will constraint the final price by putting much more responsibilities on the consultant, while T&M seems a way to create a continuos stream of expenses. However, in reality, there is a more fundamental law which regulates any kind of software project, despite the rules of engagement, and it is related to the existing and unavoidable strong relationship among three distinct, fundamental quantities: Time, Cost and Quality.

The Basic Conjecture of Time, Cost and Quality

“When analytic thought, the knife, is applied to experience, something is always killed in the process.” 
― Robert M. PirsigZen and the Art of Motorcycle Maintenance: An Inquiry Into Values
It's almost incredible how many people actually think they can leverage Fixed Price to be in total control, at the same time, of these three quantities:
  1. The elapsed time spent from start to finish, so the final delivery date;
  2. The total cost, in terms of direct money and indirect materials;
  3. The overall quality of the final product.
This assumption has been historically proven false by practice, for any non-trivial software project or consulting engagement. It's a conjecture and not a theorem or a physical law, but reality has taught me that, in software development and software consulting, it is possible to accurately be in control of only two over three of these quantities.

Some examples are needed: if a project has a fixed delivery date and a fixed price, then the only left quantity one can possibly control is quality. Would it maybe explains why so many FP projects suffer from poor perceived quality?
On the other hand there is another fundamental speculation which states that "Nine women can't deliver a baby in one month". It means that, if delivery dates and quality are fixed, one is tempted to keep adding resources, in terms of people and infrastructure, loosing control about costs. In practice this tactic even leads to also increasing delivery time, because adding people on a late project usually delays it even more.
A third case is when we try to fix both cost and quality, but then we accept that elapsed time can't be predicted accurately. This is the case, for example, of companies trying to outsource development to offshore facilities, where cheaper labor force can easily be hired. Statistically this strategy has led many projects to both an indefinite development time but also poorer quality.

It's all about one single truth: software development is inherently not a traditional engineering activity. Actually, Programming is Gardening, not Engineering

Agile Development and Agile Consulting

"Simplicity is the ultimate sophistication". ~ Leonardo da Vinci.
So, at first sight, it seams there is no escape from the T, C & Q rule. But we are not doomed. In fact, as I wrote before, the rule applies for any non trivial software project. So the trick here is: to transform big projects or big consulting engagements into a finite sequence of very focused, well defined, little activities or mini-engagements. This is why time-boxing or feature-boxing usually work effectively, and that's what actually Agile Methodologies are, more or less, trying to achieve: transforming complexity into something more manageable and predictable, by splitting big activities into little, possibly trivial, short tasks, which can be handled in few hours or days by very few people.

I think that agile methodologies can and should be successfully applied also to pure software consulting, so to the kind of engagement usually performed in T&M. The main pillars of this strategy are nothing new and can be summarized as:
  • Focus on User Stories;
  • Short iterations, usually no more than two or three weeks long;
  • Continuos Integration and Continuos Delivery of valuable pieces of software;
  • Acceptance tests at the end of each iteration or when a single deliverable is ready.

As a side note: if you are strictly required to be on-site then that is not a Fixed Price project by definition! Fixed Price engagements MUST be off-site, exactly because you don't want to waste time renegotiating the scope each single moment. It is necessary to have customers involved daily and keep things very flexible, but asynchronous interruptions must be avoided at all costs. The main objective is to understand what final users want and adapt when requirements are changing, but this must be addressed by the process, not individuals. T&M is very different: as customer pays for your time he is entirely entitled to interrupt you and change tasks even in the middle of them. That's why committing on any detailed deliverable in a T&M assignment is extremely dangerous.

The Need for Good Architectural Decisions

"We are searching for some kind of harmony between two intangibles: a form which we have not yet designed and a context which we cannot properly describe." ~ Christopher Alexander.
The missing piece, for a Use Case made of several User Stories, is a comprehensive and reasonably complete Technical Architecture. In other words, I do believe in that kind of bottom up, emerging software design coming from an Agile, iterative process, but I think this must be developed within the frame of a clear up-front Architecture.

I mean that refactoring code and design is not only necessary, but even desirable. Then, in my experience, refactoring wrong initial architectural decisions can be extremely expensive and usually leads to big failures. I strongly believe that the role of an experienced architect is key to produce quality software systems, and this fact sounds to me to be often too underestimated in the field of Agile Methodologies. Do not fool yourself by believing your so-called "rockstar developers" (horrible term!) alone can also actually imagine, design and implement a complete and working technical architecture.

Speaking of consulting, even a short T&M engagement should be performed in the context of a well designed architecture, because even the best expert can be unable to deliver anything useful if the architectural context is broken. Focus your first steps at customer site on two main things: understanding their existing architecture and development process, and start fixing them if they are clearly broken. Otherwise the risk of failing and not get paid will be too high, despite the fact it's T&M or FP.

Tuesday, April 10, 2012

Managing Multiple JDK on Mac OS X

Recently I had to install the OpenJDK 7 on my Apple MacBook, but keeping the original JDK 6 as my main Java environment. After browsing the Internet I came to a decent set of instructions (Tested with OS X Lion 10.7.3).

In summary:
  1. Get the OpenJDK from http://code.google.com/p/openjdk-osx-build/ (I chose JDK 7u4, which at present seems to be the latest stable build); The JDK 7 is now a regular download by Oracle: http://www.oracle.com/technetwork/java/javase/downloads/index.html
  2. Install the downloaded .dmg package;
  3. Change your Java Preferences accordingly, by moving on top the "OpenJDK 7" item (by default Java SE 6 is the first item - see below picture);
  4. Automatically set the JAVA_HOME variable, so that shell tools work.
To automatically setup the JAVA_HOME variable it's necessary to add few lines to the .profile.
So, edit this file (it's in your home directory) adding the following lines:

# Change your JAVA_HOME
function setjdk() {
   if [ $# -ne 0 ];
      then export JAVA_HOME=`/usr/libexec/java_home -v $@`;
   java -version;
# Automatically set the JAVA_HOME
export JAVA_HOME=`/usr/libexec/java_home` 

Note that the optional setjdk function allows for dynamically changing the JAVA_HOME if you switch items in your "Java Preferences", otherwise many Java tools won't work if JAVA_HOME is not in synch with the System settings.

Whenever you change the default JDK using the "Java Preferences" tool, then any new terminal will automatically pick-up the new JAVA_HOME by executing /usr/libexec/java_home, so executing the setjdk function is not usually necessary, unless you really don't want to close and re-open the terminal (opening the terminal reloads the .profile). Alternatively issue the command source .profile in you shell.

That's it, now you can install multiple JDKs and select them dynamically, by just using the "Java Preferences" tool, without touching any system file by hand.

Saturday, April 7, 2012

XML Schema and WSDL modules for Netbeans 7.x

A couple of years ago I wrote a blog entry about how to install the missing XML Schema Editor and related utilities from the dev update center in Netbeans 6.9. Now there is a unofficial update center:


It contains the development branches of these and instructions on how to install, thanks to Geertjan Wielenga.

I installed the plugin on Netbeans 7.1 [update: I installed it also in 7.2] and it seems to work, even if I did not test it intensively, primarily because these days I'm no more working that much with XML Schemas and WSDL files.

There is an apparently disabled Hudson project for the XML Tools. Now, if you want to put this nice plugin back into the regular plugin repository, please vote for this issue!