JMS Over HTTP Using OpenMQ

DZone has an interesting article about new OpenMQ's features:

"OpenMQ provide different transport protocol and access channels to access OpenMQ functionalities from different prgramming . One of the access channels which involve HTTP is named Universal Message Service (UMS) which provide a simple REST interaction template to place messages and consume them from any programming language and device which can interact with a network server. UMS has some limitations which is simply understandable based on the RESTful nature of UMS. For current version of OpenMQ, it only supports Message Queues as destinations so there is no publish-subscribe functionality available."

Read the full article.
About OpenMQ.

stcqueueviewer: Dinamically Monitoring JCAPS JMS Server Queues

Disclaimer: the below procedure is undocumented and not officially supported, so you are using this at your own risk only. Please contact Sun's JavaCAPS support or Professional Services for more

I recently had to dynamically monitor some JMS queues within JavaCAPS , especially my need was to count the number of messages in a given queue before posting additional messages, to avoid unnecessary queue flooding. JavaCAPS is fully JMX compliant, the embedded Sun Application Server is well-documented on that side but the SeeBeyond IQ Manager (default JavaCAPS' JMS server implementation) is lacking some info, so I decided to go for some hacking.
My colleague Paul pointed me in the right direction, he suggested to have a look at the com.stc.jms.stcqueueviewer.jar contained into

logicalhost\is\stcms\lib

This library is almost undocumented, so I had to reverse-engineer its classes by using the nice DJ tool. The most interesting file is called Server.java, the Server class contains what is necessary to fully monitor the SeeBeyond IQ manager from a Java program. Below an example:

package queuemonitor;

import com.stc.jms.queueviewer.*;

public class Main {

    public static void main(String[] args) throws Exception {

        Server sv = new Server();

        sv.connect("localhost", 18007, "Administrator", "STC");

        QueueStatistics queueStatistics = new QueueStatistics();

        sv.getQueueStatistics(queueStatistics, "qStoreDocument");
        System.out.println("MinSeqNo=" + queueStatistics.MinSeqNo);
        System.out.println("MaxSeqNo=" + queueStatistics.MaxSeqNo);
        System.out.println("MessageCount=" + queueStatistics.MessageCount);
        sv.disconnect();
    }
}

To compile and run this with Netbeans 6 you need to import some additional JAR files from the JavaCAPS Logicalhost logicalhost\is\stcms\lib folder:

• com.stc.jms.stcjms.jar
• com.stc.jms.stcqueueviewer.jar
• jms.jar

By the way, if you have not tried the new Netbeans 6 yet, then shame on you!

If you want instead to do the same within a JavaCAPS JCD you need to execute the following steps:
1. create a new JCD
2. Import the com.stc.jms.stcqueueviewer.jar JAR file into repository
3. Import the above JAR into the JCD


4. Do something useful with the dynamic information you get


The above JCD example is pretty silly, just to show you something

That's it. There other useful methods into Server class, you can explore and discover other interesting things. Hopefully somewhere more queueviewer's documentation is available, so it won't be necessary to decompile the Java classes...

HermesJMS with JCAPS

Here you can find how to setup HermesJMS with JCAPS.

EAI AntiPatterns: Using a JMS Server Like a Database

Do you know a very simple rule of thumb to verify the health of your integration flows? Well, everything could be consider reasonably fine if queues in your ESB are on average empty or quite close to be empty. I have heard many times in different projects complaint like "this JMS server doesn't work well, I have about 10,000 messages in a queue and everything looks so slow...". 10,000 messages parked in a queue? You have a problem here and it is not the fact that the JMS server is not very good at dealing with that, simply your flows are unbalanced and your overall design is broken! Basically your consumers are much slower than your producers so messages quickly accumulates in the system. But a JMS server is not a database, is definitely not for long term storage. You can't easily query messages in your queues as you could do in a database, you can't easily delete or edit them. Additionally, a good JMS server like the SeeBeyond IQ Manager, which is the default JMS implementation in JCAPS, by default activates message server throttling. It means that if persistent messages in the server go beyond a certain threshold the single producer or even the entire JMS server are stopped until a proper consumer lag is reached. For the SeeBeyond JMS these values are by default 1000 messages per single queue, after that messages producers are freezed, and 100,000 messages for the whole server, after that all the producers connected to that particular JMS server are stopped until a certain amount of messages are properly consumed. This is a safe net to avoid producers flooding the JMS server.

Probably the guy above complaining about the 10,000 messages knew about this throttling feature and he decided to simply increase the default threshold. This is definitely a bad idea, he needs to fix the balance of his flows and really understand what is happening in his system instead of looking for easy workarounds. The default limit of 1000 messages per queue is there for a good reason, and it is that a JMS server is not a storage device! It is a quick asynchronous delivery mechanism instead, where messages must stay in a queue for the minimum possible time. The message persistence is there for a complete different reason: it is a way to avoid message losses in case of a temporary hardware or software failure of the messaging system, that's it (oh well, you need an highly available filesystem for that, otherwise that becomes your new single point of failure...). If too many messages are usually staying in the system for a long time you'll notice a proliferation of .dbs files under your stcms folder. Briefly these files are where the IQ manager stores persistent messages, when they are too many all the system becomes inefficient because of files segmentation and reallocation (yes, there is a kind of garbage collection in action, and you want to avoid too much of that).

Then you can say: "but in my flows consumers are slower than producers by nature". This can be easily true because, for example, consumers are performing slower I/O operations with external systems (quite the norm in EAI). So, even if with JCAPS you can easily deploy a set of distributed consumers, this would solve the problem only if the processing is CPU-bound, but not if it is I/O-bound, as scaling horizontally does not help very much if external systems are inherently slow. So what? Well, your flows looks too me not to be nearly real-time, requesting an asynchronous delivery, but instead nearly batch. You need to take control of this and design a proper solution instead of complaining against the technology you are using (you might say that some other JMS servers can be configured to persist messages into a regular relational database. The bad news are that this does not solve your design issue at all: you are using a messaging solution the wrong way, regardless of the underlying persistence mechanism of your particular JMS vendor).

The solution? You should apply the "store and forward". Store your incoming messages into a regular database table then forward them into the destination queues using a more controlled process, moving batches of messages using a scheduled procedure, keeping your consumers busy but without flooding queues. You then might need to implement a reconciliation service. This depends on the context, but probably you would like to know how many messages entered and exited your system, and you probably like to have the possibility to re-submit or delete messages. A database table is a good fit for it, you can decide to store some additional information into table's fields and the run queries on it. Your reconciliation service will expose counters so that you know how many messages traveled through your EAI flow, for each single processor (a JCD in the JCAPS jargon). You than can decide to delete messages from the database when they are picked up by the first consumer or you might prefer to do so only at the very end, when all your process is successfully completed. This second solution could allow to remove the persistent flag from all the intermediate queues in the processing pipeline, to further speed up the JMS server, but this is a design consideration quite dependant on the applicative context. For example, in some scenarios could be simpler and cheaper to repeat the whole process from the beginning in case of failures, instead of maintaining an intermediate state, but in some other scenarios it could be too expensive to repeat and so it is mandatory to store intermediate processing results within the message itself, so that it must be stored in a persistent queue.

JUnit testing JMS systems with Netbeans and Mockrunner

Scenario

You are developing a JMS-based application and you want to use those nice test-driven development habits that you have learned are so good in keeping your code clean and bug-free, but sending messages to a real JMS server is slowing down your edit - compile - test process. First because you always need to remember to start your JMS server each morning, second because running JUnit against the real system is a bit slow. Maybe you have an external server hosting your JMS server, so the first point does not apply to you, but in my case I use to run everything on my laptop, so it does matter.

A mock JMS implementation

Here is where Mockrunner comes in the scene.

"Mockrunner is a lightweight framework for unit testing applications in the J2EE environment. It supports servlets, filters, tag classes and Struts actions and forms. Furthermore it includes a JDBC, a JMS and a JCA test framework and can be used in conjunction with MockEJB to test EJB based applications.

Mockrunner extends JUnit and simulates the necessary behaviour without calling the real infrastructure. It does not need a running application server or a database. Furthermore it does not call the webcontainer or the Struts ActionServlet. It is very fast and enables the user to manipulate all involved classes and mock objects in all steps of the test. It can be used to write very sophisticated unit-tests for J2EE based applications without any overhead. Mockrunner does not support any type of in-container testing."

Actors

• Netbeans 5.5
• Java CAPS 5.1 default JMS server
• Mockrunner
  

Setup

Download the mockrunner-0.3.7.zip file and unzip it somewhere. Then you need to setup your Netbeans project to make use of Mockrunner's libraries.


Add Mockrunner libs to your project's test library by right-clicking the libraries node in your project and selecting "Properties"


In my case I'm connecting to CAPS JMS server, so I need both jms.jar and com.stc.jmsis.jar files. The latter is part of the CAPS eGate APIkit distribution and allows to connect to Sun SeeBeyond JMS server implementation from a generic Java client: you need to put here your specific JMS server client libraries. My intention is to run my custom Java application against the real JMS server (as said, in my case it is CAPS), but to run automatic JUnit tests against the mock implementation.


Now add Mockrunner's necessary libraries to the "Compile-time Test Libraries" tab:



The final result should looks like the picture below:

A simple test-case

Here I show a simple test for the "sendText" method of my class which uses Mockrunner's stubs. Mockrunner internally uses MockEJB libraries to simulate a J2EE container:

/*
 * TesterTest.java
 * JUnit based test
 *
 * Created on 28 November 2006, 17:33
 */
package it.stc.meter.jms;

import com.mockrunner.ejb.EJBTestModule;
import com.mockrunner.jms.JMSTestCaseAdapter;
import com.mockrunner.mock.jms.MockQueue;
import it.stc.utils.Config;
import java.io.IOException;
import java.util.List;
import javax.jms.JMSException;
import javax.jms.TextMessage;
import javax.naming.InitialContext;
import javax.naming.NamingException;

public class TesterTest extends JMSTestCaseAdapter {

    private EJBTestModule _ejbModule;
    private MockQueue _queIN;
    private MockQueue _queOUT;
    private InitialContext _initialContext;

    public TesterTest(String testName) {
        super(testName);
    }

    protected void setUp() throws Exception {
        super.setUp();
        _ejbModule = createEJBTestModule();
        _ejbModule.bindToContext("connectionfactories/queueconnectionfactory",
                getJMSMockObjectFactory().getMockQueueConnectionFactory());
        _queIN = getDestinationManager().createQueue("quIN");
        _queOUT = getDestinationManager().createQueue("quOUT");
        _ejbModule.bindToContext("queues/quIN", _queIN);
        _ejbModule.bindToContext("queues/quOUT", _queOUT);
        _initialContext = new InitialContext();
    }

    protected void tearDown() throws Exception {
    }

    public void testSendText() throws IOException, NamingException,
            JMSException {
        final String text = "prova";
        final int numSamples = 3;
        Tester m = new Tester(_initialContext, numSamples);
        m.sendText(text);
        verifyNumberOfReceivedQueueMessages("quIN", 3);
        List receivedMessages = getReceivedMessageListFromQueue("quIN");
        for (Object message : receivedMessages) {
            assertEquals(text, ((TextMessage) message).getText());
        }
        verifyNumberQueueSessions(1);
        verifyAllQueueSessionsClosed();
        verifyQueueConnectionClosed();
        verifyAllQueueSessionsCommitted();

    }
}

Conclusions

This article briefly shows how to setup Netbeans to make use of Mockrunner to test JMS clients without the need to connect to the real JMS server. In a future article I'll go deeper on the CAPS JMS server facts and I'll show how to apply this scenario to create automatic JUnit test cases for EAI flows built in SeeBeyond ICAN 5.0 or Sun CAPS 5.1. Automatic testing of EAI solutions is usually sligthly more complex than testing normal applications.

JMS topics and message selectors in CAPS

A topic is a JMS message destination that conforms to the publish-and-subscribe messaging paradigm. In this tiny example I show how to publish a message to a topic in Java CAPS, using message selectors to filter message destinations.

Publisher JCD
This JCD (Java Collaboration Definition in CAPS idiom) is triggered by a File eWay and publishes a JMS text message to a topic

public class jcdPublisher
{

    public com.stc.codegen.logger.Logger logger;

    public com.stc.codegen.alerter.Alerter alerter;

    public com.stc.codegen.util.CollaborationContext collabContext;

    public com.stc.codegen.util.TypeConverter typeConverter;

    public void receive( com.stc.connector.appconn.file.FileTextMessage input, com.stc.connectors.jms.JMS JMS_1 )
        throws Throwable
    {
        String content = input.getText();
        com.stc.connectors.jms.Message msg = JMS_1.createTextMessage();
        msg.storeUserProperty( "dest", content );
        msg.setTextMessage( content );
        JMS_1.send( msg );
    }

}

the line

msg.storeUserProperty( "dest", content );

stores the user property that will be used later by message selectors to filter messages.

Consumer JCD
This JCD simply receives a text message from the JMS source (a topic, but it is not specified here but in the Connectivity Map) and writes it to a local file using the File eWay

public class jcdConsumer {

 public com.stc.codegen.logger.Logger logger;

 public com.stc.codegen.alerter.Alerter alerter;

 public com.stc.codegen.util.CollaborationContext collabContext;

 public com.stc.codegen.util.TypeConverter typeConverter;

 public void receive(com.stc.connectors.jms.Message input,
   com.stc.connector.appconn.file.FileApplication FileClient_1)
   throws Throwable {
  FileClient_1.setText(input.getTextMessage());
  FileClient_1.write();
 }

}

Connectivity Map
In the connectivity map there are four services:
* svcPublisher contains an instance of jcdPublisher
* svcConsumer1, svcConsumer2 and svcDefaultConsumer all contain a copy of jcdConsumer



svcPublisher receives a file content from FileIn (File eWay instance) and publish it into tpcPublic topic. the three consumers receives messages after they are selected by message selectors.
In this toy example the input text file which triggers the process contains just a list of numbers:

4
1
2
2
1
2
3
1

message selectors are built so that they filter messages using the "dest" user property

Message Selectors
svcConsumer1 consumes only messages with an user property dest='1'



svcConsumer2 consumes only messages with an user property dest='2'



svcDefaultConsumer consumes only messages with the 'dest' user property that is neither '1' nor '2', so it can be expressed as

NOT (dest='1' OR dest='2')

Deployment Profile
The DP maps project components present in a Connectivity Map to a given Environment



Results
Results are written in three distinct files: output1.txt (it will contains only '1'), output2.txt ('2') and outputDef.txt (contains all the other numbers that are not '1' or '2', present in input.txt)

Conclusions
What happens if one of the Connectivity Maps link does not contain any message selector? The obvious response is that the connected consumer would receive *all* the messages, regardless of the user property stored into the message header.

Java CAPS - eInsight Business Processes Correlation HowTo

The Java Composite Application Platform Suite

The Java Composite Application Platform Suite (Java CAPS) allows companies to assemble large-scale applications built on existing systems and infrastructure. Java CAPS is an application-level network that unifies connectivity among people, application systems, and devices in different locations and across organizations. Business services facilitate the implementation of extended applications. Service oriented architectures (SOA) clarify design and enable reuse by sharing logic and data among different client systems and users.

eInsight and Java CAPS

eInsight is a component of Java CAPS. eInsight delivers Business Process management features and functions to Java CAPS. Business Process management is a strategic orchestration of the movement of information and the flow of complex processes between participants (systems, users, and organizations) to accomplish larger business objectives. eInsight uses the standard BPEL language to describe processes

1 Introduction

eInsight provides the means for matching existing Business Process instances to messages that are arriving into a Business Process. Correlation keys are individual data values contained within both the incoming message and the eInsight engine. When arriving messages contain data that matches the configured correlation keys, unique Business Process instances then continue processing on to the next step of a given Business Process.

2 Message Correlations

A correlation key is a value that you can assign to a Business Process, like a Purchase Order number. The correlation key provides a way to associate and route information about specific Business Process instances. For asynchronous message exchange between components, you must implement correlation of the instance identification.

An example of when you use asynchronous message exchanges is when you create a Receive Activity in the middle of a Business Process.

To have the correlation mechanisms working properly, right correlation keys are mandatory.

3 Example 1 - File trigger

In this scenario we have a calling process that sends a JMS message to another process and then waits for a reply, correlating instances through a unique ID.

Invoked Process - bpReplier

The bpReplier process is called from the main one. It is very simple for this example: it just copies the Jms Correlation ID and adds some information in the text field.

Main Process

The main process reads a correlation ID from a file, set it into the CorrelationID field of JMS->MessageProperties and then sends the message to the above mentioned bpReplier. An “Event Based Decision” waits for events: one it is a JMS.receive, second is a “Timer Event” in case nothing is received within a fixed amount of time.

Correlation key

The correlation key is made of two aliases:
* /Message/MessageProperties/CorrelationID
* /FileTextMessage/text


The correlation is used in two activities in the BP:
1 - FileClient.receive - with Initialize Set ='yes'
2 - JMS.receive (in the event based decision) - with Initialize Set='No'

4 Example 2 - JMS trigger

The second example shows how to use correlation in a process triggered by a JMS message instead of a file. The invoked service remains the previous simple "bpReplier", while the main process is slightly different.

Main Process

In this case the main process is split in two: first the trigger, which reads from a file and uses its internal BPID as the CorrelationID for the JMS message:

Second, the “real” main process:

This receives the above message, initializes the correlation key, send a message to the bpReplier (copying the received CorrelationID) and the waits for an event. The JMS.receive, which uses the correlation key but does not initialize it, in the event based decision correlates inbound messages, while the usual Timer Event would raise a timeout in case of long inactivity.

In this case, as we only have one kind of inbound event (a JMS message is received) the only necessary alias for the correlation key is:
/Message/MessageProperties/CorrelationID

Resources

* Sun Java CAPS public forum
* Product's official documentation