Jian Wu's Tech Notes: It's an Event-Driven World

Migrate Web Service From XFire to Apache CXF

2008-09-20T23:45:00.000-07:00

Recently, we migrated our Production Web Services from XFire 1.2.6 to Apache CXF 2.1.1, and it went very well. So I want to share some good experience using Apache CXF.

Why We Want to Migrate:

1. XFire Project itself has been discontinued

Accoring to XFire and Celtix Merge FAQ, XFire Open Source Project has been discontinued, Apache CXF is a continuation of the XFire project and is considered XFire 2.0, which has many new features and a ton of bug fixes.

2. Using Spring 2.5 to develop Java Middle-tier Server

Spring 2.5 has become a de facto standard for Java Middle-tier Server Development, and it is also adopted internally as standard framework to build Java Middle-tier Servers. But, XFire is built and bundled with old Spring 1.0, which becomes a road block preventing us from using some new Spring 2.5 features, such as Spring AOP. Migrating to Apache CXF 2.1.1 will immediately solve this problem since Apache CXF is built on top of Spring 2 and can work with SPring 2.5 without any problem.

3. Using new CXF Features, which are not available in XFire:

Since Apache CXF is JAX-WS compliant, it is safe to use CXF JAX-WS Front-end, developing Web Services using JAX-WS Annotations

We want to inject certain HTTP Headers at both Client and Server side for network routing and load balancing, and this should be separated from the business logic of Web Services. CXF Interceptor Framework is a perfect tool to implement this kind of features

CXF supports both dynamic Javascript WS Client and statically generated Javascript client, this actually enables us to build a simple soapUI-like tool inside web browser and is working perfectly with our light-weight jQuery based Admin Tool/Console for monitoring and debugging

Issues We Encountered:

Obviously, there were some CXF issues we have to resolve:

1. Light-weight SOAP Message Validattion

For the backward compatability with existing XFire Client/Server, we are using CXF JAX-WS Front-end with Aegis Data Binding, we want to have a simple light-weight SOAP Message Validation as alternative to full-fledged Schema Validation. This actually is achived by adding a "XML Validation Interceptor" to the right place in the CXF Interceptor Chain. You can find more detail here from the CXF User Mailing List

2. WS Client Connection Pool

XFire WS Client is built on top Commons HTTP Client 3.0, which allows us to set up a connection pool to control the concurrent requests and the load at the client side. But, since CXF is not using Commons HTTP Client as Client Transport Layer, this is not available from CXF anymore.

This problem was partially resolved by building a simple CommonsHttpClientTransportFactory with a simple CommonsHttpClientConduit. It would be nice that CXF can provide a full-fledged CommonsHttpClientConduit. More detail can be found here and here from the CXF User Mailing List

Apache CXF Actually Performs Better:

To evaluate the performance of CXF, We did one round of load testing by puting two WS Servers behind one Load Balancer, one using XFire and another using Apache CXF, the results show that the performance of CXF is compatable with XFire.

We also run the similar deployment in the real production environment to compare the performance, and our new servers using CXF are wroking very well. We actually see the CPU Utilization on the Servers using CXF is less than the existing servers using XFire. This is a little good surprise for us since we originally expect CXF should consume more resources given the fact that CXF has to build two intercept chains, "in" and "out", for each request.

It's All About the Traceability

2007-09-05T01:16:00.000-07:00

I just come across a very interesting blog Log Everything All the Time talking about logging information in a production environment.

As a software developer, I just want to further elaborate on this:

1. It is all about the Traceability!

  Since in the real production environment, there is always some failure somewhere: router/switch could die or rebooted, connections will timeout if the firewall is jammed, SSL Certificates will expire, and there'll be DB upgrade, OS Patching, ... ..., the failure could happen just at the time a new merchant customer just send out his second credit authorization request, or approval of a financial transaction with $10,000 just arrived, ... ... finally, you'll grab all the forensic information you could find to diagnose.

2. All these logging are neither for Operation Team nor for Support Engineers, they are actually for the developers themselves.

  Borrowing a word our Infrastructure Architect often said: "if there is anything you wish to see when you get called at 2AM, you should log it for yourself"

3. Strong Infrastructure Support

  Obviously, to make "Log Everything All the Time" work, you need a very powerful Logging Server/Bus to consume all these generated log info, you also need some powerful Log Miner or Search Tool to help you query, filter and correlate. Fortunately, in my current working enrionment, we have these kind of infrastructure available so we have no excuse but write logging code.

  Given a production environment with firewall, load balancer, front service gateway, Application Server, and backend DB, if it is not feasible to really log everything, you should at least log any request/response coming in and out any system/component, any critical state change in any system/component, and any state change in any system/component if it is possible. Again, it is all about the traceability.

Can you design a good benchmark to compare Apache Mina and Python Twisted?

2007-08-16T00:13:00.000-07:00

There is an interesting blog entry Mina and Twisted Matrix benchmark with update, it claimed that Twisted out-perform Apache Mina with the blogger's benchmark.

It actually raised a very interesting question: can we design a good and fair benchmark to evaluate Apache Mina and Python Twisted?

The following are some of my rough thoughts on a might-be good benchmark to evaluate Apache Mina and Twisted for TCP based Networking Application:

1. The benchmark should run on a platform which both Mina and Twisted can utilize its advanced Non-blocking Networking feature, such as Linux with epoll, otherwise we are not really benchmark two frameworks on a fair playground

2. Since both Apache Mina and Twisted have protocol processing component built in, we can define a simple Text Protocol with a simple format like "[Header][PayLoadLength][PayLoad]" and a binary protocol sending image for protocol processing. Based on these, we can create a group of sample protocol messages with different size/length for benchmark

3. We can create a test driver which will open a TCP Connection to the server, send a sample protocol messages to the server, the server will decode the message, construct a response with a different format like "[Header][PayLoad][PayLoadLength]", and echo back the payload received and close TCP Connection, the driver can process the response and make sure that the payload is correctly returned by the server.

We can increase concurrent connections to compare total connections established, the processing time and total payload size measured at driver side

4. We can create a test driver which will open a persistent TCP Connection to the server, and keep sending protocol messages to the server and server keep echoing back the payload received in a similar fashion as 3.

We can increase concurrent connections to compare total connections established, the processing time and total payload size measured at driver side

5. Since both Apache Mina and Twisted support building Networking Client and Server, the test actually can be done in the following combination:
-- Apache Mina Client against Twisted Server
-- Twisted Client against Apache Mina Server
-- Apache Mina Client against Apache Mina Server
-- Twisted Client against Twisted Server

Python Twisted and Apache Mina, Two Great Asynchronous Network Application Frameworks

2007-04-20T23:45:00.000-07:00

For last 10 months I have been working on a high scalable asynchronous network messaging gateway project using Apache MINA(Multipurpose Infrastructure for Network Applications), which is a Java Network Application Framework based on Java NIO.

One challenge to develop a high scalable asynchronous network messaging gateway is how you test it for functionality, load and performance. I eventually developed several Test Simulators and Drivers using Python Twisted, "an event-driven networking engine written in Python".

Twisted obviously is THE python framework if you want to develop a scalable asynchronous network application using Python. Personally, I think that Apache Mina has a very good chance to become the De facto java framework for developing NIO based network applications.

If you compare Twisted with Mina, they both adopt the "reactor pattern", they both have the goal to encapsulate/hide the complexity of Non-blocking network IO and provide a simple and elegant framework for developer to use. Actually, they are very similar at concept level as the following table shows:

Python Twisted	Apache MINA
Protocol	ProtocolEncoder/ProtocolDecoder
LineRecevier	TextLineEncoder/TextLineDecoder
protocol.ClientFactory	IoConnector
protocol.ServerFactory	IoAcceptor
Deferred	IoFuture
callback chain/errback chain	IoFilterChain
reactor ThreadPool	ExecutorFilter

Building Billing Processes with BPEL for Telecom Web Services

2006-08-02T00:06:00.000-07:00

  This is the last prototype project I did before I left Oracle. The goal is to explore a general approach for building flexible business processes to control Telecom Service Delivery by integrating Oracle BPEL PM with Oracle Web Services Manager (OWSM). So, Oracle BPEL Process Manager and Oracle Web Services Manager (OWSM) can be used as foundation platform to build Telecom Service Gateway.

  The sample Telecom Service we used is a Parlay X Send SMS Web Service built on top of Oracle Application Server Wireless Messaging. The whole Service Delivery Platform environment I set up is shown in the following diagram:
  For this project, one key thinking is that, in the governed SOA world, exposed Web Services or Business Services should be protected by certain Control Policies and we can orchestrate at least some type of Control Policy, such as Orchestrate Pre-Paid Billing Policy, Using BPEL.

  In general, we can see at least two types of control policies in the real world: declarative policy and process-driven policy. “Only the users with manager role can access Approving-PO Web Service” is a typical example of declarative policy. Process-driven policy usually involves a list of process steps to enforce the policy before and after service invocation. “Billing Policy” used in Telecom Industry, especially “Pre-Paid Billing Policy” is a typical example of process-driven policy.

  To use a service enforced by Pre-Paid Billing Policy, the service consumer will usually pay certain a mount of money as credit before he can start to use the service. Later, every time he/she uses the service his/her credit will be deducted based on the usage. Once the credit is used up, he/she will not allowed to use the service anymore unless he/she put new money in. Actually, it can be described as a list of processing steps as the following:

A Service Request come in and the User has been authenticated
User will be authorized if his account has enough credit for the service involved
An adequate fund has been reserved from user account
If the Service Request has been fulfilled successfully, the reserved fund will be transferred to service provider. If the service request failed due to network or server or other issues, the reserved fund should be returned to user's account as credit available

  Since Process-Driven Policy often invoke a list of pre-processing and/or post-processing steps, some of them might also be long-standing steps, BPEL actually is an ideal language/tool to describe and enforce it. The Pre-Paid Billing Policy described above can be easily implemented by orchestrated BPEL Processes.

  Assuming there is a BPEL Engine like Oracle BPEL PM running all the Pre-Paid Billing Policy Processes which integrated with existing external Billing System through exposed Web Service Interfaces, and there is a Service Execution and Enforcing Environment, such as Oracle Web Service Manager which will notify Billing Policy Process at different execution phases. The Pre-Paid Billing Policy can easily be orchestrated by multiple BPEL Processes as the following diagram shown:

  When the service request is received in the Service Execution Environment, the associated Policy Enforcement Point (PEP) will invoke the "Rating And Fund Reservation Billing Process" with collected Service Detail Record (SDR), which will check the user's credit at real time, reserve the fund if there is enough fund available and start "QoS And Charging Billing Process".

  "QoS And Charging Billing Process" will in turn start "Service Notification Process" waiting for the Asynchronous Service Notification.

  Then the Policy Enforcement Point (PEP) will notify "Service Notification Process" whether the Service Request is fulfilled, which will in turn notify the "QoS And Charging Billing Process", which will cancel the reserved fund if the request is failed to fulfill, otherwise, it will start "Post-Pay Confirmation Process" waiting for the final status notification once the final fulfillment is confirmed, which will finally let the "QoS And Charging Billing Process" deduct/transfer the reserved fund.

  In addition, "Service Notification Process" is also responsible to notify the "QoS And Charging Billing Process" in case that the Service Invocation is crushed without a chance to send out the notification.

  The Pre-Paid Billing Policy logic are mainly implemented in two orchestrated BPEL Processes:

Rating and Fund Reservation Billing Process, its business process logic is straightforward as the following diagram shown, when the process receives the service invocation request containing the Service Detail Record (SDR), it will invoke the external "Authorize Pay Web Service" to check whether the user has enough fund for the service request, if it is true, it then will reserve the fund through external "Reserve Fund Web Service" and start "QoS and Charging Process" and return the status of “Success”, otherwise the status of "Failed" will be returned

QoS And Charging Billing Process, as the following diagram shown, when it get started, it will immediately start "Service Notification Process" waiting for the Service Notification, if the "service fulfilled" status is received, the "Post-Pay Confirmation Process" will be started waiting for the final delivery status to invoke the external "Transfer Fund Web Service" and finish the whole Pre-Paid Billing Policy Processing. In case of failure, "QoS And Charging Billing Process" will invoke the external "Cancel Reserved Fund Web Service" to return the fund back to the customer's account.

As demostrated, using BPEL to build process-driven policies make these policies not only naturally executable but also easily integrated with both internal and external enterprise software systems

Using Maven2 to Build J2EE Application in a Corporate Environment

2006-04-14T00:38:00.000-07:00

Maven2 is a great building tool, but using Maven2 is not easy since the lack of document make it not user friendly, Timothy M. O'Brien, who co-wrote the book Maven: A Developer's Book, just wrote a blog entry Maven Project Info Reports Considered Dangerous talking about some difficulties a developer will encounter when trying to use Maven2.

Right now, as developer, I'm trying to set up a new build process using Maven2 for our development group in the coming projects. I'll try to document and share some tips, work-arounds and maybe the issues related to Maven2.

In general, to make Maven2 successfull in a Corporate Environment, we need some guidelines or best practices in the following areas:

Building and Managing a Internal Remote Repository shared among developers
(Note: The two very useful documents I saw on the Internet are Corporate https maven2 repositories and Using Maven in a corporate environment)

Optionally set up a Maven-Proxy inside Firewall to ease the access to central Maven2 Repository at Ibiblio

Developing RAR, WAR, EJB Archetype or Project Templates for common J2EE Component Projects
(Note: So far, the Maven2 ArcheTypes posted on Maven2 Repository and MyFaces ArcheType are all single project based ArcheTypes, which means that each archetype only generates one pom.xml)

Write or customize Maven2 Plug-ins for your own building process

A Typical J2EE Application Project Layout or Directory Structure

Continuous Integration with Nightly Build and Reports

Caveats and Workarounds:
7.1 How to exclude Maven Property files such as pom.xml and pom.properties in the generated JAR, RAR, and EAR File

7.2 How to set the RAR File name through the pom.xml

7.3 Maven2 ArcheType Plug-in did not recognize internal remote repository

In the last three weeks, I already spent a lot of night time browsing through Maven Web site, articles, blog entries, Maven Mailing List to gain some knowledge in the above areas. I'll update this blog entry with more Maven2 related detail when I make progress with my project in these areas.

Sun Certified Developer for Java Web Services (SCDJWS)

2006-03-10T10:13:00.000-08:00

I just passed Sun Certified Developer for Java Web Services Exam. I thought that combining Sun Certified Developer for Java Web Services (SCDJWS) with Certified BPEL Engineer (CBE) demonstrates my strong technical knowledge as a Senior/Principal Software Engineer working in SOA related Project/Product Development

Certified BPEL Engineer

2006-01-04T17:03:00.000-08:00

I was very pleased to pass the actvieBPEL's Certified BPEL Engineer Exam with a 95% (57/60) score.

The following are my learning and working experience related to BPEL which help me pass through the Certified BPEL Engineer Exam:

Working Experience:
- R&D Project for Smart Client using Oracle BPEL PM 10.1.2 with WSIF for routing the request from mobile device to the related business applications
- R&D Project using Oracle BPEL PM 10.1.2 with Oracle Web Service Manager 4.0.3 to demostrate a comprehensive Pre-Paid Billing Policy orchestrated with 9 BPEL Processes for controlling Service Invocation
Learning Experience:
- Oracle University Training Course "Oracle BPEL Process Manager: Services Orchestration"
- Business Process Execution Language for Web Services Version 1.1 Specification
- Matjaz B Juric's Book "Business Process Execution Language for Web Services"
- SOA Best Practices: The BPEL Cookbook from Oracle Technology Network

Also, Antony Reynold's Blog is a very good resource for BPEL, especially Oracle BPEL PM

Network Connection Limitation when using BlackBerry Device with "Direct TCP/IP Mode"

2005-11-06T01:09:00.000-08:00

I recently had developed some prototype using J2ME/MIDP 2.0 on BlackBerry 7290 Device with Cingular GPRS Network.

I encountered a problem when making direct network connection from BlackBerry Device through Cingular APN without using BlackBerry Enterprise Service ( Note: "What are - The different ways to make an HTTP or socket connection" is a good article describing three different ways to make socket connections from BlackBerry devices ). The exceptions was only occurred on the real device with trace on device log like:

======================================
app:Java Exception data:
ArrayIndexOutOfBoundsException
No detail message
net_rim_cldc_io_tcp
StreamDatagramTransportBase
close
0x3474
net_rim_cldc_io_tcp
StreamDatagramConnectionBase
close
0x29B7
net_rim_cldc_io_tcp
Protocol
shutDownConnection
0x5A5
net_rim_cldc_io_tcp
TcpIOThread
run
0x51B9
===================================

After talking to a nice BlackBerry Support Engineer I learned that "The BlackBerry supports a maximum of 5 direct TCP connections to occur at a time" and this limitation is existed in both 4.0, latest 4.0.2 and the coming 4.1 Handheld Software

So, this has a very sad implication: you have to use BlackBerry Enterprise Service(BES) to communicate with BlackBerry Device if you want to develope a real Enterprise Mobile Software targeting BlackBerry Device. BlackBerry is realy not an Open Platform for developing Enterprise Mobile Software. You can not develop a Mobile Enterprise Solution for BlackBerry Device ONLY, you have to be a RIM Partner or VAR to build a BES centric solution and I hope this was not RIM intended to be.

( Note: My personal guess is that, for BlackBerry Devices, "Direct TCP/IP Mode" actually is a shortcut of BES Solution, the internal TcpIOThread actually is just an on-device light weight BES Daemon thread for network communication )

I'll use this blog as my personal tech notes

2004-07-15T11:55:00.000-07:00

I'm a software engineer working here in Silicon Valley,
you can find my detailed working experience from my on-line resume