batch versus single transaction processing

[spookyboo]

Did you ever face the dilemma to choose between batch processing and individual processing in a SOA environment. The outline of the project:
A file with 120.000-350.000 orders if received from a third party supplier and must be offered to a financial module, written in Oracle (PL/SQL). My initial choice was to design a Java adapter which performs validation of the file, splits each record individually and puts it on the Enterprise Service Bus. A JMS listener reads the orders again and offers them one by one to the financial module. With this method it is possible to handle batch files and individual transactions the same; the listener just gets them from the service bus and doesn't care where it came from. Another advantage is that processing of each individual order can be a transactional process. You don´t want processing of a complete file to be one transaction.
Of course I know that batch processing is faster (sql*loader does it very fast), but I didn´t want to optimize too fast. The Oracle developers thought I was a complete fool (which may be the case), but I wanted to stick to the SOA philosphy as much as possible. Don't say that SOA does not exclude batch processing; in the end, if the hardware is up to it, batch processing is an ancient idea. For the time being, you have to stick with it from time to time.

Does anyone has similar experiences?

[syedhs]

I think you should listen to the Oracle guy One of the biggest advantage of using batch is network traffic is used diligently, ie instead of small packets send back and forth 10,000 times, you are sending 1 big packet and done!. In enterprise application, the most bottleneck lies on the network throughput exactly like this.

And in my opinion, it is okay too to have one transaction for thousands of records because let say 1-2% of them are screwed up for some reason, you will want to abort the whole batch - not just 1-2 record. Otherwise, you will have to remember which records are not ok in translation/transmission.

And lastly, like you mention batch processing is very very fast. I once write codes in both C++ and stored procedure to read a 50MB++ file, split and subsequently store them into tables - it takes maybe 20 seconds the most. Versus if you do it one by one, it takes at least 2 minutes. And I am talking about everything done locally.

[spookyboo]

Don't forget that with modern systems you just pump millions of messages over a service bus. We run some tests on a crappy testserver and it already processed 10.000 messages per minute. The bottleneck unfortunately is the database system.
In your example you are talking about a difference between 20 seconds versus 2 minutes. If the NFR's say that they need to be processed within 15 minutes you still can decide to choose for individual processing.
The reason I came with this disussion is that developers sometimes have a narrow vision. What didn't work 10 years ago may have been changed by now. I am not a fool, I know that processing a batch as a batch can't still be beaten, but slowly batchprocessing becomes someting of the past.

[Brian Adsboel]

there is a great problem with non optimisitc concurrency as it blocks too much too early and thus degrade performance extremly ... thus we end nearer single transactions with rollback option ...

[Minthos]

I think you're making a mistake in discounting batch processing and performance optimizations in general. Sure enough it is becoming unnecessary for a wider range of problems as hardware resources improve, but at the same time yet bigger problems become feasible to solve and the demands for hardware resources and optimizations increase.