VA 7.x Web Services performance problem

[Sam De Block]

We are working on an application that relies heavy on web services with soap protocol, however we are experiencing some performance problems.

It seems that one image won't process more than 5 simultaneous requests(connections to a specific webservice).
At first we thought is was the listenBacklog ivar in SstSocketConfiguration, which defaults to 5. However increasing this only makes visualage accept more simultaneous connections, but it won't process them.

An example might clear things up:

if 1 client has 5 simultaneous threads to a server, the request response time is an average of 500ms
if 2 clients each have 5 simultaneous threads to a server, the request response time climbs to an average of 1000ms, effictively doubling the response time.

The server is running on a win2k3 server, clients are 1 xp pro and 1 win2k3.
We are using visualage with SST Web services on the server.
We are load testing with Jmeter.

Now my question is: is the max connections/max threads some setting that is in visualage or is it maybe some kind of windows seting or ...?

Second question, I've noticed that in the roadmap for VisualAge there are planned enhancements for Web Services support. Is there any documentation available on some of the planned changes?

[solveig]

Sam:

What sort of hardware are you running?

What sense are you using "connections" and "threads"?

As for the roadmap, we are planning to improve the user documentation of the Web services and include more live examples.

We are also planning to include a wizard to give a web service developer one place to turn a Smalltalk class into a web service with the necessary XML, WSDL map and schema files all generated together.

Regards,
Solveig

[Sam De Block]

Server is running on an Intel Xeon with Win2k3.
Threads and connections I bascially mean the same thing

the problem is:
With 1 client I get a throughput of 10 requests/second, with 2 clients each client gets 5 requests/second. Since I'm not hitting the my servers capacity, each client should get a throughput of 10 requests/second. Which means (I think) that the server only handles 5 requests at a time. The other 5 just go in the backlog.

I tried using the SstThreadedDispatcher instead of the SStPooledDispatcher but that didnt work.

[marten]

We are working on an application that relies heavy on web services with soap protocol, however we are experiencing some performance problems.

Do you use nonblocking or blocking ip communication ? What about
changing this configuration parameter ...

Marten

[Sam De Block]

I tried the SstPooledDispather, I tried increasing the workerManager high/low /strict count, I tried changing nonBlocking but nothing works.

I put together some small instructions so you can test this for yourself: WebServicePerformanceTest.zip
Basically I don't want to run 2 images to get the performance, I wan't it in 1 image.
So if 5 connections give around 4requests/second, I should get around 8reqs/second with 10 connections.

[solveig]

Sam:

I usually think of threads as having to do with processes.

VA Smalltalk only uses one operating system process. VA Smalltalk executes messages round robin. Each message continues until it completes or until it needs to wait for something like I/O. At that point, the dispatch loop picks up another message to execute.

An exception to this is asynchronous calls which are made in a separate operating system thread. When the asynchronous call is blocked waiting for a resource, only that thread blocks and not the thread that VA Smalltalk is running in.

You mentioned switching dispatchers from SStPooledDispatcher to SstThreadedDispatcher. At the risk of confusing things, the "threaded" does not refer to the use of asynchronous calls. Rather it refers to the creation of Smalltalk Process classes. These are scheduled for execution according to Smalltalk's Common Process Model (CPM.) This subsystem provides a process model (shared memory threads) for VA Smalltalk.

Even with these ideas of threadedness, VA Smalltalk executes one of your web service requests at a time. It will continue to execute the request until that request blocks on something, like a database call.

Since you are using SOAP, I assume the type of client-server connection is a per-request variety rather than a persistent variety.

What is the nature of the requests? Are they computationally intensive? This would mean each request would not block on I/O and might take a long time to complete.

Marten's question of the type of IP communication you employ might also be pertinent.

Regards,
Solveig

[Joost Bossuyt]

Solveig,

I work on the same project as Sam.

The background of our question is that we have a new project.
The front-end will be Websphere Portal and the back-end (business model) is VASmalltalk. The communication will be by means of web services.

The web services are used for validating, retrieving and storing things in our database.

We will have several smalltalk servers. But we want to running more than one image on the same server.
What we see is that we're having problems when we have more than 5 concurrent requests on one image.

We made a sample web service that retrieves 500 person names from our database. What we notice is that the database communication is very fast, but the generating of the xml takes most of the time.

Let's say we have 10 concurrent sessions.
Most of the time will be spend by generating the SOAP answer.
Do you mean that the generation of one SOAP answer will block the others? If this is the case than only one request at a time will be handled.
How many servers do we need to serve 100 concurrent requests?

Thanks,
Joost.

[marten]

Solveig,

Most of the time will be spend by generating the SOAP answer.
Do you mean that the generation of one SOAP answer will block the others? If this is the case than only one request at a time will be handled.
How many servers do we need to serve 100 concurrent requests?

Thanks,
Joost.

You may put "Processor yield" throughout some code to get task
switching functionality - if this is really needed.

If you have dual core machine (hyper threaded) and it shows 50% CPU time, then the Smalltalk image is running at highest speed - then actually nothing more - beside optimizing - can be done.

Marten

[daswartz]

The background of our question is that we have a new project.
The front-end will be Websphere Portal and the back-end (business model) is VASmalltalk. The communication will be by means of web services.
...
We will have several smalltalk servers. But we want to running more than one image on the same server. What we see is that we're having problems when we have more than 5 concurrent requests on one image.

We made a sample web service that retrieves 500 person names from our database. What we notice is that the database communication is very fast, but the generating of the xml takes most of the time.

Let's say we have 10 concurrent sessions.
Most of the time will be spend by generating the SOAP answer.
Do you mean that the generation of one SOAP answer will block the others? If this is the case than only one request at a time will be handled.

It is true that VAST will only execute user code for one of your requests at a time. Of course, it is also true that any single processor CPU can only actually execute one program's code at a time as well. A dual processor machine can only execute two at a time. The trick is all in how task switching is handled. VAST can do a fine job of switching between multiple smalltalk processes. It would normally do this when control is released by executing something like a database request.

I didn't hear you say that Smalltalk was rejecting your SOAP requests, I heard you say it isn't processing them fast enough. Is that correct? If true, you don't really have a concurrent requests problem, you have a throughput performance problem.

Ideas to get more throughput:

If you are on a multi-core machine, run multiple Smalltalk images servicing the SOAP requests. Websphere portal can load balance the requests to the images.

Profile your requests using the Smalltalk code profiler and tune the code. If you haven't done so, you may be amazed at the improvements you can gain with very small code changes. The VAST profiler is very good, and I find I can often get an order of magnitude of improvment pretty easily out of code which has never been profiled. You may be correct that the XML generation is very slow, but use the profiler to prove it and pinpoint the problem areas.

How many servers do we need to serve 100 concurrent requests?

No-one can answer that question without knowing what response time SLAs you have, and the resource consumption of each request.


Doug Swartz

[quixotik]

I didn't hear you say that Smalltalk was rejecting your SOAP requests, I heard you say it isn't processing them fast enough. Is that correct? If true, you don't really have a concurrent requests problem, you have a throughput performance problem.

Actually, the Smalltalk image is rejecting requests. With 10 concurrent clients, only 5 or 6 out of the 10 get served. The Smalltalk code does nothing more than return some static data in our test case.

I will be looking at the code in a couple of days, so any other tips or experience reports that will lead to a solution will not earn anyone a VASmalltalk license, but maybe a few beers at the upcoming Smalltalk Solutions