HTTPS-WebService Client kills our app

[bonndias]

Hi,
in Oktober I asked for help on invoking a HTTPS-WebService from within Smalltalk in this thread:

Invoking HTTPS-WebService from VA 7.0?
viewtopic.php?f=12&t=2248

Everything worked fine in single user mode but in the meantime our app is used in production environment (heavy multi user) and now invoking the webService randomly leads to a not responding application.

Our App is a TCP/IP based server application runnning as windows service with Java-Clients, using a custom framework for the server part.
As a new requirement on logon of a Java-Client we call the HTTPS-WebService for authentification. This works fine but with multiuser access and heavy load the system fails to asnwer after an unpredictable amount of time. The application refuses to respond to any kind of request and can't even be stopped via windows management console.

My implementation is like this:

pruefeUser: aUser password: aPassword
| service |

self configure ifError: [:error | ^error].
service := self getWebService.
service ifError: [:error | ^error].

^service
invoke: 'checkPermission
withArguments: (
Array
with: aUser
with: aPassword

The webService is created like this:

getWebService: wsdlName
| container |

container :=
SstWSContainer
containerNamed: self containerName
ifNone: [
container := SstWSContainer createContainerNamed: self containerName.
container
deploy: wsdlName;
yourself].

^container allServices first

Best Regards
Joa

[tstalzer]

Joa showed me the problem yesterday - here is some additional information

If he looks at the process via the "Sysinternals Process Explorer", the application (which is packaged as a Windows Service) is in a state "Wait:DelayExecution".

The server operating system is Windows 2003 server.

The fault can be reproduced using a tool which simulates a large number of clients. If the coding regarding the https WebService is commented out, the fault does not occur.

The application uses other Sockets to provide communication with the clients.

--Thomas

[tc]

Hello,

I want nail down the sequence of events, a java client connects to an app on the server and through a custom framework, your ST service makes a web service call to authenticate the logon?

How does the custom framework handle multiple logons coming in around the same time? Are they queued and fired off after the previous web service call was answered or are they fired off as the logon comes in?

--tc

[tstalzer]

Hi tc

In the target implementation, a request gets sent to the server via a socket from a Java client. The server makes the https WebService call (using standard VA/Smalltalk WebService functionality - basically the code Joa mentioned in his link). After getting results from the WebService call, data is being sent back to the Java client.

Joa made a couple of tests, each with the same result

- forking of the WebService call in another thread and waiting for the results with a semaphore
- forking of the WebService call in another thread and returning "dummy" data to the Java client

None of these changes made a significant change.

The application works stable without the https WebService call (and it gets a couple thousand requests from the Java clients)

--Thomas

[tc]

Hello,

How many users/requests is a 'heavy' load?

I guess what I am trying to figure out is can the web service handle requests being fired off, should many logon requests come in at one time, ie is that contributing to the problem?

Thanks.

--tc

[bonndias]

Hi,
we changed our test to invoke a different WebService using HTTP instead of HTTPS. Within 30 minutes ~3500 client logons were executed by 50 concurrent clients without any problems by our server. This leads us to suspect a problem using OpenSSL.

Any ideas?

Joa

[louis_andriese]

Hi,

this looks a lot like an issue we've had with outgoing https-calls hanging up our VA-service on windows. The problem was worked around by Instantiations (Solveig) by making all relevant openssl-calls sync. Since that fix we've never seen the issue again.

Kind regards,

Louis

[bonndias]

Hi Louis,

this looks a lot like an issue we've had with outgoing https-calls hanging up our VA-service on windows. The problem was worked around by Instantiations (Solveig) by making all relevant openssl-calls async. Since that fix we've never seen the issue again.

that sounds like a possible solution. Actually we are using VA 7.0.2, the version of SciSslSocketInterface is V 6.0.2. Are the changes you mentioned included in this version?

Joa

[louis_andriese]

Hi Joa,

no, the changes came to us in a post-7.5 update from Solveig.
For the rest: Alas, I'm only the manager
I'll ask one of our developers to reply with some more details.

kind regards,

Louis

[benvandijk]

Hi Joa,

As Louis told you, we have the same problem. It is caused by the async calls to SSL on a multi core machine.
On a single core machine it works fine and it can also work fine one a multi core machine with a low workload.

I put the changes we got from solveig, along with a few changes of our own, in the attachment.

Hope this will solve your problem,

Ben

[tc]

Hello,

After your reading your reply, I did a search on Google:

http://www.ddj.com/architect/184416574

However, since the OpenSSL code has a UNIX heritage, most of the examples available assume a UNIX-style sockets architecture. The preferred Windows architecture of asynchronous sockets doesn't fit well with this

http://www.lenholgate.com/archives/000456.html

OpenSSL is an open source implementation of the SSL and TLS protocols. Unfortunately it doesn't play well with windows style asynchronous sockets.

--tc

[bonndias]

Hi All,
the problem seems to be solved! We used Ben's code and in the meantime successfully executed our test with 50 concurrent users. Final tests are running at the moment, but the change to "call" instead of "asyncCall" seems to be necessary for using OpenSSL at least on Windows .

Thanks a lot

Joa

[bonndias]

Unfortunately we did not achieve to bring our application to production using HTTPS-WebService client. Ben's fix seemed to solve the problem first. But later, when more tests were done, our application failed again. After a lot of testing and trying we can say:
If we use HTTPS to call a WebService our application fails. If we use HTTP to call a WebService our application runs without any problems.

After running a script for about an hour we get an TCP/IP error 10014 when trying to send an answer to a client request. The socket in error is the one, which did the accept in the main loop of our server. Our test simulates 30 to 60 clients working with our server application. Each client creates a socket, sends its request and waits for the answer, which is sent back on the same socket. While the client waits for the answer, the server does calculating, eventually calls a webservice using HTTPS, creates a PDF etc. For calculating etc. we use Dlls, which are called using the PlatformFuntion>>asyncCall: protocoll.

Regards

Joa

[marten]

After looking at OpenSSL I read, that their developers state, that OpenSSL is not thread safe by its own. To make it thread-safe one has to add two callbacks in its own program ?!

[bonndias]

Hi Marten,
we are using Ben's fix, which uses callWith: instead of asyncCallWith: when invoking a function call into the OpenSSL code. We even tried to serialize the calls to the HTTPS-WebService, but no success. Where do we have to register the callbacks? That's the first time I here of this.

Joa