NT service freezes sometimes for hours

[PhotonDemon]

HI,

I have five or six Windows NT services running, in production, at a customer. They are built with VAST 5.5.2. This morning, three of them froze or went to sleep or what ever. I am deliberately trying not to use a real technical term to describe their state as I don't really know what it was. After about three hours they woke up or came back to life. During the time they were out, they did nothing and here I really do mean nothing.

All of the programs have a high priority fork that wakes up every minute to see if there is anything they need to do based upon the time of day. Every other minute this fork logs a time stamp with #outputToTTY. During the time these servers were frozen, these log entries are missing. I think I have seen some missing log entries for a few minutes but didn't connect that with anything else (I just thought the disk didn't get updated or something).

Two of these servers are connected to outside hosts via TCP/IP that send them messages. One of which gets between 1 and 2 million a day. During the time they were frozen these connections were not serviced. When the program woke up, it went back to reading from the connection. About 20 messages were read and then the line went dead. I'm sure the other side had killed the connection hours ago. I would think that as soon as the program woke up and tried to talk to the connection it should have been told the line was dead and not been able to read 20 messages. I'm not sure what this means but I thought I should mention it.

All these servers are also listening on a port for a connection from one of my workstation programs that can ask it questions or tell it to do something. After this program connected to one or more of the servers, they seem to wake up. I don't know if this is what woke them up or it is a coincidence.

I don't see how this could be caused by Smalltalk code. It could be a VM problem but that doesn't make a lot of sense because I can't think of any reason why the VM wouldn't dispatch any of the forks of Smalltalk code.

It could be windows not dispatching the Smalltalk VM but I can't think of a good reason for that.

Has anyone seen anything like this? By this I mean the program freezing, the stuff about the sockets and such is just how I know the programs were frozen.

Lou

[benvandijk]

Hi Lou,

It can be that your image is sending a message to another party and the other party for whatever reason does not answer.
Your image will wait indefinitely for the answer but will wake up when the other party breaks the connection.
The requests processed after the wake up were probably just waiting in your input backlog.

We had a similar problem when calling a https webservice using openssl.
All calls are done with synchronized calls which will cause the image to wait for answers.
We changed the calls to async calls which solved that problem, because the wait will not hang the image.
Later we replaced openssl with winhttps because the openssl implementation in VAST will not work correctly on multicore machines.

[PhotonDemon]

Hi Ben,

Thanks for the post. All of the programs in question use Totally Objects SocketSet non-blocking TCP/IP sockets to communicate with each other and outside hosts, so that's not the problem. Also, at least three of five programs (I will have to check the logs for the other two) froze at the same time.

I was thinking that somehow the VM wasn't getting dispatched but I couldn't come up with a good reason for that. Now I am thinking that one or more of the requests for information that I make to Windows may block. I periodically ask Windows things like what time is it, how much memory am I using and how much CPU time am I using? I guess Windows could have gotten stuck answering one of these or other questions. Or there could have been a hardware problem saving the time and all the programs would back up behind that. The case in question, things were frozen for a few hours. Really odd?

Lou

[TriSebastian]

Hi!

Well, we had a similar problem. It's quite a while. It also affected out NTServices.
The reason for these problems where Windows Updates. Where there any in the past days on your machines?

We also had non responding Services a few month ago, but the reason for this was in deed a situation like that one you just described. The service waited for ansers and didn't get any. It was a programming gap in our codeing. A long running backend process didn't acutally find its server session becouse the connection/Session got shutdown meanwhile...

So in the end it wasn't a SST problem.

Sebastian

[PhotonDemon]

Hi Sebastian,

Well, we had a similar problem. It's quite a while. It also affected out NTServices. The reason for these problems where Windows Updates. Where there any in the past days on your machines?

Windows Updates were applied around the time of the problem. Do you know any more about how they cause the problem? Does a re-boot clean thing up?

Thanks, Lou

[TriSebastian]

I don't have more detailed information on this, sorry.

I can just say that a reboot didn't solve the problem. As far as I can remember the updates didn't really install properly. Nothing new with Windows, isn't it?

Another thing you might want to check are tcp timeouts (we have on client which needs a lager timeout as the defaults...) on the network layer level. Changes here should usually lead to timeouts from client side,... but you never know.

As I already said, I'm not sure, and I couldn't ask anybody arounf right now. But I feel like having in mind that there was also an update of the .Net framework involved...

Best might be to reinstall the update cleanly.

Sebastian