VA Server Smalltalk 5.5.2 hiccups

[ykc666]

We are having a new problem (described below) with our app running as application server in VA 5.5.2 which was not there for the last 4 years. I am not sure if it is related to all the patches we are applying to the windows 2000 server. If any of the smalltalk guru's can help us out or point to where we can find help, we really appreciate your help

Thanks in advance,
ykc

Problem:
While running some scripts, the application hangs for 10-15 seconds and becomes unresponsive too often. Not sure if it is doing garbage collection during that time, but this problem was not there for the last 4 years. Below is a sample code that illustrates this problem. I run the parsing script on a collection of xml strings and you can see that times when it hangs randomly for the same set of input xml strings.

|count|
count := 0.
XmlResults do:[:eXml |
|time|
time := Time millisecondsToRun:[IdXMLReader parseDocumentInString: eXml].
count := count+1.
Transcript cr; show: count printString, ') Parsing Time:', time printString].


First Run:
1) Parsing Time:281
2) Parsing Time:468
3) Parsing Time:516
4) Parsing Time:547
5) Parsing Time:172
6) Parsing Time:500
7) Parsing Time:672
8) Parsing Time:10328
9) Parsing Time:312
10) Parsing Time:187
11) Parsing Time:313
12) Parsing Time:187
13) Parsing Time:188

Second Run:
1) Parsing Time:203
2) Parsing Time:422
3) Parsing Time:484
4) Parsing Time:10547
5) Parsing Time:156
6) Parsing Time:484
7) Parsing Time:547
8) Parsing Time:266
9) Parsing Time:296
10) Parsing Time:204
11) Parsing Time:296
12) Parsing Time:10125
13) Parsing Time:188

Third Run:
1) Parsing Time:203
2) Parsing Time:406
3) Parsing Time:469
4) Parsing Time:500
5) Parsing Time:157
6) Parsing Time:468
7) Parsing Time:10469
8) Parsing Time:266
9) Parsing Time:312
10) Parsing Time:188
11) Parsing Time:297
12) Parsing Time:172
13) Parsing Time:203

[marten]

Does Smalltalk really runs during the bad numbers ? Or are other processes cosuming the CPU time ????

[ykc666]

Our production server is a dedicated server just for the smalltalk app. I don't believe there are other apps other than OS running on this box. The server app is preloaded with data and runs around 1.5GB memory footprint on a 4GB box

Does Smalltalk really runs during the bad numbers ? Or are other processes cosuming the CPU time ????

[wembley]

What values do you specify for memory allocation when you start the image -- for example, -mo on the commandline or oldSpaceSize= in the .INI file?

[ykc666]

mo = 16000000 (16mb)

Should we increase this size?

What values do you specify for memory allocation when you start the image -- for example, -mo on the commandline or oldSpaceSize= in the .INI file?

[wembley]

I suspect that you are running into garbage collection. Garbage collection on a 1.5GB image is not going to be fast

Try specifying -mo a bit larger than your steady state image size (for example, 1.6GB: -mo 1600000000) and see if it helps. What you want to do is keep garbage collection from happening by giving the image enough room to allocate all its objects at steady state.

Note that you may need to rebase some of the VAST and Windows DLLs (see the MSDN documentation for EDITBIN) in order to get this much memory).

[ykc666]

Thanks for you reply. I will try this and see if this makes a difference. The settings didn't change for the last 4 years and the image size was always around 1.5 GB. I was just curious why the garbage collection is acting weird this last 2 months. Did any of the patches on the windows 2000 server replace any core windows libraries that were working fine with VAST 5.5.2 VM before the patches?

I suspect that you are running into garbage collection. Garbage collection on a 1.5GB image is not going to be fast

Try specifying -mo a bit larger than your steady state image size (for example, 1.6GB: -mo 1600000000) and see if it helps. What you want to do is keep garbage collection from happening by giving the image enough room to allocate all its objects at steady state.

Note that you may need to rebase some of the VAST and Windows DLLs (see the MSDN documentation for EDITBIN) in order to get this much memory).

[wembley]

It's certainly possible that Windows released a replacement DLL with a different base address -- this could be due to any number of reasons. This may have resulted in Smalltalk having difficulty allocating additional memory and so driving Smalltalk to do GC more often. Of course, this is all just speculation based on very little concrete information.

If my speculation is correct, then you will need to rebase some DLLs to free up sufficient continguous memory for allocation.

[ykc666]

Could you tell me more about "rebase some DLLs ". I am not familiar with this process. Are these windows DLL's or VAST DLL's?

It's certainly possible that Windows released a replacement DLL with a different base address -- this could be due to any number of reasons. This may have resulted in Smalltalk having difficulty allocating additional memory and so driving Smalltalk to do GC more often. Of course, this is all just speculation based on very little concrete information.

If my speculation is correct, then you will need to rebase some DLLs to free up sufficient continguous memory for allocation.

[wembley]

The following is a simplified discussion that overlooks some of the subtleties of memory allocation on Windows, but is sufficiently correct for our purposes.

DLLs are typically loaded from low memory to high memory while heap space is allocated from high memory to low memory. However, DLLs can have a load address specified when they are linked and they will be loaded at that address if it isn't already allocated.

As heap space is being allocated, the allocation address will eventually bump into a DLL -- oops, no more memory! But there may be memory holes between DLLs which could be used to satisfy the allocation IF they were continguous with the rest of the free memory.

So, the trick is to use a tool like ProcessExplorer (download from Microsoft) to see the layout of DLLs in memory. Then use EDITBIN (download from Microsoft) to pack the DLL space by moving the "rogue" DLLs to lower memory.

The "rogue" DLLs can be Windows, VAST, or other program's DLLs. With VA Smalltalk 7.5.1 we linked many (but not all) of our DLLs so they occupy contiguous memory. With VA Smalltalk 7.5.2 all the VA Smalltalk DLLs will be packed together to maximize available heap space.

[ykc666]

Will try this. Thanks for the information

The following is a simplified discussion that overlooks some of the subtleties of memory allocation on Windows, but is sufficiently correct for our purposes.

DLLs are typically loaded from low memory to high memory while heap space is allocated from high memory to low memory. However, DLLs can have a load address specified when they are linked and they will be loaded at that address if it isn't already allocated.

As heap space is being allocated, the allocation address will eventually bump into a DLL -- oops, no more memory! But there may be memory holes between DLLs which could be used to satisfy the allocation IF they were continguous with the rest of the free memory.

So, the trick is to use a tool like ProcessExplorer (download from Microsoft) to see the layout of DLLs in memory. Then use EDITBIN (download from Microsoft) to pack the DLL space by moving the "rogue" DLLs to lower memory.

The "rogue" DLLs can be Windows, VAST, or other program's DLLs. With VA Smalltalk 7.5.1 we linked many (but not all) of our DLLs so they occupy contiguous memory. With VA Smalltalk 7.5.2 all the VA Smalltalk DLLs will be packed together to maximize available heap space.