Very poor EMSRV Performance on AIX

[heyrothu]

Since our library is getting close to the 2GB limit, we had to migrate from the Linux to an AIX based server. Everything is fine except for the performance. Compared to the Linux System the AIX server is up t 5 times slower than the Linux one. E. g. connecting the standard image to the Linux lib will take 15 to 20 seconds and connecting to the AIX library wil take 75 to 90 seconds.

For me, the one big difference (except the obvious OS) between the two servers: Linux is working on a 100MBit and AIX is working on a 1GBit network card.

Copy operations with winscp show a transfer rate of 1.5MB/s from my workstation, which is on 100MBit.

We installed the server, following the installation guide to the letter.

Any comments and help are deeply appreciated.

Regards

Uwe

Following is hopefully some useful information.

EmSrv configuration of the AIX:

Code: Select all

EMADMIN 7.0a
Copyright (C) IBM Corporation 1989-2002
Code page (ANSI): 1252

==============================================================================
EMSRV 7.1a for AIX May 28 2002 14:22:14 (EST)
Operating system: AIX 2 5 005EC00C4C00
Locale: en_US en_US en_US en_US en_US en_US

Total Connects:                      1  Total Disconnects:                   0
Total Opens:                         0  Total Closes:                        0
Active Locks                         0  Unexpected Connection Closes:        0
Total Locks:                         0  Total Unlocks:                       0
Total Reads:                         0  Total Writes:                        0
Total Reads Failed On Lock:          0  Total Locks Failed On Lock:          0
Times Lock Limit Hit:                0
Total Requests Serviced:          4283  Requests in last interval:           1
Largest Packet Sent:                62  Largest Packet Received:            12

Working Directory : /opt/IBMvast/6.0/bin
Server Has Been Alive For: 0 Days 1 Hours 11 Minutes 41 Seconds
==============================================================================


Network configuration of the AIX:

Code: Select all

oslevel -r
5200-02

lsattr -El ent3
alt_addr       0x000000000000   Alternate ethernet address                    True
busintr        183              Bus interrupt level                           False
busmem         0xf0080000       Bus memory address                            False
chksum_offload yes              Enable hardware transmit and receive checksum True
compat_mode    no               Gigabit Backward compatability                True
copy_bytes     2048             Copy packet if this many or less bytes        True
flow_ctrl      yes              Enable Transmit and Receive Flow Control      True
intr_priority  3                Interrupt priority                            False
intr_rate      10000            Interrupt events processed per interrupt      True
jumbo_frames   no               Transmit jumbo frames                         True
large_send     yes              Enable hardware TX TCP resegmentation         True
media_speed    Auto_Negotiation Media speed                                   True
rom_mem        0xf0040000       ROM memory address                            False
rx_hog         1000             RX buffers processed per RX interrupt         True
rxbuf_pool_sz  2048             RX descriptor queue size                      True
rxdesc_que_sz  1024             RX descriptor queue size                      True
slih_hog       10               Interrupt events processed per interrupt      True
tx_que_sz      8192             Software transmit queue size                  True
txdesc_que_sz  512              TX descriptor queue size                      True
use_alt_addr   no               Enable alternate ethernet address             True


Code: Select all

entstat -d ent3
-------------------------------------------------------------
ETHERNET STATISTICS (ent3) :
Device Type: 10/100/1000 Base-TX PCI-X Adapter (14106902) Hardware Address: 00:09:6b:be:3e:7c Elapsed Time: 5 days 20 hours 34 minutes 8 seconds

Transmit Statistics:                          Receive Statistics:
--------------------                          -------------------
Packets: 297721                               Packets: 5235181
Bytes: 94062410                               Bytes: 658126487
Interrupts: 0                                 Interrupts: 4545086
Transmit Errors: 0                            Receive Errors: 0
Packets Dropped: 0                            Packets Dropped: 0
                                              Bad Packets: 0 Max Packets on S/W Transmit Queue: 8 S/W Transmit Queue Overflow: 0 Current S/W+H/W Transmit Queue Length: 2

Broadcast Packets: 1815                       Broadcast Packets: 4001355
Multicast Packets: 0                          Multicast Packets: 727532
No Carrier Sense: 0                           CRC Errors: 0
DMA Underrun: 0                               DMA Overrun: 0
Lost CTS Errors: 0                            Alignment Errors: 0
Max Collision Errors: 0                       No Resource Errors: 0
Late Collision Errors: 0                      Receive Collision Errors: 0
Deferred: 0                                   Packet Too Short Errors: 0
SQE Test: 0                                   Packet Too Long Errors: 0
Timeout Errors: 0                             Packets Discarded by Adapter: 0
Single Collision Count: 0                     Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 2

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Running
        Simplex 64BitSupport ChecksumOffload
        PrivateSegment DataRateSet

10/100/1000 Base-TX PCI-X Adapter (14106902) Specific Statistics:
--------------------------------------------------------------------
Link Status : Up
Media Speed Selected: Auto negotiation
Media Speed Running: 1000 Mbps Full Duplex PCI Mode: PCI-X (100-133) PCI Bus Width: 64-bit Jumbo Frames: Disabled TCP Segmentation Offload: Enabled TCP Segmentation Offload Packets Transmitted: 8408 TCP Segmentation Offload Packet Errors: 0 Transmit and Receive Flow Control Status: Disabled Transmit and Receive Flow Control Threshold (High): 32768 Transmit and Receive Flow Control Threshold (Low): 24576 Transmit and Receive Storage Allocation (TX/RX): 16/48

[daswartz]

You say that the one big difference is the speed of the network card. Are the Linux and AIX boxes on the same network segment? If there are even one or two more router hops to the AIX box versus the Linux one, then that may explain your differences.

ENVY is VERY chatty. It does almost no large network transfers and lots and lots of small ones. For this reason the raw speed of the NIC on your emsrv box is almost always much less important than the distance (in hops) between the client and the server.

Doug Swartz

[heyrothu]

Hi Doug,

they are not only in the same segment, but on the same switch.

Additionaly our network guy told me, that the GB card ist set to automatic speed negotiation, whereas the 100MB card ist set to 100MB Full Duplex. We changed the setting of the GB card to to 100MB Full Duplex as well, but it didn't matter.

Uwe Heyroth

[koschate]

I worked on VA in an AIX environment in a past life. The repository was always slow, to the point where we disabled some of the features of VA Assist Pro to get adequate performance.

Have you looked at cloning your repository to clear out the old cruft? This might give you the headroom you need to keep your repository on Linux. Unless you're keeping non-code artifacts in the repository, 2GB of active code represents an enormous application.