Slow network performance with virtualization

Posted by George Potemkin on 11-Oct-2019 16:21

Customer moves their databases from CentOS 6 to CentOS 7 (3.10.0-957.27.2). Both are VMWare on the same metal box. Network driver vmxnet3
Performance dropped down approximately by 9 times

Test against the sports2000 database:

ETIME(TRUE).
FOR EACH OrderLine NO-LOCK /*NO-PREFETCH*/ /* SHARE-LOCK */ :
  ACCUMULATE "rec":U (COUNT).
END.
MESSAGE ETIME VIEW-AS ALERT-BOX.

Table OrderLine has 13970 records.

Results:

CentOS 6

Run  NO-LOCK NO-PREFETCH SHARE-LOCK
1       79      1353     1736
2       75      1364     1624
3       80      1368     2049
4       80      1328     1729
5       93      1508     1683
6       79      1368     2015
7       99      1550     1730
8       73      3714     1924
9       93      1643     1668
10      90      1426     1679
11      87      1392     1809
AVE    84.4     1637.6   1786.0

CentOS 7

Run  NO-LOCK NO-PREFETCH SHARE-LOCK
1       88      7137     26541
2      102      6281     26013
3       87      7259     25303
4       82      7381     25318
5       88      6883     27381
6       84      6472     27532
7       79      6101     25775
8       87      5960     27045
9       82      6029     25494
10     111      4909     26087
11      80      6284     26263
AVE     88.2    6426.9   26250.2

Results on CentOS 6 are expected:
NO-LOCK is 20 times faster than NO-LOCK NO-PREFETCH (-Mm 16384): a few messages vs. one tiny record (43 bytes in average) per network message;
SHARE-LOCK is just a bit slower than NO-LOCK NO-PREFETCH. It’s still one record per network message.

Results on CentOS 7:
NO-LOCK is as fast as on CentOS 6;
NO-LOCK NO-PREFETCH is 3.92 times slower than the same code on CentOS 6;
SHARE-LOCK is 4.08 times slower than NO-LOCK NO-PREFETCH.


Note: the factor 4 in both two unexpected cases.

It was suggested to run ‘rngd -r /dev/urandom’ but the customer can’t find the rngd on both CentOS.

What is the root case of the slowness?
How to solve the issue?

All Replies

Posted by Libor Laubacher on 11-Oct-2019 16:30

Make sure you are running latest VMTools.

Also double check if the NIC is defined as passthru.

Sent from my iPhone

On 11 Oct 2019, at 18:23, George Potemkin <bounce-GeorgeP12@community.progress.com> wrote:

<ProgressEmailLogo-png_2D00_150x42x2-png> Update from Progress Community
<4U3VBQTP4GDL-jpg_2D00_70x70x2-jpg>
George Potemkin

Customer moves their databases from CentOS 6 to CentOS 7 (3.10.0-957.27.2). Both are VMWare on the same metal box. Network driver vmxnet3
Performance dropped down approximately by 9 times

Test on sports2000 database:

ETIME(TRUE).
FOR EACH OrderLine NO-LOCK /*NO-PREFETCH*/ /* SHARE-LOCK */ :
  ACCUMULATE "rec":U (COUNT).
END.
MESSAGE ETIME VIEW-AS ALERT-BOX.

Table OrderLine has 13970 records.

Results:

Centos 6

Run  NO-LOCK NO-PREFETCH SHARE-LOCK
1       79      1353     1736
2       75      1364     1624
3       80      1368     2049
4       80      1328     1729
5       93      1508     1683
6       79      1368     2015
7       99      1550     1730
8       73      3714     1924
9       93      1643     1668
10      90      1426     1679
11      87      1392     1809
AVE    84.4     1637.6   1786.0

Centos 7

Run  NO-LOCK NO-PREFETCH SHARE-LOCK
1       88      7137     26541
2      102      6281     26013
3       87      7259     25303
4       82      7381     25318
5       88      6883     27381
6       84      6472     27532
7       79      6101     25775
8       87      5960     27045
9       82      6029     25494
10     111      4909     26087
11      80      6284     26263
AVE     88.2    6426.9   26250.2

Results on Centos 6 are expected:
NO-LOCK is 20 times faster than NO-LOCK NO-PREFETCH (-Mm 16384): a few messages vs. one tiny record (43 bytes in average) per network message;
SHARE-LOCK is just a bit slower than NO-LOCK NO-PREFETCH. It’s still one record per network message.

Results on Centos 7:
NO-LOCK is as fast as on Centos 6;
NO-LOCK NO-PREFETCH is 3.92 times slower than the same code on Centos 6;
SHARE-LOCK is 4.08 times slower than NO-LOCK NO-PREFETCH.


Note: the factor 4 in both two unexpected cases.

It was suggested to run ‘rngd -r /dev/urandom’ but the customer can’t find the rngd on both CentOS.

What is the root case of the slowness?
How to solve the issue?

View online

 

You received this notification because you subscribed to the forum.  To stop receiving updates from only this thread, go here.

Flag this post as spam/abuse.

Posted by George Potemkin on 11-Oct-2019 16:33

Thanks, Libor!

Posted by George Potemkin on 15-Oct-2019 14:44

Customer installed the old version Centos 7 (2015 year) and network performance was fine (30-35Gib/s according to  iperf). Then he updated OS to the recent version (using 'yum update') and performance dropped down to 12-15 Gib/s. So some of the updates of Centos 7 has an issue.

This thread is closed