Author Topic: SFTPCiphers and performance  (Read 423 times)

Offline dak

  • New user
  • *
  • Posts: 5
    • View Profile
SFTPCiphers and performance
« on: March 09, 2017, 03:22:01 pm »
Hi,
So. Performance.

I'm transferring large files between sites that have a 1Gbit uplink. Some times our clients need to upload somewhere along the 500GB range. I don't want to use FTPS because it's caused connection/firewalling problems for me and is nearly impossible to debug. mod_sftp is the way and the light.

We've had some performance issues. Some clients have reported speeds as low as 1MB/s and after extensive testing and debugging, I'm starting to presume the selected ciphers are at fault.

Test protocol:
- Client is  'scp' command on a Debian 8 server.
- Server is ProFTPd with mod_ssl on a Debian 8 server. ProFTPD Version 1.3.5
- Test file is a 10GB file containing random data. generated using "dd if=/dev/urandom of=random-10G bs=10M count=1024"
- All tests had full use of 1 core on the Client (Xeon CPU in a server). All tests saturated the CPU to 100% usage, so I'm assuming that the transfer speed is CPU-bound.
- ProFTPd Server machine never reached 100% CPU usage
- ProFTPd Server machine is hosted on SSD SAN with 500MB/s+ write speeds available.

Mistakes:
- There are other things running on the "client" machine. It is a KVM server. While there are 12 available threads, there's at least 11 virtual machines running on the same machine, however most of them are idle.
- There are other things happening on my network and the 1Gbit uplink to the data center is not always 100% available to me.

Further tests should control for these two variables, however I'd like to post my initial results.

cipher - speed - time
aes256-ctr  22.7MB/s   07:31   
aes192-ctr  42.5MB/s   04:01     
aes128-ctr  15.8MB/s   10:48   

aes256-cbc  37.1MB/s   04:36   
aes192-cbc  26.5MB/s   06:26     
aes128-cbc  23.6MB/s   07:14   

blowfish-cbc  43.0MB/s   03:58   
cast128-cbc  41.3MB/s   04:08 
3des-cbc  13.5MB/s   12:39
 
arcfour256  33.4MB/s   05:07   
arcfour128  36.7MB/s   04:39   

What surprised me the most is that aes192-ctr was faster than aes128-ctr. I'm blaming this on one of the variables listed under "mistakes" though.

When transferring the same file using OpenSSH as Server and using aes256-gcm@openssh.com as cipher, I get transfer speeds upwards of 50-70MB/s, and the client is no longer using 100% CPU (hardware acceleration), so the bottleneck is moved to some other place.

My question is this:
- Are there any plans of supporting any extra ciphers in mod_sftp?
- Any chance that there will be hardware accelerated ciphers in the future?
- Are there any other ways I can improve transfer speeds, other than prioritizing the lower-bit aes ciphers in my SFTPCiphers configuration?

Thank you in advance.
« Last Edit: March 09, 2017, 03:24:47 pm by dak »

Offline dak

  • New user
  • *
  • Posts: 5
    • View Profile
Re: SFTPCiphers and performance
« Reply #1 on: March 09, 2017, 03:30:33 pm »
Tests using OpenSSH as server:
Code: [Select]
scp -c aes256-gcm@openssh.com random-10G xxx@xxx:
random-10G                                                                                   100%   10GB  45.7MB/s   03:44
scp -c chacha20-poly1305@openssh.com random-10G xxx@xxx:
random-10G                                                                                   100%   10GB  54.5MB/s   03:08

Offline dak

  • New user
  • *
  • Posts: 5
    • View Profile
Re: SFTPCiphers and performance
« Reply #2 on: March 09, 2017, 04:34:49 pm »
Another batch of tests. These results are all over the map.
Code: [Select]
$ for cipher in aes256-ctr aes192-ctr aes128-ctr; do echo $cipher; scp -c $cipher -P2222 random-10G xxx@xxx:; done;
aes256-ctr
random-10G                                                                                   100%   10GB  31.1MB/s   05:29   
aes192-ctr
random-10G                                                                                   100%   10GB  19.3MB/s   08:52   
aes128-ctr
random-10G                                                                                   100%   10GB  19.1MB/s   08:55   

$ for cipher in aes256-ctr aes192-ctr aes128-ctr; do echo $cipher; scp -c $cipher -P2222 random-10G xxx@xxx:; done;
aes256-ctr
random-10G                                                                                   100%   10GB  44.7MB/s   03:49   
aes192-ctr
random-10G                                                                                   100%   10GB  30.9MB/s   05:31   
aes128-ctr
random-10G                                                                                   100%   10GB  25.2MB/s   06:46   

$ for cipher in aes256-ctr aes192-ctr aes128-ctr; do echo $cipher; scp -c $cipher -P2222 random-10G xxx@xxx:; done;
aes256-ctr
random-10G                                                                                   100%   10GB  37.1MB/s   04:36   
aes192-ctr
random-10G                                                                                   100%   10GB  44.1MB/s   03:52   
aes128-ctr
random-10G                                                                                   100%   10GB  45.1MB/s   03:47   

It's surprising to me that aes256-ctr shows higher throughput than the other ciphers. Could anyone illuminate?
« Last Edit: March 09, 2017, 04:53:25 pm by dak »

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5373
    • View Profile
    • http://www.castaglia.org/
Re: SFTPCiphers and performance
« Reply #3 on: March 10, 2017, 05:09:22 pm »
For comparison, you might run the same tests using 'sftp', instead of 'scp'.  In particular, I am wondering about the amount of bidirectional traffic during the transfer, i.e. are you getting close to the full bandwidth-delay product of your link during the file transfer?

Offline dak

  • New user
  • *
  • Posts: 5
    • View Profile
Re: SFTPCiphers and performance
« Reply #4 on: March 13, 2017, 01:08:19 pm »
I'll run some tests and report back :)

Offline dak

  • New user
  • *
  • Posts: 5
    • View Profile
Re: SFTPCiphers and performance
« Reply #5 on: March 13, 2017, 03:20:10 pm »
Using SFTP, here are my results. The ciphers I used are limited to the ones I deemed the fastest in my previous testing.


Code: [Select]
SFTP tests
aes256-ctr:
37.2MB/s   04:35
34.4MB/s   04:58
40.4MB/s   04:13
41.6MB/s   04:06

aes192-ctr:
45.6MB/s   03:44
32.3MB/s   05:17
43.6MB/s   03:54
40.2MB/s   04:15

aes128-ctr:
39.8MB/s   04:17
29.9MB/s   05:42
45.6MB/s   03:44
45.8MB/s   03:43

blowfish-cbc:
43.5MB/s   03:55

cast128-cbc
43.3MB/s   03:56

One thing to note, is that the SFTP client is not using 100% CPU like SCP did. The CPU usage on the server jumps from 70-80% to 80-99% when using SFTP instead of SCP.

Offline castaglia

  • Administrator
  • Support Hero
  • *****
  • Posts: 5373
    • View Profile
    • http://www.castaglia.org/
Re: SFTPCiphers and performance
« Reply #6 on: March 18, 2017, 05:33:41 pm »
OK, interesting info.  As for the topic of adding higher-performing ciphers to mod_sftp, e.g. the GCM suites and ChaCha20-Poly1305, see:

  Support AES Galois Counter Mode (AES-GCM) in SSH
    http://bugs.proftpd.org/show_bug.cgi?id=3759

  Support chacha20-poly1305@openssh.com SSH cipher
    https://github.com/proftpd/proftpd/issues/456