I thought I’d post quickly about automating FTP using explicit TLS/SSL with cURL. No, this isn’t a particularly advanced subject in the grand scheme of things (the command line is very simple) but some of the connectivity issues you might see if you’re doing this in an enterprise environment could boggle your mind, so I’ll try to explain them.
There are a number of flavours of file transfer with FTP. A few of the favourites are, of course FTP (in the clear FTP), SFTP (SSH File Transfer Protocol), FTPES (FTP over explicit TLS/SSL) and FTPS (FTP over implicit TLS/SSL).
For the purposes of this post, I’ll be concentrating on FTPES.
FTPES will happily listen on TCP/21 and generally, that is how it is implemented. The main reason for this is so that providers can offer normal (unsecured) FTP on TCP/21 at the same time as FTPES. When the client wishes to use a secure channel, it simply issues the AUTH TLS command and in most clients I have seen, the normal standard is for clients wishing to use FTPES to immediately send the AUTH TLS command when connected. A client wouldn’t (shouldn’t) authenticate in the clear and then ask for a secure connection, that would defeat the object of having a secure connection.
The only serious problem with this implementation of a secure FTP data transfer method is that modern Unified Threat Management (UTM) devices as deployed in enterprises the world over try to enhance security of FTP by watching for outbound connections to TCP/21 using an often well hidden feature called the “FTP helper service”. This is all well and good up to the point where you need to connect to an FTPES server…
When a typical FTP session begins, the client will connect to TCP/21 and issue the USER command, followed by the PASS command with everything sent in the clear. The client connects, downloads the list of files in the directory and the FTP helper service on the enterprise firewall is happy to deal with that data because it looks like it should, FTP data.
When a typical FTPES session begins, the client will connect to TCP/21 and issue the AUTH TLS or AUTH SSL command. The FTP helper service allows the command because it is sent in the clear. It is at this point that the command channel, and thus data, sent back by the server is encrypted. Unfortunately, it is because of this encrypted data that the FTP helper service spits out its dummy because it was expecting simple clear text FTP commands. As a result of seeing what the FTP helper service deems to be garbage (actually just encrypted data), the UTM device will kill the session.
There is nothing that can be done to resolve this situation without disabling the FTP helper service on the UTM device or (if it is powerful enough) by enabling it to process encrypted sessions using a transparent man-in-the-middle style interception (intercept-decrypt-analyse-reencrypt-forward).
If the provider of the service is offering FTPES on TCP/21 then there is a fairly safe bet they offer FTPS (Implicit TLS/SSL) on TCP/990 also. The only issue here is that there is a good chance that TCP/990 isn’t a port that your firewall administrators would typically allow you to connect to, so it becomes a tough situation to resolve satisfactorily.
The main issue here is that the FTP helper service is getting in the way of the secure connection. The best way for enterprise firewall admins to think of this problem is that it’s like trying to scan HTTPS traffic as it traverses TCP/80 and expecting it to be normal HTTP traffic. The engine dealing with the scanning of data sees garbage (because the data is encrypted) so it just drops the packet.
Unfortunately the list of options at this junction is very short and very ugly from a security perspective:
- Disable the FTP helper service on the UTM device – generally this is a non-starter because doing so effectively renders the firewall incapable of scanning normal FTP (unencrypted) data for everybody that might connect to an FTP site. The purpose behind this change is to prevent the UTM device from trying to scan the traffic as normal (in the clear) FTP traffic – for FTPES the traffic is encrypted, so it simply won’t work.
- Connect to the service provider’s FTPS service on TCP/990 – If they’re offering FTPES, they’re probably also offering FTPS (FTP over implicit TLS/SSL) too but you will need to persuade your firewall administrator that opening TCP/990 is OK. They will be able to secure the IP and port combination and the FTP helper service won’t interfere on this port so this is the preferred fix.
OK, and with that out of the way, if you don’t have an enterprise UTM device trying to be overly helpful, this is a typical cURL command line to upload a file to an FTPES server.
curl.exe -T "d:\myfile.txt" ftp://username:password@server.domain.com -k --ftp-ssl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 315k 0 0 100 315k 0 89899 0:00:03 0:00:03 --:--:-- 90708
This will upload a file called
myfile.txt from
D:\ to
ftp://server.domain.com (on the default port TCP/21!) and will authenticate as
username with the password
password, ignoring any certificate errors (
-k) and authenticating using FTPES (
–ftp-ssl).
Of course if you’re automating on Windows (like me) you’ll need a windows binary for cURL that has the SSL library compiled in. You can get that from the cURL downloads page. Scroll down to Win32 – Generic and select the binary with SSL support. [At the time of posting, the latest release is 7.21.0 and the binary I used was built by Günter Knauf]
Don’t forget to read the man[ual] page that comes with the download for more details on automating at the command line with cURL.
Hope this helps you!
EDIT: You may be concerned that when you connect to an FTPES or FTPS server using cURL that you have to specify the -k option to ignore certificate validation errors and it seems that the server’s certificate isn’t verified by a trusted root CA. This doesn’t necessarily mean that the server’s certificate is invalid. In the most recent binary builds, cURL no longer comes bundled with the public keys of valid root CAs like Verisign, Thawte etc. so, you have two options. 1. Use the VBScript that’s included with the binaries to generate your own (when I ran it, it didn’t work so YMMV) or, 2. Provide your own bundle of verified public root CA certificates by visiting the cURL website, click the “CA Extract” link in the menu, download the cacert.pem file in to the same directory as your cURL binary and rename it to curl-ca-bundle.crt. All of these instructions can be found at http://curl.haxx.se/docs/sslcerts.html which is also mentioned in the error message about the cert that cURL spat out when you connected to the server without the -k option.
-Lewis
Yes, I’m finally, FINALLY Microsoft certified. I’ve been doing this stuff successfully for ten years without the compulsion to get myself certified but I decided recently to add credence to my experience by taking the exams.
I’ve spent a few minutes thinking about this one this morning and came up with what I believe is a decent solution to a common problem. Despite trawling the net for ten minutes or so, I haven’t seen this solution elsewhere, though I have seen some overwhelmingly over-complicated solutions which probably work.
ETN.fm