lewisroberts.com

Well, it’s not complete FUD – it did happen and it was simply to highlight that rural broadband is slow but let’s be serious, are they actually going to start strapping USB keys to carrier pigeons to transfer data? Yes, that was a rhetorical question. Take your mouse point from the comment box now please…

I would like to point out that they were testing an upload but then complaining about browsing speed. At least get your test right if you’re going for as much publicity as you can. I bet I could stick a Blu-ray on a flight to Turkey and it beat my upload speed and I live 5 miles outside of a major city! Never mind rural broadband! Urban broadband is slow! Tell the Prime Minister! Tell the Queen! We need regulations!

The article goes on to complain that school children need to be able to access the internet for learning purposes. OK, fair comment in this day and age but if it’s so important for a child’s learning (and that is the priority they’re bleating (pun intended) about), why not move to somewhere that there is decent broadband at your child’s school?

On a side note, I didn’t need to use the internet to come out of school with decent grades and then embark on a career in computing. In fact I didn’t touch a computer throughout the whole of high school! Just saying

I’ve made observations previously about rural broadband and the phone companies’ reluctance to put in the infrastructure in a previous blog entry. Digital Britain = Tax.

-Lewis

So, you’re attempting to grant some users permission to read the event log on a Windows Server 2003 server and all of a sudden you’re plunged deep in to the world of SDDL and needing to amend a random registry entry to grant access. You’re under pressure. What do you do?

Task:

Grant access for a user account to read the Application Event log but without making them a member of Local Administrators or any other local group.

Solution:

Locate and edit the following registry key:

HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\Application > CustomSD

Problem 2:

The key above is in SDDL format and it’s not something you’ve seen before. What’s it all about and how can you edit it without breaking it? Example:

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0x7;;;BA)

Solution 2:

The SDDL format can be broken down in to four parts colour coded in red, green, blue and orange below:

O:BAG:SYD:(D;;0xf0007;;;AN)(D;;0xf0007;;;BG)(A;;0xf0007;;;SY)(A;;0×7;;;BA)

O = Owner
G = Primary Group
D = DACL Entries
S = SACL Entries (not included in example)

The owner (O:) of the object in the above example is the Builtin Administrators group (BA). You can decipher the SID String entry from the list provided by Microsoft here: http://msdn.microsoft.com/en-us/library/aa379602.aspx

The primary group (G:) of the object in the above example is the Local System (SY). Again, decipher the SID string entry from the list provided by Microsoft: http://msdn.microsoft.com/en-us/library/aa379602.aspx

Then we have the Discretionary Access Control List entries (D:) followed by several parentheses which are the actual DACL entries. Basically these entries wrapped in parentheses are the same entries you can see when you open up the security tab on any file but these entries are written in the SDDL format. Each set of parentheses contains all of the information for an entry.

Our first entry in the DACL section (D;;0xf0007;;;AN) breaks down using the following format separated by semi-colons.

ace_type;ace_flags;rights;object_guid;inherit_object_guid;account_sid

In our example you’ll notice there are some empty entries between the semi-colons, this is OK and actually quite normal for most entries written in SDDL format.

So, again using Microsoft’s helpful (well, mostly) website at http://msdn.microsoft.com/en-us/library/aa374928(v=VS.85).aspx, we can decipher what each of these separate entries do for us when we apply it to our example. The account_sid entry relates to the SID String (link above)

ace_type: D (Deny)
ace_flags: N/A
rights: 0xf0007 (a hexadecimal string which denotes the access mask)
object_guid: N/A
inherit_object_guid: N/A
account_sid: AN (Anonymous)

OK, the only point of confusion above is really the rights section, probably the most important of all of the sections above. Here, in the SDDL links provided by Microsoft you can use either a hexadecimal string (as in our example) or a concatenation of any of the many strings listed at http://msdn.microsoft.com/en-us/library/aa374928.aspx

Converting the access mask in our example in to strings is nigh on impossible because Microsoft don’t give us the conversion values for each of the strings on their website which is kind of frustrating for me since I can’t explain how best to create an SDDL entry using a hex string. My recommendation is to build the permissions you require using the concatenate strings method.

Let’s assume our access mask defines Full rights. When combined with the rest of the DACL entry, it actually denies full access rights to the Anonymous user. We can probably re-write our DACL entry, replacing the hex mask as follows:

(D;;GA;;;AN)

All I’ve done is to replace the 0xf0007 part of the DACL with GA, which is the GENERIC_ALL right as listed on http://msdn.microsoft.com/en-us/library/aa374928.aspx

If we wanted this same entry to grant full permission to the Anonymous user account (NOT A GOOD IDEA!!!) then we would write it as follows:

(A;;GA;;;AN)

Hopefully this little intro has given you the start you need to edit your own SDDL entries. If anyone fancies adding more examples with explanations, please add your comments below.

-Lewis

Yes, I’m finally, FINALLY Microsoft certified. I’ve been doing this stuff successfully for ten years without the compulsion to get myself certified but I decided recently to add credence to my experience by taking the exams.

This is step one on the MCITP Enterprise Administrator track and hopefully this logo should grow and finally become an MCITP (Microsoft Certified IT Professional) in the next few months as I prepare and take the exams required.

Once I’ve got my MCITP I’ll follow up with the CCNA and then back to Microsoft for Exchange (Messaging) and probably SCCM to round things off. The next few months should be interesting and if I’m honest, I’m looking forward to it!

-Lewis

…is not made easy by our friends at VMWare.

Oh sure they’ll provide you with a tool called VMWare Converter Standalone which purports to convert your VMWare workstation guests in to fully fledged ESXi compatible versions but does it actually work? In short, no, it doesn’t.

I’m sure that statemt’ll wrangle some VCPs and possibly VMWare themselves but I simply can’t give it anything that could be misconstrued as a seal of approval. The damn piece of software just doesn’t work.

Let me quantify that statement with an example: Last weekend I began the process of building my new ESXi 4 server. Having an existing one that I wanted to tear down and use some parts from, I decided I’d use VMWare Converter Standalone to take the machines from ESXi 4 and put them on my Workstation installation for a short spell while I decommissioned and tore down the old server for parts. Now, to give it its dues, VMWare Converter worked wonderfully for this part of the process and I was left confident that I would be able to reverse the process once the new server was built. Oh how wrong I was.

Having built my new ESXi 4 server, the following 3 days turned in to a litany of disasters and failed attempts to persuade VMWare Converter that if it REALLY tried hard enough, it might actually successfully convert a machine and put it on the ESXi 4 server.

I had two machines I wanted to convert. One is a Fedora 13 install (the one that runs this site) and the other is a Windows Server 2003 R2 box which I run my non Exchange mail server from. Nothing special about these machines and only the Windows box has a second hard drive that is used to store the mail databases so it should be easy.

Fedora 13 box has 1 x 40GB thin provisioned VMDK. The Windows server has 1 x 12GB thin provisioned VMDK and 1 x 30GB thin provisioned VMDK for the databases.

Attempt 1: Simple conversion of Windows Server. Failed to clone disk 1. Time = 3 hours.
Attempt 2: Simple conversion of Windows Server. Failed to clone disk 1. Time = 3 hours.
Attempt 3: Copy of source files of Windows Server to ESX server. Failed to identify source disk V7. = 3 hours.
Attempt 4: Creation and deployment of OVA for Windows Server. Failed to deploy.
Attempt 5: Use Workstation 7 for creation and deployment of OVA for Fedora 13. Success! Huzzah. Quick tinker with settings and it was up and running.
Attempt 6: Use Workstation 7 for creation and deployment of OVF for Windows Server. Failed to deploy. Checksum error.
Attempt 7: Investigate alternatives. Spend two hours banging my head off the desk trying to come up with a solution.
Attempt 8: Copy the files like in attempt 3. Run vmkfstools to convert (clone) the disks directly on the ESX server. Manually rename the disk file name references in the VMX file. Cross fingers, arms, legs, toes, hair folicles. Success!

Yes ladies and gentlement, the only viable option for converting a VMWare Workstation  7 guest to an ESXi host is copy/paste and vmkfstools.

To give you the full info, the command I ran to convert the disks was:

vmkfstools -i "my-source-disk.vmdk" -d zeroedthick "my-destination-disk.vmdk"

I ran the command (obviously swapping the names appropriately) on each of the disks and then manually edited the VMX file to replace the old names with the new disks and it worked. I had some additional playing about to do with the CD-ROM and a couple of other bits and pieces but it took at least eight attempts and a number of clumps of my hair to get those guests on to ESXi.

Not a process I want to have to repeat again any time soon.

If you’re looking for the full breakdown of what I did:

1. Use Browse datastore to create a folder for the source guest’s files.
2. Upload the source machines guest files to the folder created above.
3. On the ESXi host machine, enable the Remote Tech Support Mode (SSH) from Troubleshooting Options. F2 > Login > Troubleshooting Options
4. Download Putty (an SSH client) and connect to your ESX server. Login as root with your password.
5. Change directory in to the folder where you uploaded your guest’s files. This is usually located in /vmfs/volumes/[datastore]/[Folder created in step1]
6. Use the vmkfstools command to import your VMDK. This process will create a new VMDK that is compatible with ESXi.

   vmkfstools -i  -d zeroedthick [destination.vmdk]

7. Rename your original source vmdk file(s) by using the mv command:

   mv  

8. Open the VMX file with Vi (a text editor)

   vi 

9. Find the references to the files and edit them to reflect the new names of the [destination.vmdk] files. Use the Insert key to edit and then press Esc when you have finished editing.
10. Save the edited VMX file. :w
11. Exit Vi. :q
12. Back in vSphere Client, Browse datastore and import the virtual machine VMX file.
13. Your VM should start up.
14. It’s up to you if you want to delete your files to save space.

-Lewis

My usual process for deploying Windows Server boxes involves the installation of VNC for remote console management. I know Remote Desktop is the best practice way to do this but as they say, old habits die hard.

When I recently deployed a Windows Server 2008 R2 x64 box for the purposes of running IIS7.5 at another site, I followed my usual process and of course installed VNC Enterprise (4.5.1).

We have a fairly modest application server farm so investing time and effort in deployment techniques for .NET 4 web applications to just a few servers seems a little like overkill so we generally just use VNC’s rather useful copy/paste to dump the binaries on the box. Now we’ve been using this method on our Windows Server 2003 R2 IIS6 application servers for a couple of years without any issues whatsoever. Again, I realise this isn’t “best practice” but there’s little point jumping through hoops and over-complicating matters as it just means spending more time than is really required.

However, after deploying several copies of our application to our IIS6 app servers and then the same copy of the application to the first IIS7.5 app server, we were a little perplexed to see that the IIS7.5 box wasn’t playing ball and was giving strange errors when we tried to access the back-end services. The error seemed to take us in the direction of a 32-bit application incompatibility but even after forcing 32-bit (WOW64 mode) for the application pool, we still got the same error message.

All this occurred on a Sunday and obviously being in weekend mode, I didn’t investigate quite as thoroughly as I should have. The box is pre-production so it wasn’t a huge concern that the app didn’t work though it was cause for some concern and would require investigation to resolve.

After discovering a really useful alternative tool to BgInfo called DesktopInfo, I thought I’d copy it over to the Windows Server 2008 R2 x64 app server this morning to give it a try. The server isn’t in production yet so it seemed the perfect time to give it a go. I fired up VNC and simply copied and pasted the folder with the binary over to the target server. After the copy completed successfully, I executed the binary only to be told it wasn’t a valid Win32 application. Err, what? OK, I know this is a 64-bit OS but it’ll still run 32-bit binaries… oh… hang on a minute!

I immediately dropped the VNC connection, fired up Remote Desktop and re-copied the folder, executed the binary and had no problems at all so I re-copied the previously failing application to the Windows Server 2008 R2 x64 box using Remote Desktop. I fired up the application in Internet Explorer and it worked without a problem!

Obviously I’ll have to write this one down to experience. Maybe the latest version of VNC Enterprise 4.5.4 has rectified the binary file copy corruption that seems to occur when using the copy/paste functionality on a target that is Windows Server 2008 R2 x64 but until I’ve tested that, I think I’ll stick to Remote Desktop.

The old habit might just have to die after all…

-Lewis

I thought I’d post quickly about automating FTP using explicit TLS/SSL with cURL. No, this isn’t a particularly advanced subject in the grand scheme of things (the command line is very simple) but some of the connectivity issues you might see if you’re doing this in an enterprise environment could boggle your mind, so I’ll try to explain them.

There are a number of flavours of file transfer with FTP. A few of the favourites are, of course FTP (in the clear FTP), SFTP (SSH File Transfer Protocol), FTPES (FTP over explicit TLS/SSL) and FTPS (FTP over implicit TLS/SSL).

For the purposes of this post, I’ll be concentrating on FTPES.

FTPES will happily listen on TCP/21 and generally, that is how it is implemented. The main reason for this is so that providers can offer normal (unsecured) FTP on TCP/21 at the same time as FTPES. When the client wishes to use a secure channel, it simply issues the AUTH TLS command and in most clients I have seen, the normal standard is for clients wishing to use FTPES to immediately send the AUTH TLS command when connected. A client wouldn’t (shouldn’t) authenticate in the clear and then ask for a secure connection, that would defeat the object of having a secure connection.

The only serious problem with this implementation of a secure FTP data transfer method is that modern Unified Threat Management (UTM) devices as deployed in enterprises the world over try to enhance security of FTP by watching for outbound connections to TCP/21 using an often well hidden feature called the “FTP helper service”. This is all well and good up to the point where you need to connect to an FTPES server…

When a typical FTP session begins, the client will connect to TCP/21 and issue the USER command, followed by the PASS command with everything sent in the clear. The client connects, downloads the list of files in the directory and the FTP helper service on the enterprise firewall is happy to deal with that data because it looks like it should, FTP data.

When a typical FTPES session begins, the client will connect to TCP/21 and issue the AUTH TLS or AUTH SSL command. The FTP helper service allows the command because it is sent in the clear. It is at this point that the command channel, and thus data, sent back by the server is encrypted. Unfortunately, it is because of this encrypted data that the FTP helper service spits out its dummy because it was expecting simple clear text FTP commands. As a result of seeing what the FTP helper service deems to be garbage (actually just encrypted data), the UTM device will kill the session.

There is nothing that can be done to resolve this situation without disabling the FTP helper service on the UTM device or (if it is powerful enough) by enabling it to process encrypted sessions using a transparent man-in-the-middle style interception (intercept-decrypt-analyse-reencrypt-forward).

If the provider of the service is offering FTPES on TCP/21 then there is a fairly safe bet they offer FTPS (Implicit TLS/SSL) on TCP/990 also. The only issue here is that there is a good chance that TCP/990 isn’t a port that your firewall administrators would typically allow you to connect to, so it becomes a tough situation to resolve satisfactorily.

The main issue here is that the FTP helper service is getting in the way of the secure connection. The best way for enterprise firewall admins to think of this problem is that it’s like trying to scan HTTPS traffic as it traverses TCP/80 and expecting it to be normal HTTP traffic. The engine dealing with the scanning of data sees garbage (because the data is encrypted) so it just drops the packet.

Unfortunately the list of options at this junction is very short and very ugly from a security perspective:

1. Disable the FTP helper service on the UTM device – generally this is a non-starter because doing so effectively renders the firewall incapable of scanning normal FTP (unencrypted) data for everybody that might connect to an FTP site. The purpose behind this change is to prevent the UTM device from trying to scan the traffic as normal (in the clear) FTP traffic – for FTPES the traffic is encrypted, so it simply won’t work.
2. Connect to the service provider’s FTPS service on TCP/990 – If they’re offering FTPES, they’re probably also offering FTPS (FTP over implicit TLS/SSL) too but you will need to persuade your firewall administrator that opening TCP/990 is OK. They will be able to secure the IP and port combination and the FTP helper service won’t interfere on this port so this is the preferred fix.

OK, and with that out of the way, if you don’t have an enterprise UTM device trying to be overly helpful, this is a typical cURL command line to upload a file to an FTPES server.

curl.exe -T "d:\myfile.txt" ftp://username:password@server.domain.com -k --ftp-ssl

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 Dload  Upload   Total   Spent    Left  Speed
100  315k    0     0  100  315k      0  89899  0:00:03  0:00:03 --:--:-- 90708

This will upload a file called myfile.txt from D:\ to ftp://server.domain.com (on the default port TCP/21!) and will authenticate as username with the password password, ignoring any certificate errors (-k) and authenticating using FTPES (–ftp-ssl).

Of course if you’re automating on Windows (like me) you’ll need a windows binary for cURL that has the SSL library compiled in. You can get that from the cURL downloads page. Scroll down to Win32 – Generic and select the binary with SSL support. [At the time of posting, the latest release is 7.21.0 and the binary I used was built by Günter Knauf]

Don’t forget to read the man[ual] page that comes with the download for more details on automating at the command line with cURL.

Hope this helps you!

EDIT: You may be concerned that when you connect to an FTPES or FTPS server using cURL that you have to specify the -k option to ignore certificate validation errors  and it seems that the server’s certificate isn’t verified by a trusted root CA. This doesn’t necessarily mean that the server’s certificate is invalid. In the most recent binary builds, cURL no longer comes bundled with the public keys of valid root CAs like Verisign, Thawte etc. so, you have two options. 1. Use the VBScript that’s included with the binaries to generate your own (when I ran it, it didn’t work so YMMV) or, 2. Provide your own bundle of verified public root CA certificates by visiting the cURL website, click the “CA Extract” link in the menu, download the cacert.pem file in to the same directory as your cURL binary and rename it to curl-ca-bundle.crt. All of these instructions can be found at http://curl.haxx.se/docs/sslcerts.html which is also mentioned in the error message about the cert that cURL spat out when you connected to the server without the -k option.

-Lewis