TIP: Amazon Web Services Operates Split-horizon DNS

While making some tweaks to our AWS setup we learnt that Amazon run a Split-horizon DNS setup. This is only really relevant if you have services that are internal (or backend) to your overall system such as a database (not RDS) behind a web server. And you want to run it off a static IP (Elastic IP) as you should, plus a custom DNS entry such as mydatabase3.abc.com.

In the above scenario you would have you database secured with a rule only allowing access to the web server. You do this by adding a firewall rule by Security Group name i.e. any EC2 instance with the given Security Group attached (the one defined by name in the rule) is allowed access to the database ( more on EC2 Security Groups). Then you attach an Elastic IP to your EC2 database instance, create an A Record under your domain pointing to the Elastic IP, then use this A Record in the connection string in your application running on the web server.

This will not behave as you expect, your web server will lose connectivity to the database. The reason for this is that the Public DNS name of the database server you were using, on a EC2 instance,  still resolves to it’s private IP address. Where as from your PC in the office or at home it will resolve to the a Public IP address i.e. Split-horizon DNS. Firewall rules based on security group names resolve against private IP addresses to identify EC2 instance therefore allowing this traffic. When you try and connect using your custom DNS name (which always resolves to the Elastic IP) the database server is not receiving traffic from the a private IP the firewall can associate back to an instance and hence back to attached security groups  hence blocks the traffic.

The solution is to always create custom DNS names as CNames to the EC2 Public DNS record rather than A Record directly resolving to the Elastic IP and all will work nicely.