NetMatch and IPv6
NetMatch is a company that specializes in making complex web applications for (but not limited to) the travel industry. Besides that, NetMatch also hosts its own sites, and provides hosting to its customers.
At NetMatch we embrace new technologies if they can improve the way we work, or if they allow us to build better applications for our customers. IPv6 certainly does that for us, as I will explain below.
This page was created for fellow content providers to get started with IPv6. Perhaps if there's more to say in the future, I'll make a blog around it or something.
Why IPv6?
I assume all readers are familiar with IPv6. If not, look on Wikipedia for a technical introduction, or here for less technical introductions (Dutch). We all know IPv6 is coming, and a lot of us are actively implementing IPv6 right now. We all know we're running out of IPv4 addresses and that NAT is Evil, so I won't go there. I will look at things from the perspective of a content provider here. The intended audience is systems administrators and decision makers on IPv6.
IPv6 is coming, and we all know it. At the time of writing (Oct 2010) the estimated date when we'll run out of IPv4 blocks is June 2011, 18 months away. No one knows what will happen exactly at that date, but my guess is that it will be almost impossible to get IPv4 blocks, or they will become extremely expensive. This shouldn't be a problem, because that's where IPv6 comes to the rescue. It is a problem though, because not everyone is doing IPv6 yet. If you publish a new site on IPv6 only, then IPv4-only clients won't be able to access it. So you still need those pesky old IPv4 addresses around for a while.
Where are we now?
In the ideal world, the entire Internet would use IPv6. This means all content providers would have their services enabled over IPv6, and all components in the path to the end user, like ISP's, home routers and end-user computers, are IPv6 enabled. However, there seems to be a chicken-and-egg problem here. Content providers are reluctant to support IPv6 because too little end-users use IPv6, and ISP's don't implement IPv6 because too little content providers have services available over IPv6. The whole IT sector has been putting the transition off for too long, except for some early adopters.
Some big players proposed a timeline for introducing IPv6, and application porting was to begin around 2002, ISP adoption around 2002, consumer adoption around 2003 and enterprise adoption around the same time (source). We all know how this went, and even in 2010 some companies haven't even heard of IPv6. The millenium bug was the last big industry-wide project that needed this amount of attention, and one reason often heard for putting off IPv6 implementation is that apparently a lot of IT-managers are thinking along the lines of: "Well, the millenium bug wasn't this big of a deal as they made me believe, so I guess this IPv6 thing won't be that important as well". Guess again :)
Face it.. As Randy Bush said: IPv6 was not economically interesting. You have to put a lot of energy and money in fixing something that already works for now. That it will not work in the future is a worry for later.
Luckily, in the past few months we've seen more and more IPv6 awareness amongst both internet providers and content providers, and I think all ISP's are currently testing IPv6. I think the IT-managers at those companies probably feel the IPv4 address depletion issue breathing down their necks now. Here in the Netherlands some ISP's offer IPv6 to their customers, though the customers all have to opt-in. The wait is still for the first Dutch ISP to start handing out IPv6 ip's natively.
On the content provider side where I am, there doesn't seem to much willingness. I think this is because for most content providers, the network infrastructure is not their core business. The core business is getting apps running, and making sure they are fast and reliable. IPv6 is just another thing that's on the agenda somewhere, and it's not a top priority. It would be a top priority if there were IPv6-only clients out there, but there aren't. Another thing that is hindering IPv6 deployment is the lack of supplier support. Sure, most suppliers are doing something with IPv6, but there are still a lot of load balancers or other appliances which can't do the same over IPv6 as they can over IPv4.
Another incentive that is helping the adoption is user demand. Companies will start requiring IPv6 from their suppliers, be it for their networking infrastructure, their ISP's hosting environment, or for internet connectivity. So the less capable suppliers, which can't offer IPv6-enabled products yet, will be left out. Here we'll see survival of the fittest as well, and hopefully, companies which don't have their priorities straight will either have to make a run for it, or go out of business.
According to the IPv6 Matrix currently 1.5% of the found DNS hosts in the world have IPv6 records (4% in NL). That's not much.
The good thing is, now you have the time to think about it, and get experience with IPv6 in your own pace. Nobody knows how things will look after all the IPv4 netblocks run out, but you might have to make a run for it, studying all weekend to get the next project done for your boss, and suddenly it's IPv6 here, and IPv6 there. If I were you, I'd start now..
What does NetMatch do?
Since 2002-2003 we've been playing around with IPv6, we've been using it in production since around 2005 for email, and since around 2006-2007 for DNS and HTTP. There aren't many end-users accessing our services over IPv6, but hey, someone has to break the chicken-and-egg circle.
Now, in 2010, all of our new services are IPv6 enabled, and we try to change every old service to IPv6 as well when we do some maintenance on it (or decommission old servers). Some of the larger project we have like http://www.zoover.nl, http://www.weeronline.nl and http://www.snp.nl for example, have been IPv6 enabled for years. We have some sites running on antique IIS6 servers, where we don't use IPv6, and we have some other issues where IPv6 is not possible yet, like some mail server software not supporting IPv6. At the time of writing, out of the 1318 www records in our DNS, 971 have AAAA records. This number isn't very accurate though, since some of our customer's domains are managed by third-party DNS servers, and some of their DNS managers have never even heard of AAAA records.
As you will read below, we're not done yet. There are still some unresolved issues, and we still have some software running here and there that doesn't support IPv6. However, that's no reason not to go forward yet.
Great, how do I start?
Simple, just start making services are available over IPv6. Just start with one, then move on to the next, etc. :)
Now the biggest problem with implementing IPv6 is a lack of knowledge about IPv6. There is probably a ton of resources on the net that can help you with that. The second biggest problem I think when designing and implementing IPv6 in a network infrastructure is the fact that your network might need to change. You've probably already got an architecture in place with NAT. With IPv6, where you won't be doing NAT, it can be confusing, as you will probably have NATted and non-NATted traffic over the same network. This does not make things easier. Below, I'll show some example configurations, perhaps they'll give you some inspiration for updating your own network.
IPv6 on your management workstation
First thing you will probably want to do, is get IPv6 internet access on your management workstation in your office. (Most content providers have their servers at a colocation provider, and their own management workstations in their office, in a seperate network). It's nice to have your web sites available to others on the net over IPv6, but if you won't be able to access them from your office, that will make testing and troubleshooting a lot more difficult. If your ISP does not support IPv6, get a tunnel at sixxs or HE and get it up and running.
Get IPv6 from your colo ISP
Then get a subnet from your colocation ISP. This might be the trickiest part, based on the ISP you're using. At datacenters like xs4all, BIT, Interconnect, etc, you won't have a problem getting a subnet. Whether you need a single subnet or more, depends on the architecture you're choosing. Look at the configurations below. If your ISP doesn't do IPv6, move your services to a more competent ISP. Do you really want to have your servers and data in the network of an ISP thatsets its priorities like this?
Addressing
When you have the subnet, you're ready to go. Decide whether you want to use static configuration or autoconfigured addresses for your hosts. Since at NetMatch we do all our IPv4 server configuration static, we chose static configuration for IPv6 as well. As an example: one of the subnets xs4all gave us is 82.94.225.0/128. One of the IPv6 subnets we got for this network is 2001:888:2085::/64. So when I have a host with ip 82.94.225.2, I could give it a static IPv6 address 2001:888:2085::2. This makes it easy for me to remember ip addresses and their relationships to IPv4 addresses. Of course this makes it easier for attackers to guess ip addresses as well, and port scan sweeps are easier than when you use some random EUI64 network address. Everyone should decide for themselves.
Enabling IPv6
Now you've decided on what ip address to use for a server, just add it to the operating system, and make sure your services listen on IPv6 addresses. Some applications, like Postfix and Bind, two well know open source applications on Linux/Unix, have a configuration setting that lets them listen on IPv6. Punch holes in firewalls if needed, and try to connect and test your application.
SMTP
Of course you would first enable IPv6 on your test servers, and test there. If you don't have dedicated test machines, you might want to start with enabling SMTP over IPv6. Just make sure your mail server responds to SMTP over IPv6, and add an AAAA record to that host in DNS. That's it, now you can monitor your SMTP server log files and see what's happening. The reason I'm suggesting you start with mail, is that SMTP is quite a forgiving protocol. If a remote server doesn't understand AAAA records, no problem. If the server speaks IPv6 and it can't connect over IPv6, no problem, it will just connect over IPv4. Stuff won't just suddenly break.
Keep in mind that when you use SPF records, these might need to be updated as well to include the IPv6 subnets.
HTTP
With HTTP it's a bit of a different story.If a client supports IPv6, browsers like IE and Firefox will try to connect using IPv6 first, and if that doesn't work, after about 30 seconds they'll try again over IPv4. Users will either run away because they don't want to wait 30 seconds, or they'll call you complaining that the site is so slow. Not good.
Another thing to keep in mind, is that not all applications might handle IPv6. For example in forum applications, it's quite common to record ip addresses with posts, and ban users based on ip addresses. If the forum sees an IPv6 address, what happens? The application might just start throwing errors because the ip address is not in the format it is expecting. Applications need to be tested for problems like these. If you have a staging and a production site, you could add an AAAA record to only the staging site, and then have people test the application before enabling it on production.
Monitoring
This brings me to another point, which is monitoring. Most content providers have a monitoring solution in place to make sure all their web sites are up and running. Does that monitoring solution support IPv6? Let's say your website works fine over IPv4, but because of a misconfiguration, the server can't be reached over IPv6. Or exactly the other way around. In an ideal situation, you would be notified of both as soon as the problem occurs, instead of customers calling you that the website takes 30 seconds to load. Think about monitoring, and what you want.
At NetMatch, this is an issue we are currently working on. We are in the process of moving from SolarWinds IPMonitor to Nagios, because the former does not support IPv6, and is not planning to in the near future. We're also not exactly sure yet what the best way of monitoring is in relation to IPv6. Should we create 2 hosts for each server, one for IPv4 and one for IPv6, and then make the IPv6 host dependant on the IPv4 host? And should we add every service twice? It's a work in progress. Even though we have plans to make some internal backend servers IPv6-only, we still have IPv4 on every server because IPMonitor is our main monitoring and alerting tool for now.
Network devices
Also, think about all the network infrastructure involved. Some content providers have their servers directly connect to the net, in the subnet provided by the ISP. This is the easiest case, no network devices involved here (instead of simple switches, which probably only do layer 2). More commonly however, is that the servers are behind firewalls or load balancers. These devices will probably need to support IPv6 as well, in order to pass IPv6 packets. But hey, IPv6 is not a turn-key migration, you can do one service at a time. At NetMatch we use Cisco ASA devices, and they do work with IPv6 very well, even in the GUI (ASDM).
Some high-end devices might have routing or firewalling logic 'in hardware', which means they have dedicated chips designed to do that work. When a device like that supports IPv6, it might very well handle IPv6 traffic 'in software' instead of in the hardware chips, which results in a much lower throughput. Be sure to check with your vendor what the performance difference is between IPv4 and IPv6 before you enable IPv6 on your largest bandwidth-hogging site.
NAT
Most companies use internal rfc1918 addresses (like 10.x.x.x) for the workstations and servers in their networks, and use NAT to allow multiple workstations to access the internet, or to translate traffic between the internet and servers. With IPv6, there's no need for internal addresses and NAT anymore, and each device can get a public ip. We can finally get rid of NAT, which has been giving us network admins so much trouble over the past decade. Protocols like FTP and SIP weren't designed with NAT in mind, and the workarounds we see for that are sometimes more complex than the protocols itself. Finally we can get read of problems like NAT loopback and split DNS to work around it.
One misconception is that NAT made your network more secure. Well, in a way that's true, since NAT gave the extra added security benefit of dropping 'uninvited' packets from entering the network. Without NAT, you have exactly the same level of security, if you configure your firewall to deny the same 'uninvited' packets from entering the network.
VPN
Most companies, us included, use site-to-site VPN's to connect those internal networks together over the internet. However, with IPv6, we'll give every host a public IPv6 address. Theoretically this means we have end-to-end connectivity from every host to every host, which is very cool. Hoewever, by default that traffic over the internet is not encrypted. If I were to log in from my desktop in my office to some web application in our datacenter using plain http, anyone with control over the routers in between could read my credentials. That's why we had the VPN, so we can encrypt all this traffic. IPv6 mandates the implementation of IPSec, and you should be able to encrypt traffic on one side, and decrypt it on the other side.
At NetMatch we haven't figured out exactly how to do all this with our network hardware. Apparently with a new version of software for our Cisco ASA firewalls (8.3.1) we should be able to create site-to-site ipsec VPN's that include IPv6 subnets. However, that only works from Cisco ASA to Cisco ASA, and not to third parties. Until we've found a good way, we just force all management traffic over IPv4. (Some traffic is tunneled over a layer 2 OpenVPN connection, which works very well.)
Example configurations
Directly
See figure 1. This is a configuration most often seen with budget hosters, or companies who like to keep things simple. All servers have public IPv4 addresses, and they are connected directly to the ISP's router using a switch. The customer doesn't have any firewall or routers. Machines run host-based firewalls, like iptables for Linux, or IPSec port filters for Windows.
The advantage to this setup is that it's simple. Less components means less configuration, less things breaking, less performance issues, etc. No worries about NAT loopback, no split DNS issues, etc.
The downside is that in this situation, you don't have a single place to control what's going in and what's going out. If a machine is compromized you can't trust the host based firewall anymore. All you can do is disable the switch port (of you have access to the switch), and then investigate what's happening. If you have a dedicated firewall machine in front of it, it is way easier to enforce a security policy.
Once upon a time, long ago, I managed one of our networks that was set up like this. I got an alert that a website was down on a Windows server, and I could not connect anymore using RDP. I used the Dell Remote Access Card out-of-band access to look on the console, and I saw someone clicking around on the desktop. Scary! I then realized I had no easy way to block access to the one controlling it, or log traffic. All I could do was disable the switch port or turn off the machine remotely to prevent worse. Turns out the machine (which was installed and managed by a third party) was running an old version of Apache. Through remote http requests, he managed to replace apache with a shell or something like that running on port 80. That teached me a few lessons, and one of them was that I don't like this setup unless I really have to.
Enabling IPv6 on this setup is incredibly easy. See figure 2. After you've received your subnet information from your ISP, just configure your OS with the extra IPv6 address (see below on config examples). And perhaps restart services so they listen on IPv6 addresses as well.
In this case, the ISP gave use the single subnet 2001:1234:1234::/64. In this case, I chose the same host-address as the IPv4 address. If you want to use EUI64 addresses like 2001:1234:1234:301:6451:3fb0:7710:ee42, be my guest.
If you're using a host based firewall, make sure it also filters IPv6. On Linux, you'll want to look at ip6tables in addition to iptables. On Windows 2008 and higher, you probably use an IPSec policy as a portfilter, and it should work fine with both IPv4 and IPv6.
Now test your applications, and if you're happy, add the right records to the DNS, and move on to the next server.
Natted
See figure 3. This is a setup you'll see most of the time. Most ISP's will give you a single subnet, and you don't get extra subnets to route by default. What happens is that people put a firewall in the subnet, and use static NAT to translate hosts addresses.
In this example, our ISP gave us the IPv4 subnet 123.123.123.0/24. We chose to put our servers in their own internal subnet, 192.168.123.0/24. We put a routing firewall in between with 2 interfaces: 123.123.123.254 (outside) and 192.168.123.1 (inside).
We also make static 1-to-1 NAT mappings. For example, we map Server1's 192.168.123.2 address to the public ip 123.123.123.2. We make access rules in the firewall, with which we can define what servers are accessible on the servers, and from where.
The advantage of this solution is that you have a single point of entry and exit, and you have the access rules for the entire network in a single place. This makes it much easier to enforce security policies and notice anomalies. An added benefit is that you might save on public ip's, because perhaps not every server needs a public ip address. It's also quite easy to set up site-to-site vpn's from and to the routing firewall.
There are some downsides as well: things are more complex; the solution is more expensive; since a single firewall is a single point of failure you might need a redundant cluster, even upping the price more; you have to deal with NAT loopback issues, not every application might work flowlessly with NAT, etc.
Enabling IPv6 on this network will require at least 2 subnets: one for the subnet on the outside of the firewall, one on the inside of the firewall. See figure 4. Also, your firewall will need to support IPv6.
In this case, our ISP gave us 2 subnets: 2001:1234:1234::/64 and 2001:1234:1235::/64. The ISP will also have to make a static route, that the next hop for the network 2001:1234:1235::/64 is 2001:1234:1234::2. So basicly the first subnet is used for routing only.
Since this 'routing' subnet contains so little hosts, some ISP's might even give you subnets smaller than a /64, and for the routed subnet, they might give you something larger than a /64, like a /56 or a /48, so you can make multiple internal networks with different subnets.
Notice that we don't change anything to the IPv4 setup, since we probably still want to be able to reach every service over IPv4. However, when IPv4 addresses run tight, you might be able to remove some public IPv4 addresses, since you know specific clients are already IPv6 enabled.
In this diagram we've added the IPv6 ip's to both interfaces on the firewall, and we created access rules to allow traffic.
Now just add IPv6 to the individual servers as bescribed above.
Configuring IPv6
CentOS/RHEL
Here's an example config. Since I don't want autoconfiguration, I disabled it:
$ cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=hostname.companycolo.local
IPV6_AUTOCONF=no
$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
# VMware VMXNET3 Ethernet Controller
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
HWADDR=00:50:56:1A:2B:3C
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=10.123.10.2
GATEWAY=10.123.10.1
IPV6INIT=yes
IPV6ADDR=2001:888:1234:10a::2
IPV6_DEFAULTGW=2001:888:1234:10a::1
Windows 2008+
Just add the information in the network configuration under the IPv6 protocol. If you want to disable autoconfiguration, open an administrative command prompt, and run the command:
netsh int IPv6 show addr
It will show you a list of interfaces and addresses. Take not of the interface number you want to disable autoconfiguration for, and run the command:
netsh int IPv6 set int <interface number> routerdiscovery=disabled
Angelo Höngens
NetMatch Systems Administrator
November 1st, 2010.