Joe Maller.com

iTunes slowdowns with Google DNS

Last night we tried to rent an iTunes movie on our newish Apple TV. Instead of starting right away, the Apple TV said it would be 2+ hours before we could start watching. I’ve got a healthy 15-20Mb/s connection and a clean wire to the Apple TV, so this shouldn’t be happening.

A little bit of research turned up a surprising fix: Don’t use Google DNS.

The iTunes Store has thousands of entrances. Everyone using Google DNS is trying to get in through the same door.

Some anecdotal evidence:

This totally makes sense. iTunes’ video content is delivered by Akamai who has distributed massive datastores around the world so those large files originate from nearby servers and spend less time getting switched around the network. Akamai somehow uses our DNS routing to determine our location. If Google DNS or OpenDNS routes everyone to Akamai the same way, then those Akamai nodes and the pipes leading to them get overwhelmed.

Since most people don’t know what a DNS server is, this problem primarily affects the “tech-vanguard” and those fortunate/unfortunate enough to be inside our circles of helpfulness.

I switched to my ISP’s DNS servers and now HD rentals on Apple TV are ready to watch in 10-20 seconds.

Go figure…

(I’d forgotton, but I wrote about a similar iTunes-DNS problem in March 2009: iTunes Store DNS Connection Problems)

No related posts.


  • Mirabel

    As a member of the “tech-vanguard”, why did you even use Google DNS in the first place? Its mediocre performance isn’t exactly a secret.

    • http://www.joemaller.com Joe

      I never found Google DNS to be mediocre. Using namebench and in casual testing, Google’s DNS response times were at least 20% faster than everyone else’s.They also tend to propagate changes very quickly. The main reason I switched in the first place was because of NXDomain hijacking.

      • Michael

        Joe, it would seem you don’t quite understand the issue, because the results returned by namebench have nothing to do with it.

        The problem does not like in how fast your DNS server returns an IP address when you give it a domain name. The problem is this: content delivery networks deliver *different* IP addresses for the *same* domain name, depending on where the query comes from geographically. So for instance, the IP address delivered for Akamai content in New York is likely to be different from the one delivered in Dallas. This is extremely useful for media distribution, because it allows companies like Akamai to cache content at multiple locations around the world, and serve it to you from the nearest server.

        The problem with Google DNS and other “global” DNS servers is that it frustrates this geolocation-savvy practice.

        • Michael

          I am an idiot, Joe. Sorry. I didn’t realize I was responding to the original poster :) Facepalm.

        • JL

          Michael — EXACTLY! The response time from the DNS server is just one link in the chain to delivery of content. Smart folks keep local DNS servers (via their enterprise/campus/ISP network). Maybe OpenDNS or Google DNS is 3ms faster in terms of response time every once in awhile but if you look at how fast you get content the local ones will more than destroy a few ms of speed in DNS resolution via shorter network paths, direct interconnect to a CDN, etc.

      • Michael

        Sorry the the lousy copy editing there.

  • Jack

    I noticed this over the weekend on my Macbook and switching from OpenDNS to my ISP’s DNS also fixed this – but I have my doubts because I have Win7 desktop PC that was downloading just fine before I made the DNS change. So was it OpenDNS? Or was it Mac OS X? Or 802.11N? And before making the DNS change I noticed quiting iTunes and restarting it would also fix it.

  • UNCjigga

    I switched to Google DNS last night, mainly because it fixed an issue I’d been experiencing in Chrome on 64bit Win7 machines (lots of DNS precaching issues meant pages wouldn’t finish loading, even with DNS precaching turned off in Chrome settings.) Rather than switch individual machines, I just changed DNS servers on my router. Switching to Google fixed downloading, but then my Roku box couldn’t connect, and my Android phones couldn’t hit up the Market either. I switched back. Any suggestions on how I can use Google DNS exclusively for web browsing?

    • TimR

      Change the DNS setting on your computer, not your router. That way, your web surfing takes advantage of Google and your other devices use the ISP’s DNS.

  • Keith Brady

    The “somehow uses our DNS routing to determine our location” is actually quite straightforward. All major internet services (Akamai and Google included) look up the IP address of the querying DNS server (usually your ISP’s server) in a location database (something like doing ‘whois w.x.y.z’ to look up RIPE et al) to figure out your approximate location and serve the IP address of a nearby point of presence (e.g. Datacenter or peering point).

    The tricky bit there is that they use the IP address of the DNS server, not the original requester. Usually that’s nearly the same thing since it’s your ISP (or your own one) and you are routing out to the internet in much the same way. The problem comes when you’re using OpenDNS or GoogleDNS. While they both have many servers that are globally distributed (using anycast to do that despite using a single IP address) these servers aren’t as close to you as your ISP’s and will have a funneling effect.

    There’s a modification to the standard in progress for the requesting DNS server to pass along the IP address of the querying client and that will make a lot of this go away. AFAIK, GoogleDNS does this and Google servers can take advantage of it (though it’s a proposed standard and so open).

  • http://pimpmyi.com/ iPimp

    Is there a conspiracy of Google vs Apple in this story?

    • Nate

      No.

      • etaN

        Yes.

  • http://builtwith.com Gary

    Funny I tweeted about this just a couple of days ago -

    http://twitter.com/garazy/status/16369506293129216

    It’s a well known issue in Australia by the looks of it -

    http://www.lifehacker.com.au/2010/05/why-google-public-dns-sucks-for-aussies/

    Gary

  • Steve Wart

    I don’t buy it.

    We had the proverbial 9 hour download and we aren’t using any 3rd party DNS.
    Apple’s servers and the bandwidth on their side are being overloaded pure and simple.

    Some titles work, some don’t. Let Apple make a statement. So far the silence is deafening.

    • Andre Richards

      Why would Apple respond? There’s overwhelming evidence that Google’s DNS is slow and virtually none that iTunes’ infrastructure is the cause. I switched to Google’s DNS out of curiosity a few months ago but had to switch back to Comcast DNS after getting frustrated with random slowdowns and failures.

      So let me ask you this: what do you think of Google’s lack of public statements about this? Is their silence on the proven slowness of their DNS also deafening? Or does that only apply when it’s Big Evil Apple?

      • yin

        Google’s DNS responds quickly, Apple’s infrastructure uses DNS to infer a users location, assuming (incorrectly) that all users stick to their DHCP assigned DNS servers given by their ISPs.

        • Joe Z

          Personally, I think it’s just broken design. They should look at the IP address of the request and redirect the client if it’s coming into an inappropriate or overloaded server. Any performance bottleneck should only affect the initial request that results in the redirect.

          For example, if they’re using HTTP, they can reply with “301 Moved Permanently” or “307 Temporary Redirect”.

          DNS is the wrong way to do this.

          • Smith6612

            I would have to agree with this. DNS shouldn’t determine where you receive your cached content from. IP addresses are easy to locate as they are usually assigned to certain geographic pools. Take my DSL for example. Sure, I might get on a new block of IPs that many GeoIP databases might not know about, and sure, it might screw up such systems for a little while however this IP block will generally stick around for years until the provider decides to either remove it or move it to another area. It’s also very rare for my IP to change anyways.

            I use OpenDNS on my end. I seem to have no issues with iTunes downloads and such as both of the servers I hit are in New York City. My ISP has one DNS server in New York City and another in New Jersey, which while getting the same millisecond ping time as the OpenDNS servers, tend to really slow down towards the evening hours.

          • Smith6612

            Sorry about the double comment, but I might also point out that both my ISP’s DNS servers and OpenDNS take me to the same caching servers. All of my data has to go through New York City anyways, so the difference between the two for me is simply the speed at night which OpenDNS shines in, and the flexibility between the two.

    • http://andrewembler.com Andrew Embler

      My internet connection is slow! APPLE ! THIS WILL NOT STAND!1!

    • Dan

      No it’s not the DNS servers location.

      The real problem is what address it offers for the host you are trying to connect to. In the case of Apple they have many servers which are round robin’ed at the different sites they have around the world. In some cases the DNS software tells your system to connect to a more distant address than a local one because it thinks the address it is giving you is closer to you.

      I suspect Google is having problems keeping their forwarding tables up to date and/or they can’t resolve where your system is.

      This is a Google DNS server issue not Apples fault.

    • JL

      Steve – This is not a “Steve Jobs” or an “Apple” thing. This is how CDNs achieve content localization — for all sites — and have done so for years. This is one of the downsides of using a different and non-local DNS. There’s nothing at all new here — it’s just that someone happened to notice it for the first time using an Apple TV. This also has nothing to do with decisions Apple makes — this is how the very Internet and all CDNs work.

    • Bruce

      Maybe you just reside in the same area as the Google or OpenDNS servers. No reason the location of your ISP’s DNS can’t be directing you to the same overloaded Akamai servers that Google does.

  • gregorylent

    ha, i thought it was apple being competitive with google.

  • Joe Doakes

    Here is a great free tool recently released by Steve Gibson for testing DNS servers:
    http://www.grc.com/dns/benchmark.htm

    • Michael

      This is a nice tool, as is namebench. But neither of these tools address the issue at hand, which is the serving of *geography-specific* IP addresses. In fact, I argue that using tools like this is part of the problem, not the solution. Who cares if your DNS requests take 100ms instead of 25ms, when downloading your content takes 10 minutes versus 60 minutes?

      • Joe Z

        The mistake is using DNS as sole the load balancer to begin with. In the end, you transfer the file with a connection made from your real IP. You should get directed / redirected based on *that* IP, not the IP that made the DNS request.

        Relying on DNS responses is an unreliable hack, pure and simple.

        As for why fast DNS matters… I counted around a dozen hostnames associated with this web page alone, and I have ad blocking software enabled. If you have a few DNS queries that just hang out there for several seconds, on a regular basis, that makes browsing the web painful. With that many distinct hosts in one page, the odds of that happening go up. Don’t underestimate the impact of DNS on the perceived snappiness of your net connection.

  • http://bradyjfrey.com Brady J. Frey

    I had this same problem here in San Francisco. After a week of fiddling with just about everything on my network (for a long time I just didn’t admit I’m on iTunes/AppleTV THAT much, and thought it was a network wide issue), including working with Comcast directly to decipher if they were throttling me somehow, switching back to the default DNS did the trick. Tracing my route via Google DNS had a significant slowdown to Apple’s Servers… but I found that on OpenDNS too. Switching to trusty 4.2.2.2 and/or Comcast’s DNS servers worked great for me.

    That aside, I still notice Netflix is loads faster then iTunes via AppleTV, and wouldn’t doubt iTunes is getting slammed. It stopped being fast when Apps became king, at least for me.

  • http://bradyjfrey.com Brady J. Frey

    …side note, broke my heart a little. Google DNS seemed to speed up just about everything but iTunes on my network.

    • JL

      Brady – You actually mean that Google DNS seemed to speed things up for everything but (strike iTunes) CDN content. Which is most content these days of course.

  • http://darrinrich.com Darrin

    This was the same issue I had with OpenDNS. It was really frustrating.

  • Denzyl

    I have slow download problems as well, but currently run the default DNS, not Google or OpenDNS. It took 3 hours to rent a movie last night. It also takes about 2-3 minutes to Airplay a 20 second movie from my iPhone or iPad. The longer the movie the longer it takes. From my computer the time is shorter – about half the time. The Apple store genius recommended that I restore it to factory default then redownload the software and try it again and that it could be a defective unit.

  • Paul W

    A lot of the commenters here are missing the point. Google’s DNS servers aren’t what’s slow. Rather, Apple uses Akamai to spread out the load on Apple’s servers. How this works is that Apple gives everyone the same DNS name to download from, and Akamai’s DNS servers hand out different IP addresses to each endpoint DNS server. So Google DNS gets a single IP to hand out for the download server. So does Comcast’s local DNS servers for each local network. So does OpenDNS’s server. So does every other ISP’s endpoint DNS servers. The IP address you’re supposed to get from each endpoint points to a server that is geographically near where you should be. But Google’s DNS server hands out the same IP to everyone who uses it, so people from all over the world are getting the same IP, instead of unique IPs for wherever they actually live.

    It’s a practical issue with Google’s service, sure, but only because Akamai is relying on your DNS service saying something about where you are. Google and Akamai are both using the system in ways that violate implicit Internet design expectations, but for their own needs their “hacks” so to speak improve performance. But when they come into conflict with each other that’s when you run into a problem like this.

    • Dan

      Paul,

      Each DNS server should have the same table or direct your request to a better authority that has the needed address (table). In this case it looks like Google has out of date tables as they are offering out more distant addresses so the connection is slow to Apples’ server/s which is hosting what you want.

      Think how you would call the operator when you needed to get someones number (1960′s time point). The local operator would connect you to the operator nearest to the person you where trying to call. As they would be better able to help you.

      Here Google is local to you yet they think they have a good address for you. Only problem here is it is in Japan not New York where you are located.

  • Paul W

    So to follow up, the people who aren’t using Google DNS or OpenDNS and are having problems are probably getting pointed to the same endpoints that Google DNS and OpenDNS are pointing people to, so they are hitting the same overloaded servers. Apple and Akamai can fix this problem more easily than Google can. They’re the ones you need to be contacting.

    • Mr Right

      To correct you, the DNS servers themselves are getting overloaded. That is the issue here.

      • Arnold

        No your correction is wrong – it’s not the dns servers that get overloaded, it is the akamai mirror servers. Without the dns trick doing load balancing and also not giving people mirrors that are geographically close, too many users hit the same mirror. The dns lookup is done once at the start of the download, not continuously.

      • Dan

        No the DNS servers are not being overloaded! The issue is the tables they have are not up to date.

    • Westacular

      Is it really that certain Akamai nodes are being overloaded, or rather that people from certain locations or certain ISPs suffer significantly when using the “wrong” Akamai node? There must be some other factors here; I think the outcry would be bigger if EVERYONE using Google DNS or OpenDNS was experiencing slowdowns of this magnitude.

      Specifically: It sounds like a lot of people who seem to be experiencing this problem are Comcast customers, or from Australia. Australia is self-explanatory; for Comcast, I wouldn’t be surprised if the company was heavily throttling connections to the “wrong” (ie, external) Akamai peers, as an added measure to encourage users stick to settings that will ensure they resolve to the local peers — saving Comcast money.

      Note, also, that Google DNS is not a single server — they have servers all over the place, not unlike Akamai. Instead of using name resolution tricks to point to a local node (which of course isn’t even an option for a DNS server), anycast routing is used to point packets for 8.8.8.8 to the ‘nearest’ Google datacenter. (See Keith Brady’s post above for an explanation of why this kinda breaks Akamai nonetheless.)

  • Marc Farnum Rendino

    Folks -

    It’s definitely NOT name server use that is slowing downloads/streaming in progress – the download speed is determined AFTER the name server has answered and BEFORE the download has begun. Once the download has begun, the name server has no effect – other than the side effect of algorithms that use the name server’s derived location to determine which download server to direct you to.

    Due to the nature of TCP, transferring data from/to a machine that’s farther away generally results in lower throughput (downloads take longer).

    This is worsened when network conditions change during the download – as they do, I would bet, more and more frequently now; it’s an extremely dynamic system.

    What happens is this: Most TCP stacks now attempt to adjust to conditions on-the-fly, those conditions change which results in lots of failure (resulting in retransmits which soak up bandwidth) while TCP adjusts again. And again and again. Often it ends up adjusting to recent-past conditions *just as conditions change again* – it can easily end up a worst-case scenario in which most of the available bandwidth is divided between waiting and retransmits.

    This is especially due to latency and the TCP window size (Google those terms; enormous amounts of info).

    - Marc

    • Andre Richards

      Technically you’re correct, but you’re missing the forest for the trees. The problem originates with Google’s DNS. It’s not Google’s problem per se, but rather a problem with the concept of a centralized DNS server intended to be used by so many geographically remote users. Scroll up to Paul W’s post for a better explanation.

    • Dan

      You are correct Mark!

      But … the name server (DNS) could be offering addresses to more distant hosts than one that is closer if it has a table listing of hosts that is either out of date or not able to ID the location of the requesting system so it can guide the connection to the best choice.

      Your other point is a know problem called ‘Silly Window’ but I’m not sure that is the issue here.

      When you have steaming data like watching video or music (large data blobs and simple ACK’s going the other way) it’s possible the connection pipe is not able to sustain the flow. Doing a TraceRoute might show the pathway is over a congested segment or a distant server instead of a closer one.

  • perks

    I discovered the opposite this weekend with Netflix. Netflix simply would not download more than 2 streams from my home IP address, UNLESS I SWITCHED TO Google DNS from Comcast.

    Strange.

  • Guest

    I think it’s an Apple/Akamai issue since Netflix, which uses Level 3 and Limelight, streams are instant on the Apple TV.

    I’ve spent hours on the Apple forums and have tried everything from DNS switches, new routers, ethernet vs. wifi, updating all firmware and software, etc. and nothing seems to work.

  • Podesta

    Actually, Netflix uses Silverlight, which is not compatible with iOS at all. That can lead to more hiccoughs on iOS devices than on Macs.

    But, what strikes me is that even in this realm there are so many people with crazy conspiracy theories.

  • Mike

    My iTunes movie downloads are really slow…but I have no idea what the hell this article is talking about. Problem/solution in english please?

    • Dan

      Mike,

      The thread started about problems getting good connections to Apples iTunes servers. Some where along the path Akamai DNS (a hosting service – which Apple does use) was thought to have the connections better aligned than Googles DNS.

      The corrections offered was to alter your DNS connection to OpenDNS.

      The real issue is either the DNS table entries are messed up in Googles DNS servers or the bandwidth is over saturated along the pathway. There are a few connection points within the internet that could be saturated.

      While some people believe the issue is within Akamai/Apple’s servers being over loaded. I can tell you they are not over loaded within the server farm or the local connections to the server farms.

      • Geoff

        Dan, I’m afraid you seriously don’t understand the problem. I suggest you re-read Paul W’s comments until you understand them.

        • Dan

          Geoff,

          No, I don’t think I got it messed up.

          Here’s Paul’s point:
          “But Google’s DNS server hands out the same IP to everyone who uses it, so people from all over the world are getting the same IP, instead of unique IPs for wherever they actually live.”

          He is implying Googles DNS service is offering out only one IP address to a single Akamai site, so no matter where you are connecting from. While this may be happening it is not the way DNS works when it is correctly setup.

          What should be happening here is Googles DNS (or any ones else’s) should redirect the host lookup to Apple’s (Akamai’s) DNS domain servers. In this transaction the users location should be passed to the DNS server to get them to the most local server. As this DNS is the best authority of all of the servers and their locations that host the file/s in question.

          In this case I suspect Google’s servers are not correctly setup or the tables are out of date (in some or all of their DNS servers) and/or they are not forwarding (maybe they don’t know as well) the users location to the resolving DNS (Apples/Akamai). If someone is using a tunnel (IPv6 or VPN) it’s possible the end point’s address is used instead of the tunnels end point in the DNS query it can mess things up as well.

          The other possibility here the user is on a part of the internet that must connect via a different provider which has a choke point in their interconnection (ATT & Sprint have this problem for quite sometime). to get to the server the DNS resolved as the best choice. In this case the more distant server might be better.

          In either case the issue is within Googles DNS and/or the connection pathway to the resolved server.

  • Relayman5C

    Mike, If you don’t understand this, then you haven’t done the thing that’s causing the problem. In other words, this article doesn’t apply to your situation and you can ignore it. Your slow downloads are caused by something else.

  • Charlie Root

    Nobody here knows about 4.2.2.{1-6} ?? it’s fast/free/public and been around for quite sometime!

  • Roger Andreas

    It’s amazing to me that the connection to iTunes works great on your default DNS but lousy via Google’s DNS and yet there are people here arguing that it’s somehow Apple’s fault.

    Even if you don’t know any of the technical details of DNS, surely it’s clear that such a statement doesn’t even pass the basic logic test. I realize there’s a lot of hatred pointed at Apple nowadays (for whatever reason) but if you can’t leave your biases at the door, you contribute nothing to the discussion.

    • http://rath.ca/ Christopher Rath

      Roger, the issue is that Apple is incorrectly assuming that the user’s computer is in the same physical locale as the DNS server. See Keith Brady’s comments, above.

      Here’s another piece of evidence that Apple could fix this if they wanted: try buying something from an iTunes store outside of your country… the iTunes store will refuse to serve you. So, when Apple wants to correctly determine your location they are able to do so; which is why many readers here believe that Apple should fix this.

    • Kenneth

      Roger here is right. For whatever reason people are trying to blame Apple and their fantastic tech (won’t get into a platform war right now), when in actuality they are saying “Here’s our product, just tell us where to ship it to!!” Google’s DNS isn’t telling Apple WHERE you are, so the Akamai servers said “OK, Google says you’re nowhere, we’ll just route your movie/TV episode half way across the world so that we can try to find you.”

      • Dan

        Kenneth,

        Why do you think Googles DNS is not telling Apples DNS the users location to get the best host?

        Per the IETF standard for name servers unless it has a local resolve in its’ own table it needs to refer the resolve to the name server of the domain (i.e. Apple.com) it is trying to get a resolve for.

  • calzone

    This argument is pretty idiotic really.

    Using DNS to distribute load may have seemed like a reasonable idea at the time, but so did maxing out years at ’99.

    The problem will only get worse. And with ISPs shaping traffic, you can bet more and more people will be leaving their DNS servers for neutral ones.

    But most important, basing balancing on location is deeply flawed. I’m on Orange County CA. You can bet that the traffic demands here are much higher than some county in North Dakota. Reasonable load balancing would respond to need, not location. If one server is overloaded, send a new request to a different server.

    The idea that just because I’m in OC I should prefer content from a local server is really nice, but ultimately flawed if I’m competing with 200 thousand other concurrent requests in the same region. I’ll get better performance from downloading something from some district in Japan where everyone else is sleeping and I’m only one of 50 other concurrent requesters.

    • Dan

      Calzone,

      Sorry to tell you this Server farms are located near or at dense use areas to limit cross network flows.

      DNS tables are setup to aim you to the most local server based on your IP address which is set by your location (region).

      The connection bandwidth at the server farm is not likely to become saturated (I’ve only seen it once in my 30+ years designing & building networks)

      As to Apple limiting access based on your location: Yes, they do that as they have contracts that limit what can be sold to whom and at what cost. BTW – we pay the least for music & vids in the US!!

  • calzone

    After some thought, I realized that the issue is that there are two completely different problems here, but both are being conflated:

    (1) Load balancing and (2) leveraging proximity effects for an edge in performance.

    It is WRONG to use proximity for load balancing, plain and simple. Aside from population density discrepancies, there’s cultural and time-zone issues (i.e. Japan ^^)… and more significantly, all you need is a big game, an earthquake or snowstorm, or any other kind of regional situation and your entire region’s demand goes way off normal. Throw in mobile devices into the mix and location becomes less meaningful anyway as you move from zone to zone.

    Throw in some future p2p DNS scheme and you will have no inkling of location. Instead you will know something about relationships, groups, and hops.

    Resolving load balancing should be based on distributing demand equally across resources. Location should have nothing to do with that determination except when proximity effects actually outweigh demand effects, or when there is insufficient demand to make a difference to performance.

    • imMute

      Except you’re forgetting the cost of the trans-pacific link (in both terms of latency AND transit cost).

      While you have a good idea, in practice it’s not very reasonable – what’s more cost effective is just putting more servers in the Orange County Area.

    • Dan

      One other point Calzone

      Most server farms have multiple servers hosting the same data and are round-robin’ed so the load is spread out.

      In addition, the DNS tables have timeout values so if a given server and/or site goes down the load is shifted to a different server. Load balancers are also used within bigger server farms as well.

      Remember the hosting company what your business so they go to great lengths to make sure you get connected and to the best server based on location to give you the fastest connection.

      Even with all of their efforts within their control bad connections and poorly designed/over loaded interconnections can spoil it.

  • Westacular

    The post is mistaken in assuming it’s a load issue causing the slowdown. Read up on CDNs, they use additional load balancing techniques, beyond just DNS-based routing to close proximity nodes.

    http://en.wikipedia.org/wiki/Content_delivery_network

    The issue here is not that Google DNS is aggregating too many users on a single node. I can (and just did) use iTunes with Google DNS without problem, maxing out my 7 megabit DSL connection.

    The issue is that Google DNS does not give users the IP for their local nodes within the CDN… and certain ISPs — namely Comcast — are heavily throttling connections to non-local CDN nodes.

    Why? Because traffic to those non-local nodes has to go outside Comcast’s network, and which costs them more money… money they wouldn’t have to pay if those users just kept their normal DNS settings. Slowing down the traffic is a way of limiting the damage and encouraging users do to the “right” thing.

    The problem with this strategy is that the cause of the slowness is quite opaque to the vast majority of users, who — as the comments here show — are likely to blame the content provider, or the CDN, or (eventually) the DNS service, for providing a somehow faulty service, when all of them are actually doing the right thing to the best of their ability; if there was a way for the third-party DNS service to know and point to the appropriate CDN node, they would, but we don’t yet have a standard in place to handle this limitation.

    • imMute

      >The issue is that Google DNS does not give users the IP for their local nodes within the CDN

      You’re on the right track, but Google DNS is merely a recursive DNS service. When you ask GoogleDNS for some akamai hostname, Google turns around and asks Akamai (the authoritative source for that domain) which IP to use.
      Since Akamai sees the Google server requesting, it returns an IP closest to *Google*, which then gets passed back to you (and probably stored in a distributed cache to be passed to other people).

      In all honesty, it’s not Google’s fault, nor Apple, nor Akamai, nor OpenDNS, nor anyone’s really. It’s a combination of Google/OpenDNS/whatever ‘s centralization of DNS querying with Akamai’s geolocation of DNS requests (to determine the response) that makes a “perfect storm” to mess stuff up.

      tl;dr> run your own recursive/caching nameserver. it’s not hard and you’ll completely avoid problems like this ^__^

      • Westacular

        Oh, yes, I didn’t mean to imply otherwise; I agree totally. Google DNS and Akamai both

        a) rely on techniques that somewhat defy the original conceptual model of how IP and DNS work

        b) work within specs to solve valid problems in Internet architecture

        and

        c) are doing things “right” to the best of their ability

        Neither deserves blame, and most Internet users won’t even see a problem here. Greater transparency from the likes of Comcast over traffic shaping would alleviate a lot of the confusion.

        PS: by running your own nameserver, you can set it to selectively forward requests for certain domains (eg apple.com and itunes.com in this case) to your ISP’s DNS while still using Google DNS for general purposes.

        • Dan

          FCC’s ‘Net Neutrality’ Internet regulations should put a stop to any traffic shaping on Comcast’s part once it becomes enforced.

          Sadly, it won’t stop congested interconnects or overloaded networks.

      • Joe

        You are wrong, this post is completely wrong. Read up on multi-cast DNS please.

  • Dave

    The timing of discovering this post + comments is serendipitous indeed. If you’ll permit me a quick explanation:

    4-5 months ago I noticed the speed at which Software Update downloaded was excruciatingly slow, like 1k/sec. slow. For me, it used to be SU would d/l the latest OS X update in a few minutes. Now I’m lucky if it takes under eight hours.

    Then I noticed downloading iPhone app updates via iTunes was as painfully slow. Like others, I tinkered with router settings, did numerous Google searches for fixes, and even posted on discussions.apple.com (and received a whopping zero replies). Little Snitch helped me narrow down the problem to “something” having to do with Akamai, but aside from that I was stumped.

    I signed up with OpenDNS earlier this evening and plugged their server numbers into my router and Network DNS settings, restarted my MBP, and resumed downloading some iPhone apps via iTunes. Unfortunately, the poor speeds remain.

    • Dan

      It’s best to use your local ISP’s DNS for your lookups.

      As noted by others here using Google or OpenDNS can cause mis-direction of the connection to a server not local to you.

      While many DNS servers (BIND 9.x) now offer better methods of node reverse lookup not all of them are using the current RFC extension or use table lookups.

      If you don’t see any improvement when using your local DNS host you more likely have network flow issues of the connection (local to you or somewhere along the pathway to the responding server)

  • Joshua

    For you unbelievers, here is the Proof to the Theorem. Got sick of Bellsouth/ATT DNS outages a few months ago and have been using 8.8.8.8 (Google Public DNS). Last night I downloaded a Bond Flick via AppleTV (which uses the DNS of my router, which is set to Google). Took 5 hours to download over 6.0/512; couldn’t watch it last night. Today, I having a few problems with slowness and decided to do some searching for answers. Saw this blog and made ONE CHANGE – from Google DNS back to automatic DNS from the ISP. Purchased another Bond movie and it was ready to view in less than 5 minutes.

  • http://cardonationscharity.us/ Car Donations

    seems westacular really went into the meat and potatoes of why bandwidth in the US is just a commodity to be pawned!

    Couldn’t agree with you more there bud!

  • Ken

    I have an openDNS account and what you say is true – my Apple TV streaming and iCloud (iTunes Match) was really slow. I switched my router back to my ISP DNS (Virgin Media) and the data download is working great but as soon as I switch the DNS back to the OpenDNS servers it slows right down to the extent that I cannot wait for apple TV to stream a trailer – OpenDNS will be switched off till someone resolves this issue. Not sure though who is at fault – Apple or OpenDNS???     

  • http://www.facebook.com/johnthyde John Hyde

    Or maybe the buffer on the Apple TV is full. Maybe if you do not let a movie or tv run to the very end the show will remain in the buffer and make new streams difficult to load. Tested the idea by running the last show I watched to the end and suddenly everything worked again. Can anyone else replicate that?

    • MileHighJoe

      Halelujah!!!!!!!!! I’ve been trying every solution I could find, for the past two hours, to no avail. Yours worked! Now I’m back to having my shows start playing in an instant instead of having an expected wait of 3+ hours.

      I’m not very tech savvy, but your solution needs to be broadcast everywhere!!!!!!!

  • someBoredNetworkEngineer

    As I Network Engineer Veteran I had some slow itunes downloads and decided to investigate these findings. I was pleasantly surprised to confirm the author of this posting findings. I am on Comcast and have confirmed from testing, DNS packet sniffing analysis, and trace-routes that changing your DNS server settings does impact the speed of itunes downloads. I used the same test file, flushed DNS cache, and re-started itunes in between testing different DNS settings. I had the best result from using my own DNS server using free open source CentOS/Bind 9 that queries the 13 public root DNS servers directly. I know that is way too technical for 99% of the public but I recommend taking 5 minutes to test different DNS servers using the method I did minus the packet sniffing/traceroute analysis. When you get the best download speed stick with those DNS settings. Anyone can do that. Try your ISP’s DNS or here are some other public DNS you can try some with additional capability:

    Redirecting Public DNS:
    Open DNS 208.67.222.222/208.67.202.202, Comodo 8.26.56.26/8.20.247.20 DNS Advantage 156.154.70.1/156.154.71.1, OpenNic 216.87.84.211/23.90.4.6,

    Non-directing Public DNS:
    Verio 129.250.35.250/129.250.25.251 Level 3′s 4.2.2.1 through 4.2.2.6, and the slowest for me Google 8.8.8.8/8.8.4.4