"Connection Reset" errors, MTU, DHCP, and Time Warner Cable

So long, AT&T DSL

Not too long ago, I made the move from AT&T DSL to Time Warner Cable for my family's home Internet connection. AT&T's pricing was no longer competitive, and their terms of service were nothing to be proud of.

Hopefully most readers have heard about the recent AT&T policies regarding 150 GB data caps for DSL connections. While most people have recently been complaining about similar data caps for mobile data plans - these caps are something I largely agree with, but only for wireless. There is only so much wireless spectrum available to share for everyone within a given area - and the laws of physics don't make these limits the easiest to overcome, at least without having towers on every street corner. (This is what we have Wi-Fi for.) While it is great that the wireless industry is marketing their mobile video capabilities, etc. - they need to ensure that they are offering the actual network services to match, instead of balancing on the edge of false advertising. However, for wired connections, there should be few excuses - as a wire can always be upgraded, or another wire (or fiber!) can always be added. Especially as my family is watching more Netflix and other video content online - combined with regular remote sessions to work and other computer-related activities, hitting 150 GB would not be too difficult. Personally, I also don't expect to be able to watch Netflix videos on mobile - I don't even have a mobile data plan. There is no place for data caps on wired Internet connections.

The other concern that I had was a provision that allowed AT&T to forcefully upgrade us from our DSL account to a higher-priced U-verse account at their discretion. I agree that U-verse is cool, but I'd really like to see it stand up to competition by also having something like Version FiOS available to the same customers. Verizon FiOS is not currently available in my area. I had also expressed some related thoughts in a previous post, when Appleton was considering a bid for Google Fiber. In that post, I had also stated some concerns with Time Warner Cable (TWC) - but they seem to have cleaned-up their act a bit since then - including with a new, impressive local retail presence.

"Connection Reset" errors, MTU, and DHCP with Time Warner Cable

So we switched to TWC's Road Runner service for our Internet service. Of course, it would be too easy if this was without issue. Without any other changes in my computer or network configurations, I noticed "The Connection was reset" errors the very first night with the service. Unfortunately, these issues were very difficult to troubleshoot, as the errors were quite sporadic, and I wasn't able to reproduce on-demand. The issues also were more prevalent on some devices than others - even though all devices worked just fine when connected to other networks. Interestingly, Google sites and services also seemed to be affected more than others. I'm guessing this had to do with most of Google's services being accessed over https:// - which increased the packet sizes and likely led to some of the issues. Having the issues happen the most often with encrypted connections also made troubleshooting with Wireshark, etc., quite difficult.

I had called TWC on this, but got the typical run-around. (Obligatory technical support comic: http://xkcd.com/806/.) They don't see any issues with their service - simply saying that it had to be an issue with my computer or router. They were somewhat correct - but only because my router was following TWC's direction.

eth1      Link encap:Ethernet  HWaddr **:**:**:**:**:**
          inet addr:75.87.***.***  Bcast:255.255.255.255  Mask:255.255.240.0
          UP BROADCAST RUNNING MULTICAST  MTU:576  Metric:1
          RX packets:807 errors:0 dropped:0 overruns:0 frame:0
          TX packets:482 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:85701 (85.7 KB)  TX bytes:77492 (77.4 KB)

See the issue? Frustratingly, it had taken me at least another week to spot it. Per TWC's own Road Runner help pages, the Ethernet standard for the Maximum Transmission Unit (MTU) is Ethernet standard of 1,500 bytes. So why was my router being configured for an MTU of only 576 bytes? Here was the latest from my /var/lib/dhcp3/dhclient.eth1.leases file after reproducing the issue:

lease {
  interface "eth1";
  fixed-address 75.87.***.***;
  option subnet-mask 255.255.240.0;
  option routers 75.87.192.1;
  option dhcp-lease-time 43200;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.65.64.1;
  option interface-mtu 576;
  option broadcast-address 255.255.255.255;
  option host-name "********";
  renew 0 2012/04/01 05:31:32;
  rebind 0 2012/04/01 10:41:37;
  expire 0 2012/04/01 12:11:37;
}

As I mentioned, my router was only doing "as it was told". I'm not exactly sure what is acting as the DHCP server or where it is located, but as the 10.65.64.1 address is not resolvable nor does it appear active, my guess is that this is being served from the cable modem itself, and a function of firmware loaded and controlled by TWC's Road Runner's servers. I'm also guessing that this option is not requested by or respected by most other devices (including SOHO routers and Microsoft Windows, etc.) - otherwise I'm sure TWC would have recognized this and corrected it by now. Fortunately, dhclient under Linux provides an easy work-around - with "work-around" probably being too drastic of a name, as it is really just a simple configuration change. Here is a portion of the default /etc/dhcp/dhclient.conf on my distribution, with the critical detail highlighted:

request subnet-mask, broadcast-address, time-offset, routers,
  domain-name, domain-name-servers, domain-search, host-name,
  netbios-name-servers, netbios-scope, interface-mtu,
  rfc3442-classless-static-routes, ntp-servers,
  dhcp6.domain-search, dhcp6.fqdn,
  dhcp6.name-servers, dhcp6.sntp-servers;

I simply commented-out this field, and my Internet connection has been stable since.

I called back TWC to report my findings, in an effort to hopefully help others having the same issue. Realizing that they refused to recognize that anything was wrong, I also wrote-up a summary of the issue and hand-delivered it to the local TWC retail office - including an invite to contact me with any needed requests for further details - hoping that it would be directed to someone who would be empowered to fix this. Needless to say, 4 months later, this issue still has not been addressed or resolved by TWC.

Thoughts on Google Fiber for Appleton / ISPs

Tonight I attended a public hearing at Appleton City Hall (PDF) regarding the city's consideration to submit a response to Google's request for information on the Google Fiber for Communities experiment. (Don't miss Google's project overview and other linked pages.) Also, please join the Google Fiber for Appleton Facebook group.

I was pleasantly surprised to see this public hearing bring the attention of Green Bay TV stations WFRV / CBS channel 5 and WGBA / NBC channel 26. The public hearing started late due to another meeting in the same room. There were probably a few more than 20 people in attendance, and the response was overwhelmingly positive. This echoed the current status of the City of Appleton's Survey, which was mentioned to also be overwhelmingly in favor of submitting a proposal. In general, the responses given at the hearing were mostly focused on the points that Appleton should proceed with submitting a favorable proposal in order to remain competitive as a community, to bring additional competition and choices for Internet service, and many other convincing reasons.

There were only 3 responses against: 2 from AT&T representatives in attendance for somewhat obvious reasons in concern for their business, and 1 gentleman concerned with the physical cost necessary to connect to connect gigabit networking to his Apple computer. (He apparently assumed that the fiber would need to be connected directly to his computer. Most computers sold in the past few years already have gigabit Ethernet cards, or they are readily available for less than $50. Additionally, there are many potential uses beyond a computer, such as video and other multimedia.) As also mentioned by another resident at the meeting, even being able to connect at only 100 Mbps (vs. the 1 Gbps / 1,000 Mbps being advertised) would still be about 10x faster than most consumer broadband connections available today.

My Own Thoughts

In general, additional competition for residential Internet service can only be a good thing - whether that competition is from Google or another provider. At my current residence in an apartment just outside the official city limits (in Grand Chute, near the Fox River Mall), my choices for broadband Internet are currently limited to Time Warner Cable and DSL. Wireless / "3G" was tried, and is not current viable for primary / serious use as previously detailed. Interestingly, the reps from AT&T at the hearing used their wireless network as one of their primary arguments against submitting a proposal to Google. After looking into Time Warner Cable, they presented themselves as one of the shadiest operations in town, at least based upon my experience at their local office. I'm currently using AT&T's DSL. While I would like to sign-up for AT&T U-Verse, I'm told that it is not available to my particular building, despite several of my neighbors in very close proximity having the service.

Unlike many of the comments and hype, I would hope that Google's offering is not all about the speed - even though, unfortunately, this is all many residential consumers are aware of or take into consideration. A few questions to consider of any ISP:

• In addition to the speed, what is the latency (or lag)?
• What is the support for IPv6?
• Are industry standards properly followed, such as RFC 2308 - Negative Caching of DNS Queries? Or is the ISP involved in DNS hijacking?
• Is service advertised as "unlimited" truly unlimited, or are there limits involved that are only shown in fine print, if at all?
• What is the ISP's stance on network neutrality?
• What support / allowance is there for operating as a server - either a web server, or something more "residential" such as allowing remote desktop connections, allowing for peer-to-peer file transfers, or playing games that require being able to accept incoming network connections?
• What are the results from the ICSI Netalyzer hosted by UC Berkeley, which tests for most of the above as well other issues?

Google, in particular, has a positive record for properly supporting the above requirements and avoiding the listed issues:

• In order to help solve the issues that many ISPs cause with their crippled DNS servers, Google started offering their own public DNS, promising to improve performance, security, and to keep strict adherence with standards.
• Google is embracing and supporting IPv6:
  • Looking towards IPv6 (Lorenzo Colitti and Erik Kline, 2008-05-13, googleblog.blogspot.com)
  • Google over IPv6 (google.com)
  • Google: IPv6 is easy, not expensive (Carolyn Duffy Marsan, 2009-05-25, networkworld.com)
• Google is an outspoken proponent (googlepublicpolicy.blogspot.com) of network neutrality.
• Google has demonstrated the proper way to provide transparency around issues, such as the reports around their recent app engine outage: Google App Engine downtime postmortem, nearly a perfect model for others (Lenny Rachitsky, 2010-03-05, transparentuptime.com) and When the Power Goes Out At Google (2010-03-08, hardware.slashdot.org).

I can't imagine that Google wouldn't uphold the same principals in providing their own Internet service.

(On a humorous note, I can't help but recall Google's previous ISP / fiber offering: Google TiSP.)

Even if Appleton submits a proposal for and is accepted as a location for Google Fiber, there is no guarantee that I would be in the service area - especially being in a neighboring town. However, I have no doubt that I would still benefit from the increased competition. Additionally, once we're back in the market to buy a a house in the area, the availability of Google Fiber would be a serious consideration. (Someone please buy our house for sale in Wausau! - which happens to be wired with Cat-6 for Gigabit networking.)

Alltel / Verizon Wireless Internet: Disconnections, DNS, and other issues

About 9 months ago, I made a few posts around my Ubuntu Linux router upgrade project. Since then, Ubuntu has more than met the need and the goal. While my original notes were for 8.04 ("Hardy Heron"), they have since been successfully repeated on 8.10 ("Intrepid Ibex") and 9.04 ("Jaunty Jackalope").

Unfortunately, I cannot say I've had the same success with Alltel's wireless Internet service. (Alltel has since been acquired by Verizon Wireless, though there hasn't yet been any noticeable changes as a customer.) To be fair, I've been pretty impressed with Alltel's voice service. Their data services also seem to work well for temporary / mobile usage, especially for Blackberries and other smart phones. However, I often fail to comprehend how this service can be deservingly called "Internet" given some of the issues.

Disconnections

The most frustrating issue is the frequency and apparent length of disconnections. This has been an ongoing issue since I first started using the service back in September. I've learned that there is a hard 12-hour timeout, after which any connection will be terminated regardless of usage. While this is certainly an annoyance, it is an issue that I could live with - if it was the only issue. However, on a much to frequent basis, I'll seemingly get stuck in continued rounds of disconnections - repeatedly getting disconnected after periods of anywhere from a few seconds to a few minutes. This can go on for minutes or hours, and usually happens from at least once a week to several times a day. Even going to the extreme of disconnecting the wireless modem and "giving it a rest" for 10 minutes does not help. Each disconnection is rather disruptive, especially as each new connection brings a new IP address - which makes it practically impossible to gracefully resume many types of connections.

I am not the only one having these issues. A few Google searches show a number of other users with related issues with Alltel (and also Verizon), but no currently known solutions.

This is not a signal strength issue, as the reported RSSI is almost always at one of the two best reportable values: -63 or -47 dBm, or 4/5 - 5/5 bars. I've tried adding an external antenna, which made no difference. I'm unable to find any correlations to the time of day or other environmental factors.

Alltel technical support is all to eager to blame this on my use of Linux. However, the problems are identical if not worse when using Windows and their QuickLink Mobile software (made by SmithMicro Software). If anything, Linux is better able to handle the connection and any necessary re-dialing. Trying the data card on different computers (a desktop and 3 available laptops) does not change anything. My current data card is a UTStarcom / Pantech UM175AL. At one point I tried a loaner Huawei EC228 card, which also made no difference in the issues. Worse, Alltel got the accounts mixed up and ended changing the phone and account numbers on the UM175 that I had to keep, and I received a threat of legal action in the mail for breach of contract!

I've seen a number of theories for the cause of these disconnections, and have tried various solutions to them all, but without any success. This includes setting various MTU values, asyncmap settings, adjusting for LCP echo failures / timeouts, tweaking the AT commands sent in the modem chat script, and making sure that there isn't any private-address traffic being sent over the connection.

I've contacted Alltel's technical support many times to try to resolve this issue, always speaking with their separate "data card support" department, and probably totaling at least 20 hours of phone time. The original theory was that the data card was bouncing between Alltel and Sprint towers, and loosing the connection each time this happened. They updated the preferred roaming list (PRL) on the device several times, without any affect. An interesting fix they tried was setting the device to "static" mode. Supposedly this doesn't refer to a static vs. dynamic IP address, but it did result in obtaining a persistent IP address. This also did appear to help with the connection stability - but not perfectly. Most critically, this mode caused an unfortunate side effect: I could no longer receive any incoming connections. Alltel's response was that incoming connections are not supported. Many ISP's, especially residential, block common server ports - such as 80 for HTTP and 25 for SMTP. This is a debatable practice, but doesn't affect "simple" activities such as web browsing and email. However, in this "static" mode, it was as if I was behind another firewall with all incoming ports being blocked - affecting a number of Internet uses including remote desktop applications, VPN clients, instant-messaging file transfers with family and friends, or even online games.

Some of my relatives also experience the same issues. However, with pretty much only checking their email a few times per week, they seldom notice the issues. Eventually, we moved closer to Appleton - about 15 miles closer - and while this has slightly improved things, it is still not acceptable. Now that other Internet options are available, including cable and DSL, I would have switched already if we weren't stuck with the 2 year contract, and with over a year remaining yet. We would have better service at half the price.

DNS Concerns

This is another issue I found while checking-up on some things in the aftermath of the DNS cache poisoning vulnerability reported by Dan Kaminsky. The first issue I noticed was an alert on a report from the ICSI Netalyzer hosted by UC Berkeley: No DNS Port Randomization. There have been too many issues with ISP DNS servers in the past, e.g. DNS hijacking by several companies including Charter Communications, Comcast, and Time Warner. Due to this and other various ISP issues, along with being able to improve performance through local caching, I typically run my own DNS server for my home network. I thought I may have had it set to forward all DNS requests to the DNS server address received through DHCP from the ISP, and that I'd just change it to do its own recursive resolution.

This is where things got interesting. Alltel is not only playing tricks with their own DHCP-advertised DNS servers, but are intercepting all outgoing DNS traffic (UDP port 53) and acting as a transparent proxy. This is quickly confirmed through one of the same tools that can be used to check for the DNS cache poisoning vulnerability: porttest.dns-oarc.net. On my Alltel PPP connection, the peer's IP address is currently 75.116.231.29. The client IPs seen by the port test tool are 75.116.63.155 and 75.116.63.156. Neither of these addresses are mine, which are in a completely different subnet. None of these addresses currently have pointer records (PTR) registered, though I recall seeing them resolve to windstream.net on Windstream Communications only about a week ago. ARIN WHOIS does show that all these addresses belong to Alltel Communications.

The ICSI Netalyzer results seem to reflect this as well. Two points listed under Reachability Tests / General Connectivity were "A DNS proxy or firewall caused the applet's direct DNS request to be sent from another address. Instead of your IP, the request came from 75.116.63.156" and "A DNS proxy or firewall generated a new request rather than passing the applet's request unmodified". Interestingly, both were listed simply as information rather than warnings. Many people tout OpenDNS or other 3rd-party DNS servers in situations like this. Even though OpenDNS follows some of the same practices, such as redirecting invalid lookups instead of sending NXDOMAIN by default, at least they provide options to control and disable this functionality. Unfortunately, it is impossible to query any specific DNS server - OpenDNS or otherwise - while Alltel is intercepting these requests. The only work-around I can think of short of Alltel fixing this would be finding a DNS server that provides service on a non-standard port, or sending all DNS queries through some available VPN to another location. I found this forum thread from February that showed a few other users complaining of similar issues with Alltel redirecting DNS requests. While the issue apparently had been resolved, it only appears to have been temporary. Since there is nothing as a user that I can currently do to avoid these issues, I can only hope that Alltel is on top of things, including having other measures in place for avoiding DNS poisoning.

Update (2009-08-26):

Alltel finally decided to do something for me. As previously mentioned, I've lost count of how many calls I've made to Alltel technical support. Several times, I also visited the Alltel corporate retail store, and brought with me printed connection log that showed my excessive number of connections and disconnections. About a week ago, I made one last trip to the corporate retail store. After talking with the same store manager I happened to previously speak with a few months ago, he agreed to cancel my contract with no early termination fee. It wasn't an ideal solution for either of us - I'd rather have a working, mobile connection, and I'd hope that they'd value and want to keep my business. However, at least now I don't have to continue paying for service that doesn't work. I will soon be signed-up with faster, more reliable, and less expensive DSL service.

Related postings:

• EVDOforums.com: Alltel EVDO Service: Disconnections, DNS proxying, and other issues
• WirelessAdvisor.com foruums: Pantech UM175AL Disconnect
• OpenDNS:
  • OpenDNS Community Forums: Domain Name System (DNS) troubles: Verizon broadband wireless
  • OpenDNS Idea Bank:
    • Listen on another port
    • Offer an alternate port for DNS queries
    • Alternate port
    • DNS On Nonstandard Ports (Defeat stupid ISP proxies and forced DNS)