Dan Kohn (firstname.lastname@example.org)
Mon, 16 Sep 1996 07:02:33 -0700
>From: Gordon Cook[SMTP:email@example.com]
>Sent: Sunday, September 15, 1996 6:58 AM
>To: Dan Kohn
>Cc: 'Gordon Cook'; 'Wendell Craig Baker'; 'Keith Dawson'
>Subject: Re: http://www.boardwatch.com/mag/96/sept/bwm17.htm
>Thanks, I read that yesterday after michael dillon ripped it to shreds
>inet access. In addition to sowa's ignorance, it is an interesting
>commentary on the ignorance of rickard and hakala the editor and
>for publishing such a piece of garbarge.
>That is not tgo say sprint doesn't have trouble. They fired sean doran
>last monday and moral is excedingly low.
>The COOK Report on Internet For subsc. pricing & more
>431 Greenway Ave, Ewing, NJ 08618 USA ten megabytes of free
>(609) 882-2572 (phone & fax) visit
>Internet: firstname.lastname@example.org For case study of MercerNet &
>TIIAP induced harm to local community
>On Sat, 14 Sep 1996, Dan Kohn wrote:
>> [* This article contains more misinformation than fact, but it's
>> interesting to see how the Small ISPs view the changing world. I am
>> reminded of Peter Ford's statement that fundamentally inefficient
>> technical situations (like non-CIDRized addressing) will eventually be
>> cut down by the lawnmower of economic efficiency. - dan *]
>> Meltdown Or Monopolistic
>> Power Play?
>> Picture this scenario. You run a small ISP in a major urban center. You
>> begin getting phone calls from disgruntled customers who say it's taking
>> them over 30 hops to get from point to point. You get notification from
>> your upstream connection provider that all of your assigned IP numbers
>> will soon be changed, creating havoc for your customers. The costs of
>> these changes, passed on to your customers, creates a second firestorm.
>> Major customers begin to look for alternative ISPs. Sprint and other
>> major backbone carriers begin advertising their "stable" services as an
>> alternative. You lose your largest customer.
>> Your system continues to run at a snail's pace _ even though you
>> invested in the best equipment you could afford. Losses mount _ and you
>> begin to worry about how you're going to put food on the table. You call
>> your regional telco to perform loopback testing to see where the
>> problems are. Your telco tells you that your system is fine _ that the
>> problems are all upstream _ perhaps even as high as the Sprint network.
>> People are abandoning your ISP business to join Sprint, AT&T and MCI
>> networks _ when the actual slowdowns are occurring on their operations.
>> You begin to wonder if something is going on.
>> THE NEXT INTERNET MELTDOWN
>> At least one backbone provider - Sprint _ is intentionally turning away
>> small ISP traffic, and others are considering similar moves. Sprint
>> claims that it must "dampen" some externally generated traffic during
>> peak load periods to avert an Internet meltdown on its portion of the
>> Internet. When traffic is turned away, it's like being turned off of the
>> interstate highway _ and being forced to take back roads to your
>> destination. When traffic from small ISPs is selectively dampened, it
>> makes you wonder if there isn't some violation of the law taking place.
>> The fact is that trade laws which might have protected you under
>> traditional circumstances don't pertain to the Internet.
>> Even if they did, the fear of losing the stability of the Internet would
>> probably make regulatory officials look the other way. The Internet is
>> facing a major meltdown sometime by the end of 1996, according to some
>> members of the Internet Engineering Task Force (IETF), the protocol
>> engineering and development arm of the Internet. The IETF warns that
>> today's Internet is on the verge of collapse due to routing protocol
>> implementation problems and increasingly burdensome routing updates.
>> Essentially, the Internet is on the verge of failing because: 1) The
>> superhighways of the Internet are being backed-up at the intersection
>> traffic lights (the routers); 2) Cisco routers, which dominate the
>> Internet backbone, are failing to properly handle the peak loads because
>> of inherent protocol handling problems; 3) A lot of needless traffic is
>> duplicated daily on the Net _ IP address changes, web site
>> announcements, and router updates; and 4) Many smaller ISPs clog the
>> backbone with announcements of dial-in user router updates.
>> LAST YEAR, INTERNET COLLAPSE
>> WAS ALSO IMMINENT
>> Last year, a similar collapse was imminent. That potential collapse was
>> caused when the NSFNet turned the operation of the Internet over to the
>> private sector, and major changes in routing information caused backbone
>> routers to stop forwarding packets due to overloads in peak hours. Cisco
>> Systems came to the rescue by issuing a temporary fix for their routers
>> (which hold over 85% of the market on the Internet backbone), and by
>> coming up with a new technology called route dampening, to reduce the
>> frequency of router updates and maintain router control over updates.
>> Since then, while no major collapse has materialized, periodic router
>> shutdowns across multiple backbones have been occurring with greater
>> frequency _ slowing the Internet to a crawl at times. See:
>> (www.ietf.cnri.reston.va.), (www.internic.com) and (www.saic.com).
>> CISCO'S DIRTY LITTLE SECRET
>> AND THE NANOG STUDY
>> While Cisco Systems is taking the lead in saving the Internet from
>> imminent collapse, and in pushing for IP renumbering and fees that would
>> hurt the smaller ISPs, Cisco seems to be hiding its own dirty laundry.
>> According to an engineering work group report of the North American
>> Network Operators Group (NANOG), which includes the Internet's largest
>> commercial network operators, the problems that seem to be occurring on
>> the Internet only happen where Cisco routers are used. NANOG suggests
>> that a manufacturing problem inherent in Cisco routers may actually be
>> at fault. The NANOG traffic study (see http://www.academ.com/nanog)
>> found that:
>> *Instability in Internet traffic is increasing despite use of dampening.
>> *There is a strong correlation between routing instability and network
>> congestion. Daily incidents suggest that instability peaks occur
>> between 6:00 and 8:00 a.m., and there is an abrupt drop at 5:00 p.m.
>> *An enormous number of routing update withdrawal announcements
>> are routed each morning. The study found that most such updates
>> are redundant or unnecessary, often done merely to advertise
>> the announced sites or to eliminate performance pressure
>> on specific routers;
>> *all reported problems, so far, can be traced to Cisco routers
>> and configuration errors.
>> *Use of non-Cisco routers is discouraged by the largest ISPs,
>> even though other routers do not appear to have the same kind
>> of routing problems.
>> *A problem that Cisco routers have with the Internet Protocol
>> itself is causing router update oscillation, link/router failures and
>> *Cisco routers cannot keep up with processing of the
>> announcements _ and rely on dampening technology to
>> continue operating during peak traffic periods.
>> SPRINT BLOCKS OUT SMALL ISPs
>> U.S. Sprint decided to intentionally block certain users' communications
>> that originate from small ISPs as a result of router traffic problems.
>> Sprint says it is being forced to block communications because of
>> overcrowding on its network. MCI and AT&T and the Regional Bells are
>> weighing similar moves. For details see: http://ftp.sprintlink.net and
>> The decision by Sprint to block out part of its routed traffic _
>> electronic mail, file transfers, and visits to World Wide Web sites by
>> users who are passing through Sprint from small ISPs _ in a medium where
>> everybody is supposed to be linked to everyone else, is one possible
>> alternative for the future.
>> Sprint is being applauded by some members of the IETF and by Cisco
>> Systems for being the first ISP to take action to begin to eliminate
>> address crowding. It is estimated that over 10,000 transactions are
>> blocked in the national Sprint network every day, and that in order to
>> keep Sprint's Cisco routers working properly, the number of
>> communications turned back is growing daily.
>> According to Sean Doran, a Sprint engineer who explained the Sprint
>> filtering process to NANOG:
>> "Very roughly speaking, the goal (of Sprint's filtering) is that if you
>> have a very long prefix and it flaps a couple times, you can go home for
>> the day...." (Your messages are blocked.)
>> "At the moment, you are much more likely to be dampened if you are NOT a
>> Sprint customer. That is, the normal values for 'bgp dampening' are in
>> place if you have a customer connection to SprintLink or ICM. However,
>> we will be evaluating the engineering and operational effects of a
>> similar policy with respect to SprintLink customers, as an effort to
>> encourage some of our lazier customers to begin doing their own (IP
>> renumbering) aggregation wherever possible."
>> According to Sprint, their problems stem from the recent exponential
>> growth of the Internet and the growing number of companies that attempt
>> to do major things with only temporary dial-in connections. Sprint
>> explains that the routing tables of the many linked networks that are
>> passing through Sprint have become too complex for most of its routers
>> to handle. Their chief concern, according to a Sprint spokesman at the
>> Dallas office, is that things are just going to continue to worsen.
>> Sprint counters its critics by saying that its first and foremost
>> responsibility is to preserve network stability for its own customers.
>> Most IETF members tend to agree with the Sprint approach. If you are a
>> small ISP, that means trouble.
>> PUNITIVE "SOLUTION" POSED FOR SMALL ISPs
>> Cisco Systems' Yakov Rekhter thinks he also has a solution to the
>> routing collapse problem. He proposed a major overhaul of the routing
>> procedures at the IETF meetings held in Dallas and Los Angeles this
>> spring. The proposal called for IP renumbering according to a
>> hierarchical aggregation scheme in accordance with an IETF proposal made
>> in 1992.
>> Rekhter's approach is now close to adoption by the Internet Assigned
>> Numbers Authority (IANA), the central coordinator for the assignment of
>> unique parameter values for Internet protocols. It would pose little or
>> no hardship for the largest ISPs. In fact, it conserves their address
>> space, even if they aren't using it.
>> But the proposal makes it harder for new ISPs to obtain address blocks,
>> increasing the likelihood that small service providers will have to face
>> constant IP renumbering. It also shores up the largest ISPs' ability to
>> block out IP addresses. It calls for the "immediate renumbering" of all
>> smaller ISPs and their customers _ and would pass on the costs of all
>> renumbering to those smaller ISPs and their customers.
>> Rekhter also wants IANA to consider charging fees for any IP address
>> changes _ and for advertising all router updates on the Internet. The
>> proposal does not suggest who should receive the income from these fees.
>> Rekhter's proposal drew broad support from IETF membership, which is
>> dominated by router vendors and large ISPs. Pushpendra Mohta of CERFnet,
>> for example, believes the best approach is to limit such charges to
>> those ISPs who generate the most changes in router tables _ a process
>> known as flapping, which tends to bring down routers. The largest
>> backbone providers tend to generate the most flaps because of the
>> extensive traffic they carry. But small ISPs also generate many flaps,
>> because they often advertise the status of temporary dial-up links.
>> LARGE PROVIDERS BLAME SMALL ISPs
>> The large providers single out small inexperienced ISPs as the major
>> cause of problems on the Internet. Some of the more charitable critics
>> suggest that novice ISPs need to be educated on the damages they cause
>> when they advertise to the entire world that a dial-up connection is up
>> or down. Others would like to see the total elimination of small ISPs _
>> or at least their aggregation into a larger ISP's camp.
>> The traffic problem is causing systemic changes in the Internet
>> architecture that may bode very badly for the future of small ISPs. To
>> eliminate stress on their own router platforms, many of the major
>> backbone ISPs now peer only with providers of a similar size. These
>> moves make it easier for the backbone ISPs to police their own
>> downstream providers by aggregating their traffic and dampening, or
>> delaying disproportionate updates. When dampening occurs, it slows
>> downstream traffic to a crawl until the backlog of updates have been
>> handled by the backbone router. These delays are immediately apparent to
>> customers of the small ISPs _ who often fill the Internet with flames
>> and complaints about the small ISP's inability to provide competent
>> REKHTER'S PROPOSAL IS ANOTHER
>> INTERNIC FIASCO
>> As Rekhter put it in his proposal: "There are greedy ISPs and
>> incompetent ISPs. Forcing these ISPs and their users to renumber every
>> time they change service providers _ and charging them fees for all the
>> IP changes and router updates will certainly motivate them and their
>> users to lock their organizations into a single provider or face the
>> expensive and complicated task of total network renumbering. The recent
>> change at the INTERNIC on charging for the domain name seems to work.
>> Should the route charging mechanism be similarly done?" Apparently,
>> Rekhter isn't aware of all of the Domain Name filing problems InterNIC
>> is facing.
>> The problem with Rekhter's notion is that the largest ISPs have large
>> blocks of IP numbers allocated to them, while the small ISPs don't.
>> Large ISPs will often be able to renumber within their network _ thus
>> avoiding additional fees. Smaller ISPs will face higher operational
>> costs and start-up costs, creating major barriers to market entry.
>> CONSUMERS WILL HAVE FEWER CHOICES
>> The proposed IP changes will have far-reaching implications. According
>> to Jeffrey Schiller, MIT's network manager, "(When IP renumbering
>> schemes are adopted), users will be locked into their service provider,
>> who can rake them over the coals and jerk them around, and in order to
>> change providers they will have to go through a painful and costly
>> renumbering process." Larger companies will definitely begin to shy away
>> from small ISPs. This will hurt most ISPs' profits _ and perhaps run
>> many out of business.
>> An AT&T study estimated that for an average company of 500 employees,
>> the cost of IP renumbering could exceed $100,000. Private networks like
>> Silicon Graphics' 150 sites and 7,000 hosts will spend anywhere from
>> $700,000. to over $2 million to renumber. Will this create an incentive
>> for users to go directly to the large ISPs rather than risk facing major
>> costs every time a small ISP shops for a better rate, or changes are
>> made by providers upstream?
>> As if the costs of all these changes on the small ISP won't hurt
>> business enough, the proposals also provide an exemption for non-profit
>> providers at the expense of the for-profit providers. They call their
>> solution "address lending." Certain addresses will be set aside for
>> institutional non-profits who cannot afford to implement the renumbering
>> process. These addresses, of course, will be maintained on the large
>> ISPs, and part of the fees collected from the for-profit small ISPs will
>> go toward maintaining addresses for the non-profits.
>> What are the best solutions? IETF has been mulling over this for a few
>> years now with no new alternatives imminent. In any case, they'd like to
>> hear from you _ especially if you are an ISP. They are currently
>> soliciting comments on the potential router collapse, and on the Rekhter
>> proposals for charging for IP addressing and router updates. You can
>> make your comments known at http://www.ietf.cnri.reston.va.us.
>> If you are an ISP, immediately explain to your customers what is
>> happening, and what might potentially happen. Ask them what they would
>> appreciate from you to ensure their loyalty remains with you. Consider
>> establishing ISP co-ops with other regional ISPs in your area _ and
>> consider joining legal groups and associations that can help you
>> negotiate better with the large ISPs on the issue of IP renumbering and
>> who will pay.
>> What is for certain, is that the Internet customer understands even less
>> about how the Internet works than does the small ISP. However, customers
>> are very quick to observe sluggish operations (which as a result of
>> reading this article _ some may hopefully see _ is not necessarily
>> caused by the small ISP). And, customers are fickle when it comes to
>> having such sluggish services _ or being forced to increase renewal
>> rates to offset new costs for operations and fees. IP renumbering and
>> associated fees will provide disparate pricing advantages to the larger
>> ISPs. Even the advantages of small ISP customer service cannot overcome
>> a decision by backbone operators like Sprint to turn back
>> communications. The power is in owning the IP address blocks, and unless
>> you act now to be heard by those who might do something to help you, the
>> fight to build a business on the Internet could abruptly come to a
>> painful end.
>> Editor: Jack Rickard - Volume X: Issue 9 - ISSN:1054-2760 - September
>> Copyright 1996 Jack Rickard - ALL RIGHTS RESERVED