[FoRK] Whither BEEP? Re: HTTP 0.2, 9p, and op

Stephen Williams sdw at lig.net
Thu Dec 24 19:47:08 PST 2009


I remember Marshall Rose announcing BXXP in about 1999 or 2000, however 
the RFC is dated March 2001.  SCTP was in an RFC in Oct. 2000.  It would 
be interesting to know the relationship.  Did Marshall announce BXXP and 
cause others to work on SCTP?  Did SCTP work inspire Marshall to create 
a complementary (somewhat) solution at the "application protocol 
building block" level?  We ended up not using either very much for 
various reasons.  It is possible that both undermined the other.

We get by on HTTP by sheer bandwidth, CPU, memory, and lack of good 
competition.  That is even somewhat true in mobile now, although I think 
they may be using a rate-based tunnel to a web proxy to avoid severe 
problems from packet loss, slow start on multiple streams, etc.  The 
Palm Treo / Centro models used a web proxy that resized images, 
filtered, etc.  Seemed to slow things down a lot because, I think, the 
proxy was either underpowered or it throttled.  I'm glad the current 
generation, Pre et al, are past that.

Sean Conner wrote:
> It was thus said that the Great Stephen Williams once stated:
>   
>> Similar, but without assumptions on transport and with IETF application 
>> protocol like framing (ASCII), yet very lightweight and able to be 
>> optimized (pad numbers, etc.).
>>     
>
>   Ah, silly me ... I thought that IP was the networking protocol that didn't
> make assumptions on transport, but then again, I never did like the OSI 7
> layer burrito.
>
>   
>> If SCTP were easily used both as the transport and over an arbitrary 
>> transport and if it included TLS, then it would give you most or all of 
>> what BEEP provides.
>>     
>
>   I seem to be missing something.  SCTP rides on top of IP; what else would
> you need SCTP to ride on top of, if IP isn't available?  (And yes, there are
> RFCs covering IP over SCSI, ATM and avaian carriers, so I'm having
> difficulty seeing where you would have SCTP and *not* IP)
>   

TCP, UDP, UDT, Skype, XMPP, AMQP/RabbitMQ, HTTP, SCTP->app->SCTP->app...
If you can't do SCTP directly, or if your problem involves multiple hops 
or nesting, then you need to layer.  There is a lot more to the world 
than a point to point link, even with full Internet routing.
>   
>> Another part of some problems is end-to-end flow control.  It would be 
>> convenient if you could map transactions, messages, and chunks through 
>> both each transport and over the whole message path using the same 
>> mechanism, with or without various kinds of nesting.
>>     
>
>   I'm haing trouble parsing that paragraph.  What do you mean by it all?
>   

The problems that networking protocols solve appear at multiple levels 
of abstraction and also sometimes need to be nested.  Let's say you are 
one of the few who realize that doing everything as chains of 
synchronous RPC calls is terribly inefficient, wasteful, and provides a 
fraction of the possible throughput.  Using asynchronous message queues, 
you send messages through TCP/UDT/SCTP/* to the first server.  That 
server may do some processing and/or it may be a message router / broker 
/ bus.  It produces messages that flow on TCP/SCTP/... to the next 
tier.  That tier does the same, and so on.  Now, in a simple application 
scenario, the sender may be just farming out of a few application 
requests to get some pipelining and basically expecting the results 
quickly.  Readers all along the pipeline are expected to always pretty 
much keep up with writers.  Describes some Wall Street apps.

Now, what happens when the writer has a lot to send and it is always 
faster than one or more tiers downstream?  The chokepoints could be CPU, 
an Internet data link, writing to storage, etc.  The result: packets 
pile up in memory at the various queues because the individual 
TCP/SCTP/UDT links are only doing flow control for their endpoints.  So, 
either your application is fragile and takes huge resources, or you 
build additional flow control at the next layer up.

I've architected, designed, and built solutions for this a few times.

Here is a SIP example:
http://blog.tekelec.com/blog/?Tag=SCTP
> In the following figure, a request starting at the caller traverses a 
> UDP hop, an SCTP hop, and a TCP hop before reaching the callee.
Works fine for voice traffic, might work with enough bandwidth for 
video, and would fall apart completely without a higher level flow 
control for any serious bulk transfer.
>   
>> SCTP as native & library, with UDT/Rate based flow, TLS, full TCP/select 
>> semantics, and able to run over raw packets, TCP, UDP, UDT, HTTP, and 
>> libOpenSkype, with a coherent solution for link vs. end-to-end flow 
>> control would be the ultimate solution.  Where libOpenSkype == automatic 
>> secure communication linkage between any 2 or more endpoints based on a 
>> unique ID, secure authentication mechanism, and over any required 
>> combination of communication link types to get the most efficient 
>> communication path: local packets, TCP, UDP, UDT, routed directly in 
>> whichever direction is required or routed through a "nearby" system with 
>> some policy.  All intermediated by either a DNS-like neutral 
>> coordination server, or ISP-nearby, or private configured coordination 
>> server.  Perhaps also with distributed self-healing hash/cache like HTTP 
>> cache / memcached + BitTorrent, pub/sub propagation, etc.
>>     
>
>   Um ... yeah.  The only thing I have to say is ... I hope it doesn't break
> (http://boston.conman.org/2009/12/08.1).
>   

Although I buzzworded that to the max, an open solution that fits that 
description would benefit a lot of people.  And a closed version already 
exists that provides the core of that: Skype does all of that automatic 
virtual secure network plumbing.  It just isn't the solution in general 
because they have to keep control to retain their value.
>   
>>>  I took one look at the RFC for BEEP (RFC-3080) and saw:
>>>
>>> 	At BEEP's core is a framing mechanism that permits simultaneous and
>>> 	independent exchanges of messages between peers.  Messages are
>>> 	arbitrary MIME [1] content, but are usually textual (structured using
>>> 	XML [2]).
>>>
>>>  Okay, so now my application needs to link against a MIME parser and an XML
>>> parser (SAX, lighter weight but difficult to use, or DOM, easy to use but my
>>> God the memory requirements [3]).  So now my app becomes a bit more
>>> complicated.  Thanks, I'll stick to UDP [5].
>>>       
>> Those are naive assumptions.  First, for a typical application, you 
>> would only implement the Mime types you are interested in, which 
>> basically means you write a fixed string and read a fixed string, puking 
>> on anything else.  A set of applications could even define private mime 
>> types as very short strings.  The point is that the messages are typed 
>> and allow Mime-compatible strings.  No library, no parsing. 
>>     
>
>   No library I'll grant you, but there's still parsing, even if it is
> minimal in nature.
>   
2 strcmp's for a typical application.  If minimal, fixed size "mime" 
types were used, a quick compare of 4 bytes.
>   
>> Second, 
>> BEEP messages are opaque and can be anything you want, including Google 
>> Protocol Buffers, XML, SMTP, images, etc.  The RFC is just saying that 
>> XML messages work well with the text-based framing so you can debug 
>> easily, in keeping with SMTP, Imap, HTTP, etc.
>>     
>
>   As I keep reading the BEEP specification, I keep asking myself "Self,
> what's the point?"  BEEP over TCP includes a window size, which is
> duplicating what TCP already provides (but I can see why it's done, which is
> to get around the whole "TCP dropped packets and blocking" issue, which gets
> into the whole "leaky abstraction" thang and seems to introduce yet another
> layer that can break on a whim).
>
>   Looking even further, I see SOAP over BEEP and XML-RPC over BEEP and well
> ... it just seems overly engineered and complicated (just bcecause it can)
> just to schlep bits from point A to point B and back.
>   

Then why put SOAP over HTTP?  Why do you need HTTP at all?  Answer that, 
and you will have taken your first step.

For just part of the answer, and one of the features that few other 
protocols have: BEEP provides flow control on multiple simultaneous 
channels, all running over a single TCP (or whatever) connection.  This 
means you could many separate IM/chat/presence channels with minimal 
traffic (no need to keep identifying the destination), including bulk 
data transfer at lower priority, all playing nicely with TCP and sharing 
packets when possible.  Instead of sending 5 small packets, plus acks, 
for 5 separate IM / data transfer channels, you can send 1 packet with 
the data for all of them.  Huge win in some cases.
>   It also seems that a bunch of these layers exist solely to bypass
> firewalls run by facist network admins that only allow ports 80 (which is
> invisibly proxied anyway: http://boston.conman.org/2009/11/17.1) and 443
> (and really, what's the port of allowing *any* open port if you can tunnel
> anything through it anyway? http://boston.conman.org/2009/11/18.1), which
> means that firewalls have to get more instrusive, complicated, and make
> network troubleshooting all that more difficult (true story:  setting up a
> route for a client in their cabinet at a large data center.  One hour later
> and we still can't get to it from outside because a) the IP is private, and
> b) the consultants we're working with for the client can't figure out how to
> forward the network traffice from the public side to the router).
>   

Actually, none of this is to bypass firewalls.  Layering everything over 
HTTP is often done for that reason.  BEEP, if it had become popular, 
would have been its own border protocol with explicit firewalling just 
like HTTP.  Already, various protocols are supported directly for 
security filtering by the Cisco ACE product, including XML messages, 
HTTP, and others.  It does all of the SSL/TLS wrapping and unwrappting, 
determine what is allowed where, creates audit trails when necessary, 
and is intended to be the corporate central switching point for message 
queue traffice, web, email, SOA, database (I think), etc.

And Skype is already a master at getting through firewalls.  Because 
they kept tight control of the application, have great although opaque 
security, and didn't go down the p2p file route (again, it's from the 
guys who did Kazaa), it is ignored by many corporate and other 
environments now.

For more open networks, uPnP mostly solves the reachability problem, at 
least for UDP.  Not sure if SCTP would flow through or not.  Good thing 
to know.
>   
>> Have you used SCTP?  Has anyone?
>>     
>
>   Well, _Unix Network Programming, 3rd Edition_ shows that SCTP is used for
> ISDN over IP, M2UA, M3UA (SS7 telephony signaling), H.248, H.323 (IP
> telephony) and SIP.  So it is in use.
>   
SS7 is a bit esoteric for most companies and people.
It looks like SIP is not actually available and used over SCTP yet:
http://blog.tekelec.com/blog/bid/8752/Architectural-Options-for-Core-SIP-Signaling-Networks
> What are the challenges of building a large scale SIP-based signaling 
> network using SCTP as the foundation?


>   -spc (Having a hard time wrapping my brain around all these layers)
>
>   

sdw



More information about the FoRK mailing list