[FoRK] overhead of RESTful stuff

Stephen Williams sdw at lig.net
Thu Mar 8 10:06:32 PST 2012

I've implemented a modified OAuth from scratch at Sony DRBD (Android eReader app), and I've used good OAuth libraries at 
GREE/OpenFeint.  Signpost is the standard Java library, and it is very easy to use.  Even easier once GREE's SDK is out of beta, 
because I used some of their existing ideas and improved it a lot so that it is very easy to use.  I'm sure similar exists for 
Javascript.  Implementing my own version 2 years ago, with a custom combination of signature (that had to be assembled) and crypto, 
was not easy, especially getting it to work against an opaque server (no logs, just pass/fail).  With Signpost, you just assemble an 
HTTP call like you normally would, then make a call to sign it for OAuth.

The request headers are not required for GET, you can encode everything in the request parameters.
The main problem to solve is hiding the consumer key/secret in your JS code.  Here is a solution, plus a mention that OAuth 2.0 has 
a "User-Agent" profile that address OAuth in pure client-side Javascript.


OAuth works by having you authenticate by some method (which could be anything) and then making requests based on the token 
returned.  It is an app-level session cookie, used in a highly secure way.

OAuth was created to solve the problem of authentication domains vs. application domains.  It has one or more methods, that work on 
all browsers, for getting forwarded to the actually application server after authentication.


On 3/8/12 9:51 AM, Lucas Gonze wrote:
> I got a lot of pushback recently on OAUTH for API purposes. Problem #1
> is that it relies on a request header, which isn't possible when
> you're grabbing JSONP by inserting a<script>  element, which will be
> necessary until CORS reach is complete. Problem #2 is that it's more
> complex to code than the alternatives.
> Developers I talked to strongly preferred getting a token which you
> then include in future requests as a parameter. This is highly
> un-RESTful, but developers see that argument as frippery for ponces.
> On Thu, Mar 8, 2012 at 9:42 AM, Stephen Williams<sdw at lig.net>  wrote:
>> That is assuming that you have the right application server model.  Finally,
>> good models are well known now.  However, many implementations are done
>> poorly the first time around: thread per request, no or poor queuing of new
>> requests, no pipelining or connection reuse, etc.
>> Additionally, a lot of popular components are not very memory or CPU
>> efficient.  XML libraries for instance are often ridiculously large and
>> fairly inefficient.  Communication between web servers and app components
>> can kill performance and be a bottleneck.  And then the application
>> communication model can be very poor.  Things are better with AJAX+REST, if
>> done well, but many apps are overly chatty and/or fragile.
>> Good use of memcached and similar solves a lot of people's problems.  As can
>> certain database solutions, NOSQL and sharded MySQL etc.
>> Although used incorrectly a lot of the time, lightweight message queue
>> components can drastically improve efficiency, scalability, resilience,
>> flexibility, and development evolution.  One of the best overall
>> architectures is to have a high performance web server (nginx is one of the
>> most popular) that connects directly with very efficient message queues to
>> application engines, distributed and multiplexed as necessary.
>> Although mostly ignored right now, depending on what data needs you have,
>> you might want to consider W3C EXI (which I worked on).  Although they left
>> out a couple things I think are crucial to fully optimizing things like REST
>> traffic, it is a good base.  I keep wanting to get excited about creating a
>> version that has those features (deltas, externalized metadata objects for
>> "compiled" schema, HBAF (hybrid byte aligned format)), but other tasks are
>> coming first.
>> These days, both Java and Javascript have heavily optimized JIT compiling
>> runtimes (only partway there for DEX-based Java on Android though), so
>> there's usually little to worry about performance wise on endpoints.  The
>> algorithms and architecture are far more important.
>> The SOA stack is terrible, don't use it.  Use OAUTH authenticated HTTPS with
>> simple object models.  Nearly everyone uses JSON for this now, with some
>> (usually bad) use of XML.  I like GSON for JSON parsing in Java, and my Ssx
>> parser for most application level parsing of XML.  Ssx is fast, tiny, and
>> highly simplified for concise development, but currently only parses.  For
>> outputting XML, in most cases you just want to do the obvious construction
>> which is generally faster.
>> EXI, especially with my added features, would be the most efficient by far
>> in size, parse speed, least CPU.
>> Stephen
>> On 3/8/12 7:26 AM, Gregory Alan Bolcer wrote:
>>> If you use a compiler like Excelsior Jet, performance is the same.
>>> For REST, most Java stacks are pretty mature.  There are some C++ + other
>>> scripting languages that are straight across functionality transfers.
>>> Hardware is a commodity, so any hardware costs will more than be made up
>>> for by development and maintenance costs.
>>> Most modern serves have dual or quad GBit ethernet, so a proper network
>>> architecture which takes into account routing front ends and good balancing
>>> using proper net naming and setup shouldn't be a problem.
>>> The biggest bottlenecks for almost 99% of REST based applications will be
>>> disk i/o, memory size, and then cpu.
>>> Greg
>>> On 3/8/2012 4:25 AM, Eugen Leitl wrote:
>>>> There's some enthusiasm for SOA and RESTful stuff at my dayjob lately
>>>> (yeah, we're always trailing state of the art by a decade or two),
>>>> so I'm worried about overhead. These Java wonks don't have a good
>>>> grasp of the entire hardware/network stack, and potential overhead
>>>> of calls.
>>>> First Java, how much overhead relative to C/C++ are we looking
>>>> at? I estimate a factor of 5-10 in terms of hardware budget
>>>> (more fat nodes, or many more the leaner nodes we're currently
>>>> using). Is this about right or too pessimistic?
>>>> I estimate that a lean box will crap out at about ~kHz of RESTful
>>>> calls, especially if it's all a big wad of Java oozing behind the
>>>> NIC. Is that about right? Linux/Windows differ much here?
>>>> In terms of networking, is GBit Ethernet going to become a
>>>> bottleneck for building applications from REST interfaces?
>>>> Are any of you using REST on InfiniBand or is the idea insane
>>>> in general, given the (probably awful) overhead?

More information about the FoRK mailing list