[FoRK] Quarters needed for Apple to put Microsoft out of our misery?

J. Andrew Rogers andrew at jarbox.org
Sat Jan 31 01:34:38 PST 2015

> On Jan 30, 2015, at 9:43 PM, Stephen D. Williams <sdw at lig.net> wrote:
> On 1/30/15 5:47 PM, J. Andrew Rogers wrote:
>> You do not understand the engineering and architectural constraints. Making the client thinner and dumber fits the constraints more easily for many broadly useful apps like these.
> I don't?  I think I said that there are a wide range of apps and some of those require interesting local capabilities.  Of course most are simpler on the client.

There are complex power and bandwidth constraints while operating in relatively dynamic environments. It has to work outside the lab, which leads to architectures that are not always intuitive. I’m sure the hardware engineers know what they are doing. 

> I almost referred to this as "mysterious".  My first choice was correct.  Sounds interesting.  I don't see the value yet.  

It is a real-time god view of reality at a data level, sort of analogous to Google’s view of the Internet today but an exponentially larger superset of that and much more useful. This is not a visualization, it is a continuous physics reconstruction of the physical world as a data model using every kind of streaming and slow-moving data imaginable but especially sensor data; the entire history of everything the system has seen is spatiotemporally fused and fully online for immediate analysis right up to “now” with sub-second latency. Analysis is concurrent with integrating e.g. a terabyte per minute of live data, so it is never offline. 

It should not take much imagination to understand why so many people find this valuable. People and businesses live in reality, not on the Internet.

> Also, something like the status of the world seems embarrassingly parallel: just shard it.

Mathematically, the physical world is nothing like the virtual world. Let me give you two of the several elementary problems that you have not dreamt of in your sharding philosophy:

A sharding function has the properties that it ensures efficient, uniform distribution and parallelization of data and operations. The physical world is composed of 3-dimensional shapes embedded in a 3-dimensional surface (ignore time for the sake of simplicity). Provably, no sharding function exists for this data model.

Let’s assume that you sharded the world. The physical world is not Euclidean. Cartography has never had to deal with the peculiar issues of computational geometry for sharding these types of analytical data models, so you can’t just drop in PostGIS or similar. It turns out that this area of computational geometry is poorly studied generally. No one has faked it without things ending badly.

Everyone’s immediate reaction is to tile the world. You can’t do anything like that. A solution is necessarily pretty exotic and not expressible on Hadoop, Spark, or whatever is the “big data” platform du jour. 

More information about the FoRK mailing list