[FoRK] Joyent, cloud service evolution

J. Andrew Rogers andrew at jarbox.org
Mon Jun 20 12:12:45 PDT 2016

> On Jun 19, 2016, at 5:39 PM, Stephen D. Williams <sdw at lig.net> wrote:
> Capturing and remembering everything so that you can do arbitrary queries later is crazy, but I can see that some entities might
> want to do that for various reasons.  Once you have an actual application, you ought to be able to use more than data concentration:
> data should be processed, filtered, and minimized as close to the point of collection as possible.

Nah, everyone forgets that while the data from their sensor widget is small, almost all high-value analysis of sensor events depends on contextualization with external sensor data. That external data looks small from the consumer side on a per event basis but being able to deliver that context unavoidably requires that the external data model capture and remember all of its data. And since the *business model* of most sensor widgets is to build huge aggregations of said sensor widgets and sell context to other systems… guess what, you are now in the business of capturing and remembering everything.

You can’t precompute or summarize this data for contextualization because the primary operation used in IoT contextualization is spatiotemporal graph analysis. Turning these operations on your data into a lookup table is literally boiling-the-oceans intractable both in terms of computation and storage. Impossible. Therefore, to provide this contextualization you are forced to compute exactly what you need, exactly when you need it, which *is* tractable. Which means you need to capture and remember as much data as possible. And this is why everyone does it this way.

A lot of people have this odd idea that IoT analysis is some kind of trivial time-series analysis. Almost none of the poster children industries for IoT (mobility, automotive, drones, smarter cities, remote sensing, infrastructure, etc) actually do their analysis this way.  

> I don't think that all or even most IoT applications need such extreme scalability, although they ought to be fairly scalable.  We
> need a good covering set of current and future IoT data collection, control, and application types to reason about this.

This is the standard Dunning-Kruger response given by engineers that have never actually done analysis of IoT data in any meaningful way. :-)

Big, technically-adept companies have been blowing millions of dollars on failed IoT analysis projects for many years now. IoT analysis has been around for much longer than it has been trendy. On some level, it is amusingly arrogant that web engineers think they can show these industrial companies how to do something that they have not already tried over the last several years. Most of the technical savviness in this space is not at the Internet companies.

More information about the FoRK mailing list