[FoRK] PubSub NG Re: MQTT ...
marty at halvorson.us
Sat Feb 28 15:38:50 PST 2015
On 2/28/2015 1:00 PM, J. Andrew Rogers wrote:
> Do you realize that your PubSub stream in these models is 1-10 Gbps per source and that a single*logical* stream may be the aggregate of many sources? It is truly decentralized at a fundamental level because it has to be.
Reminds me of the data streams for the experiments on the LHC at CERN
where the initial data streams from the sensors inside the detectors are
filtered then aggregated in hardware resulting in streams of 10's of
terabytes/second over high speed fiber. The streams are processed by
software in low power PC's to 10's of GB/second, and then sent to
central high power machines. The high power machines (think Cray) then
do addition filtering with results to a data base for particle
scientists to examine. The net result is about 6 orders of magnitude
reduction in the data. And this goes on for however long the ring is
I spent some time there consulting on the data transmission between the
hardware aggregaters and the PC's back in the late 80's. I got to go
down to the Aleph experiment. Very interesting standing at the bottom
of a 30 meter hole straight up for about 100 meters.
More information about the FoRK