[FoRK] PubSub NG Re: MQTT ...

Marty Halvorson marty at halvorson.us
Sat Feb 28 15:38:50 PST 2015


On 2/28/2015 1:00 PM, J. Andrew Rogers wrote:
>   Do you realize that your PubSub stream in these models is 1-10 Gbps per source and that a single*logical*  stream may be the aggregate of many sources? It is truly decentralized at a fundamental level because it has to be.
Reminds me of the data streams for the experiments on the LHC at CERN 
where the initial data streams from the sensors inside the detectors are 
filtered then aggregated in hardware resulting in streams of 10's of 
terabytes/second over high speed fiber.  The streams are processed by 
software in low power PC's to 10's of GB/second, and then sent to 
central high power machines.  The high power machines (think Cray) then 
do addition filtering with results to a data base for particle 
scientists to examine.  The net result is about 6 orders of magnitude 
reduction in the data.  And this goes on for however long the ring is 
running.

I spent some time there consulting on the data transmission between the 
hardware aggregaters and the PC's back in the late 80's.  I got to go 
down to the Aleph experiment.  Very interesting standing at the bottom 
of a 30 meter hole straight up for about 100 meters.


More information about the FoRK mailing list