[FoRK] Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts

J. Andrew Rogers andrew at jarbox.org
Thu Nov 13 17:46:26 PST 2014

> On Nov 13, 2014, at 2:39 PM, Stephen D. Williams <sdw at lig.net> wrote:
> Deep learning is not _just_ a rebranding of neural networks.  That's like saying that AWS/Google+Linux+Docker+... is just a rebranding of operating systems that we had in the 60s.  Deep learning etc. is _working_ neural networks for a wider range of applications and to a degree of efficacy that we didn't have before.  We should all be chagrined that the AI Winter prevented us from discovering the tweaks that transformed almost-working neural networks to very powerful and efficient tools that we now.  Those tweaks, like dropout, are the key to the whole thing working resiliently.

Deep learning is obscure neural network computer science from ~20 years ago with better marketing. Being unreasonably computationally expensive on the hardware of the day was not the only reason it was abandoned. Most people jumping onto the fad seem to be unaware that it is the second time around. That said I am not sure what a “neural network” actually is anymore. The best deep learning systems are not biologically inspired. 

> It is pretty clear that the brain doesn't in any way propagate signals backwards to train?  Is it magic?

I can’t speak for Jordan but back prop is a pretty limited learning mechanism regardless of whether or not neurons can do it. There are more productive things you can do with neurons that can transmit signals bidirectionally if learning is your goal.

>> Michael Jordan: I think data analysis can deliver inferences at certain levels of quality. But we have to be clear about what levels of quality. We have to have error bars around all our predictions. That is something that's missing in much of the current machine learning literature.
> Really?  Half of machine learning is all about detecting fit vs. overfit vs. effectiveness.  Probability reasoning, understanding and incorporating likelihood and success rates, has been the core of the best AI for the last decade.

I think mathematics has a better handle on limits of predictability than the machine learning literature at this point. It is partly because many machine learning models are not amenable to that kind of analysis, which in my estimation is indicative of a flaw in a machine learning model. 

>> Michael Jordan: I am sure that Google is doing everything I would do. But I don't think Google Translate, which involves machine translation, is the only language problem. Another example of a good language problem is question answering, like "What's the second-biggest city in California that is not near a river?" If I typed that sentence into Google currently, I'm not likely to get a useful response.
> IBM Watson?  Siri?  This is natural language + linked data (DBPedia etc.) with some forms of machine learning.  Various systems have elements of this already.

Nope, Jordan is correct here, though he might not know why. That question is not (efficiently) answerable with graph-like data structures e.g. Watson or linked data. Assuming otherwise is a common error; a surprising amount of money has been wasted on startups and systems where apparently no one noticed. 

More information about the FoRK mailing list