[FoRK] Human behavior is 93% predictable
dmorton at bitfurnace.com
Mon Mar 1 23:55:40 PST 2010
Interesting post on reddit about this.
Earlier this month, an article came out in Science titled "Limits of
Predictability in Human Mobility". It's had a lot of coverage in the media
and a few frontpage links in various subreddits. This line from the abstract
"Here we explore the limits of predictability in human dynamics by studying
the mobility patterns of anonymized mobile phone users. By measuring the
entropy of each individual’s trajectory, we find a 93% potential
predictability in user mobility across the whole user base."
There are several things you should understand before taking this as
scientific proof that we are all incredibly predictable:
This is only a study of people who have a regular cellphone used to make
calls at least once every 5 hours on average. It doesn't specifically say
who these people are, but given the authors, they are likely people living
in a dense urban area in China. These people may be more predictable on
average than other groups, and thus may not speak to humanity as a whole.
The location is only as specific as the nearest cell tower. Their figures
show that being sometimes as big as 100 km^2, almost always more than 20
km^2. This study does not speak to finer-grained predictability. (I'm 99.99%
confident that you will be somewhere on Earth for the next few hours, you
predictable person =P .)
The locations used are ONLY from when a call is made. This study does not
speak to the predictability of people when they do not use their phone –
it's possible that people use their phone less when doing less predictable
things. Also, they don't know where most people are around 70% of the time;
it may be misleading to extrapolate from there.
This study is based on entropy and Fano's Inequality. You don't really
need to fully understand those two, but you should understand this: Fano's
inequality provides a way to say that predictability has an upper bound
based on the entropy. The actual predictability could be much lower.
Moreover, the actual predictability that a realistic prediction algorithm
could ever provide could be even lower than that.
While this is an interesting upper limit, it doesn't show that we are really
93% predictable, just that it is possible. The authors stated it well
themselves: "our results indicate that when it comes to processes driven by
human mobility, from epidemic modeling to urban planning and traffic
engineering, the development of accurate predictive models is a
scientifically grounded possibility." It will be a long time before we know
if that possibility can ever become a reality.
This stands in stark contrast to, for instance, Aol's coverage:
"Physicist Albert-László Barabási can guess where you will be tomorrow at 3
p.m. And where you'll be Saturday night at 8. In fact, given enough data, he
can predict your location at any time, with an average 93 percent accuracy."
No, no he can't -- no such model exists yet, as the paper makes explicit,
and it is very likely that it would not be able to perform that well.
On Tue, Mar 2, 2010 at 2:14 AM, Jebadiah Moore <jebdm at jebdm.net> wrote:
> On Sun, Feb 28, 2010 at 4:19 AM, Ken Ganshirt @ Yahoo <
> ken_ganshirt at yahoo.ca
> > wrote:
> > I ask why it's interesting to you because you will have a different
> > perspective and may be able to point out something specific or an angle
> > view it from that might get me excited. Curiousity and a bit of ennui.
> - Early detection of mental disorders via habit change
> - Better game AIs, realistic Terminators
> - Obvious implications for police and military work, advertising
> - Understanding nature vs nurture
> - Cross-cultural comparison
> - "Checkpointing", the system tells you what to do next
> - Intentionally diversifying by deviating from predictions
> - Long term resource planning
> - Magic shows
> - Engaging mass "gag reflex" re: privacy, free will, control...
> - Convenience (GPS sees you taking new route and asks where you're going,
> calendar autofills so you only have to input modifications)
> - Fraud detection, credit score, insurance rates
> There are obviously a lot of possibilities, but a lot of the more
> interesting ones involve inputing crafted data to test hypotheses, esp. in:
> - faking (to derive plausible back-stories)
> - decision making
> - planning via Monte Carlo and genetic algorithms
> - meeting scheduling (finding most useful common time automatically)
> - Using "trained" patterns (based on successful, etc.) for decision making
> - Better demand-oriented pricing in non-normal situations (where the usual
> history-based demand applies) using mass data
> - Derivation of historical patterns for archaeology
> - Product/location testing
> Is most of this interesting to individual consumers? Most of it could only
> be applied at "higher" levels, and thus individuals would only see benefits
> indirectly (and probably wouldn't notice the difference). Some of it is
> individually interesting in a novelty kind of way. Some of it could be
> genuinely useful in an individual-obvious way (for instance, in
> Almost all of it is potentially scary, but I don't see how it's preventable
> on a mass scale. I think it would be wise as a government acting as a
> self-interested entity to avoid being obvious in your usage of such tech,
> the paranoia caused by a true level of awareness could lead to
> unrest/subversion (much like I'm sure the relevant intelligence agencies
> have informatics running already that they're not advertising).
> As an individual, perhaps you can dodge detection, by avoiding sensors,
> understanding the tech, intentionally randomizing your activities (or
> yet, lots of others'). Of course, you then submit yourself to a life of
> randomness, because any pattern in a field of randomness makes itself
> obvious. There's still going to be potential for deception, but to take
> advantage will require deep knowledge of the algorithms involved, as well
> a severe amount of self-awareness.
> On the deep end, highly accurate pattern detection tech opens up the doors
> to a few holy grails of AI (though I think this is fairly obvious). The
> one is that it lets you have "real" language in an AI. Predicting actions
> requires a model of human actions; any really accurate model will likely
> contain a de facto model of the internal mind, where thought patterns would
> count as "actions" which could lead to real observable actions. Then you
> could construct an artificial human "mind" that uses models of other humans
> to understand context of speech. From those models, create another, but
> feed it the actions and internal states of the "mind" as input. This
> will predict the mind's own actions with even greater accuracy, giving rise
> to what seems to me to be consciousness. (Just have the mind do whatever
> the model predicts another person would do in the same situation.)
> Of course, that's assuming that the pattern prediction gets really good.
> It's no-shit obvious that peoples' macro movement patterns are predictable
> (as are eating, sex, sleep, and similar). But smaller/subtler/more
> actions, the writing and speech and product creation and capital
> raising--that is, the interesting stuff--are much less predictable,
> I don't doubt that they are predictable on some useful scale. I think this
> is what Ken was getting at with the "who cares"; but I think that even this
> more doldrum scale of prediction that we are already capable of has a fair
> few actual uses (including most of those listed above).
> Jebadiah Moore
> FoRK mailing list
More information about the FoRK