[FoRK] Parts Zero and Minus One: Happy Death Spirals

[Jeff Bone]

(Chasing the thread backwards re: Eli's parallel argument...)

   http://www.overcomingbias.com/2007/12/resist-the-happ.html

   Resist the Happy Death Spiral
Followup to:  Affective Death Spirals

Once upon a time, there was a man who was convinced that he possessed  
a Great Idea.  Indeed, as the man thought upon the Great Idea more  
and more, he realized that it was not just a great idea, but the most  
wonderful idea ever. The Great Idea would unravel the mysteries of  
the universe, supersede the authority of the corrupt and error-ridden  
Establishment, confer nigh-magical powers upon its wielders, feed the  
hungry, heal the sick, make the whole world a better place, etc. etc.  
etc.

The man was Francis Bacon, his Great Idea was the scientific method,  
and he was the only crackpot in all history to claim that level of  
benefit to humanity and turn out to be completely right.

(Bacon didn't singlehandedly invent science, of course, but he did  
contribute, and may have been the first to realize the power.)

That's the problem with deciding that you'll never admire anything  
that much:  Some ideas really are that good.  Though no one has  
fulfilled claims more audacious than Bacon's; at least, not yet.

But then how can we resist the happy death spiral with respect to  
Science itself?  The happy death spiral starts when you believe  
something is so wonderful that the halo effect leads you to find more  
and more nice things to say about it, making you see it as even more  
wonderful, and so on, spiraling up into the abyss.  What if Science  
is in fact so beneficial that we cannot acknowledge its true glory  
and retain our sanity?  Sounds like a nice thing to say, doesn't it?   
Oh no it's starting ruuunnnnn...

If you retrieve the standard cached deep wisdom for don't go  
overboard on admiring science, you will find thoughts like "Science  
gave us air conditioning, but it also made the hydrogen bomb" or  
"Science can tell us about stars and biology, but it can never prove  
or disprove the dragon in my garage."  But the people who originated  
such thoughts were not trying to resist a happy death spiral.  They  
weren't worrying about their own admiration of science spinning out  
of control.  Probably they didn't like something science had to say  
about their pet beliefs, and sought ways to undermine its authority.

The standard negative things to say about science, aren't likely to  
appeal to someone who genuinely feels the exultation of science -  
that's not the intended audience.  So we'll have to search for other  
negative things to say instead.

But if you look selectively for something negative to say about  
science - even in an attempt to resist a happy death spiral - do you  
not automatically convict yourself of rationalization? Why would you  
pay attention to your own thoughts, if you knew you were trying to  
manipulate yourself?

I am generally skeptical of people who claim that one bias can be  
used to counteract another.  It sounds to me like an automobile  
mechanic who says that the motor is broken on your right windshield  
wiper, but instead of fixing it, they'll just break your left  
windshield wiper to balance things out.  This is the sort of  
cleverness that leads to shooting yourself in the foot.  Whatever the  
solution, it ought to involve believing true things, rather than  
believing you believe things that you believe are false.

Can you prevent the happy death spiral by restricting your admiration  
of Science to a narrow domain?  Part of the happy death spiral is  
seeing the Great Idea everywhere - thinking about how Communism could  
cure cancer if it was only given a chance.  Probably the single most  
reliable sign of a cult guru is that the guru claims expertise, not  
in one area, not even in a cluster of related areas, but in  
everything.  The guru knows what cult members should eat, wear, do  
for a living; who they should have sex with; which art they should  
look at; which music they should listen to...

Unfortunately for this plan, most people fail miserably when they try  
to describe the neat little box that science has to stay inside.  The  
usual trick, "Hey, science won't cure cancer" isn't going to fly.   
"Science has nothing to say about a parent's love for their child" -  
sorry, that's simply false.  If you try to sever science from e.g.  
parental love, you aren't just denying cognitive science and  
evolutionary psychology.  You're also denying Martine Rothblatt's  
founding of United Therapeutics to seek a cure for her daughter's  
pulmonary hypertension.  (Successfully, I might add.)  Science is  
legitimately related, one way or another, to just about every  
important facet of human existence.

All right, so what's an example of a false nice claim you could make  
about science?

In my humble opinion, one false claim is that science is so wonderful  
that scientists shouldn't even try to take ethical responsibility for  
their work, it will automatically end well.  This claim, to me, seems  
to misunderstand the nature of the process whereby science benefits  
humanity.  Scientists are human, they have prosocial concerns just  
like most other other people, and this is at least part of why  
science ends up doing more good than evil.

But that point is, evidently, not beyond dispute.  So here's a  
simpler false nice claim:  "A cancer patient can be cured just by  
publishing enough journal papers."  Or, "Sociopaths could become  
fully normal, if they just committed themselves to never believing  
anything without replicated experimental evidence with p<0.05."

The way to avoid believing such statements isn't an affective cap,  
deciding that science is only slightly nice.  Nor searching for  
reasons to believe that publishing journal papers causes cancer.  Nor  
believing that science has nothing to say about cancer one way or the  
other.

Rather, if you know with enough specificity how science works, then  
you know that, while it may be possible for "science to cure cancer",  
a cancer patient writing journal papers isn't going to experience a  
miraculous remission.  That specific proposed chain of cause and  
effect is not going to work out.

The happy death spiral is only an emotional problem because of a  
perceptual problem, the halo effect, which makes us more likely to  
accept future positive claims once we've accepted an initial positive  
claim.  We can't get rid of this effect just by wishing; it will  
probably always influence us a little.  But we can manage to slow  
down, stop, consider each additional nice claim as an additional  
burdensome detail, and focus on the specific points of the claim  
apart from its positiveness.

What if a specific nice claim "can't be disproven" but there are  
arguments "both for and against" it?  Actually these are words to be  
wary of in general, because often this is what people say when  
they're rehearsing the evidence or avoiding the real weak points.   
Given the danger of the happy death spiral, it makes sense to try to  
avoid being happy about unsettled claims - to avoid making them into  
a source of yet more positive affect about something you liked already.

The happy death spiral is only a big emotional problem because of the  
overly positive feedback, the ability for the process to go  
critical.  You may not be able to eliminate the halo effect entirely,  
but you can apply enough critical reasoning to keep the halos  
subcritical - make sure that the resonance dies out rather than  
exploding.

You might even say that the whole problem starts with people not  
bothering to critically examine every additional burdensome detail -  
demanding sufficient evidence to compensate for complexity, searching  
for flaws as well as support, invoking curiosity - once they've  
accepted some core premise.  Without the conjunction fallacy, there  
might still be a halo effect, but there wouldn't be a happy death  
spiral.

Even on the nicest Nice Thingies in the known universe, a perfect  
rationalist who demanded exactly the necessary evidence for every  
additional (positive) claim, would experience no affective resonance.  
You can't do this, but you can stay close enough to rational to keep  
your happiness from spiraling out of control.

The really dangerous cases are the ones where any criticism of any  
positive claim about the Great Thingy feels bad or is socially  
unacceptable.  Arguments are soldiers, any positive claim is a  
soldier on our side, stabbing your soldiers in the back is treason.   
Then the chain reaction goes supercritical.  More on this tomorrow.

Addendum:  Stuart Armstrong gives closely related advice:

Cut up your Great Thingy into smaller independent ideas, and treat  
them as independent.

For instance a marxist would cut up Marx's Great Thingy into a theory  
of value of labour, a theory of the political relations between  
classes, a theory of wages, a theory on the ultimate political state  
of mankind. Then each of them should be assessed independently, and  
the truth or falsity of one should not halo on the others. If we can  
do that, we should be safe from the spiral, as each theory is too  
narrow to start a spiral on its own.

This, metaphorically, is like keeping subcritical masses of plutonium  
from coming together.  Three Great Ideas are far less likely to drive  
you mad than one Great Idea.  Armstrong's advice also helps promote  
specificity:  As soon as someone says, "Publishing enough papers can  
cure your cancer," you ask, "Is that a benefit of the experimental  
method, and if so, at which stage of the experimental process is the  
cancer cured?  Or is it a benefit of science as a social process, and  
if so, does it rely on individual scientists wanting to cure cancer,  
or can they be self-interested?"  Hopefully this leads you away from  
the good or bad feeling, and toward noticing the confusion and lack  
of support.

Addendum 2:  To summarize, you do avoid a Happy Death Spiral by (1)  
splitting the Great Idea into parts (2) treating every additional  
detail as burdensome (3) thinking about the specifics of the causal  
chain instead of the good or bad feelings (4) not rehearsing evidence  
(5) not adding happiness from claims that "you can't prove are  
wrong"; but not by (6) refusing to admire anything too much (7)  
conducting a biased search for negative points until you feel unhappy  
again (8) forcibly shoving an idea into a safe box.

Posted by Eliezer Yudkowsky at 08:15 PM in Morality, Psychology |  
Permalink

   http://www.overcomingbias.com/2007/12/affective-death.html



Affective Death Spirals

Followup to:  The Affect Heuristic, The Halo Effect

Many, many, many are the flaws in human reasoning which lead us to  
overestimate how well our beloved theory explains the facts.  The  
phlogiston theory of chemistry could explain just about anything, so  
long as it didn't have to predict it in advance.  And the more  
phenomena you use your favored theory to explain, the truer your  
favored theory seems - has it not been confirmed by these many  
observations?  As the theory seems truer, you will be more likely to  
question evidence that conflicts with it.  As the favored theory  
seems more general, you will seek to use it in more explanations.

If you know anyone who believes that Belgium secretly controls the US  
banking system, or that they can use an invisible blue spirit force  
to detect available parking spaces, that's probably how they got  
started.

(Just keep an eye out, and you'll observe much that seems to confirm  
this theory...)

This positive feedback cycle of credulity and confirmation is indeed  
fearsome, and responsible for much error, both in science and in  
everyday life.

But it's nothing compared to the death spiral that begins with a  
charge of positive affect - a thought that feels really good.

A new political system that can save the world.  A great leader,  
strong and noble and wise.  An amazing tonic that can cure upset  
stomachs and cancer.

Heck, why not go for all three?  A great cause needs a great leader.   
A great leader should be able to brew up a magical tonic or two.

The halo effect is that any perceived positive characteristic (such  
as attractiveness or strength) increases perception of any other  
positive characteristic (such as intelligence or courage).  Even when  
it makes no sense, or less than no sense.

Positive characteristics enhance perception of every other positive  
characteristic?  That sounds a lot like how a fissioning uranium atom  
sends out neutrons that fission other uranium atoms.

Weak positive affect is subcritical; it doesn't spiral out of  
control.  An attractive person seems more honest, which, perhaps,  
makes them seem more attractive; but the effective neutron  
multiplication factor is less than 1.  Metaphorically speaking.  The  
resonance confuses things a little, but then dies out.

With intense positive affect attached to the Great Thingy, the  
resonance touches everywhere.  A believing Communist sees the wisdom  
of Marx in every hamburger bought at McDonalds; in every promotion  
they're denied that would have gone to them in a true worker's  
paradise; in every election that doesn't go to their taste, in every  
newspaper article "slanted in the wrong direction".  Every time they  
use the Great Idea to interpret another event, the Great Idea is  
confirmed all the more.  It feels better - positive reinforcement -  
and of course, when something feels good, that, alas, makes us want  
to believe it all the more.

When the Great Thingy feels good enough to make you seek out new  
opportunities to feel even better about the Great Thingy, applying it  
to interpret new events every day, the resonance of positive affect  
is like a chamber full of mousetraps loaded with ping-pong balls.

You could call it a "happy attractor", "overly positive feedback", a  
"praise locked loop", or "funpaper".  Personally I prefer the term  
"affective death spiral".

Coming tomorrow:  How to resist an affective death spiral.  (Hint:   
It's not by refusing to ever admire anything again, nor by keeping  
the things you admire in safe little restricted magisteria.)

Posted by Eliezer Yudkowsky at 11:44 AM in Morality, Overconfidence,  
Politics, Psychology | Permalink