Debriefing for ego threat may require more than we thought

Sad, from Flickr user Megadeth’s Girl, licensed under CC BY 2.0

Sad

When social psychologists manipulate a participant’s attitudes or beliefs, we have an ethical obligation to undo that manipulation. I explain it to my students as “putting the participant back the way we found them.”

We frequently use a debriefing procedure, in the form of a written and/or (as in the case of my lab) verbal notice something to the effect of “yuk yuk, gosh, ya know what? we were just kidding. the thing you (read/did) was fake, we made it up, and it doesn’t mean anything.”

Here is an example from the verbal debriefing script I used in a study several years ago that presented participants with a fake newspaper article about vandalism by University of Texas students.

I want to thank you for your participation here today and for your contribution to this project. We really appreciate your help with this work. Let me tell you a little bit about what we are trying to study.

First, we want to assure you that the incident you read about never happened on the campus. We created a fake newspaper article about it in order to better understand how people respond to these kinds of situations. To our knowledge, no University of Texas students have ever been involved in such an incident.

A new article in the Journal of Personality and Social Psychology (i.e., Miketta & Friese, 2019) casts doubt on the effectiveness of such debriefings, especially when they are done to undo the effects of ego manipulations—manipulations designed to change someone’s self-perception. These manipulations are usually referred to as false feedback, where participants engage in some task ostensibly to measure some personally-relevant property such as intelligence, ability, personality, etc. The false feedback is meant to manipulate the participants’ self-relevant thoughts.

“Several debriefing procedures failed to fully undo the aversive effects of ego-threatening experiences on participants’ well-being. Only an extensive process debriefing largely reestablished prestudy conditions. However, even then, negative affective after-effects persisted for at least 2 weeks. Restoring participants’ well-being after ego-threatening experiences appears to be more difficult than previously believed. The positive antidote needs to be strong. It remains for future research to continue identifying the necessary ingredients of such antidotes.” (Miketta & Friese, 2019, p. 25)

The authors of the article point out that prior research (i.e., Ross et al., 1975 & McFarland et al., 2007) has demonstrated that simple written or verbal debriefings like the one in my experiment above (what the authors refer to as Standard Outcome Debriefings) are largely ineffective at undoing false feedback results. This prior research showed that the only effective means was via a Revised Outcome Debriefing, which told participants that they got false feedback and, crucially, that the measure that was used was invalid and did not measure any underlying personal characteristics at all. This presumably kept participants from ruminating about the results.

This new article presents the results of six studies that looked at the effectiveness of debriefing methods specifically in the context of ego threat manipulations. They found in the first three studies that a written Revised Outcome Debriefing did not eliminate the adverse effects of the ego threat manipulation on self-esteem, self-perception, or mood. The detrimental effects persevered for at least several hours after leaving the experiment. The last three studies tested various supplements or changes in the debriefing, including verbal debriefing, and written debriefing with self-affirmation, and an elaborate 10-15 minute “sensitive, caring, emotionally warm, attentive, and considerate manner” (Miketta & Friese, 2019, p. 18). This elaborate debriefing required experimenters to be trained by a psychologist over the course of several weeks. a second elaborate debriefing was tested that added an explanation of the psychological basis of the perseverance of the mood manipulation effects and instructions to counteract that perseverance effect.

The only debriefing that fully eliminated the detriment to self-esteem and self-perception was the second elaborate one. None of the debriefings fully attenuated the effects of ego threat on mood.

Importantly, an internal meta-analysis of all six studies taken together showed none of the debriefings fully eliminated the effects of the ego threat.

From the standpoint of ethical treatment of our participants, we are always balancing risk and benefits. If the (potential or actual) benefits of the research to the participants and humanity at large are sufficiently large, we may be willing to accept more risks; if they are lesser, then we will accept less risk.

In the context of these findings, the authors point out that there may be little to no risk of the detrimental effects of the ego threat manipulation. That may be true at least for psychologically well individuals, and even for these individuals, it is not clear what harm they may suffer from the ego threat manipulation aside from what is indicated by these measures of self-esteem and self-perception and mood. Further, what level of detrimental effects would cause a clinically significant effect in individuals?

In addition, these detrimental effects may have other, unknown or unrecognized downstream effects. Perhaps the self-perception or self-esteem reduction might cause an individual to underperform in a job interview and lose a career opportunity. The participant might not recognize this outcome as resulting from the ego threat manipulation, and so would not report it as an adverse effect to the experimenter or the IRB.

So, what we know is that our written, and even verbal, debriefing procedures may reduce but not fully eliminate the effects of our manipulations, at least in the case of ego threat. But we do not know how much of an adverse effect the failure to eliminate the effect might have on participants, and so can’t say definitively that they do or don’t experience risk from the manipulation that is minimal: “not greater in and of themselves than those ordinarily encountered in daily life,” as defined in the federal regulations 45 CFR 46.102(j). It would seem to be very individual and contextual. As usual, more research is necessary to better understand the potential persistence of manipulation effects despite debriefings.

IRB committees can consider this study a cautionary tale, but I think there is not yet enough evidence that debriefings don’t work to undo the effects of a psychological manipulation. I think we can be a little more comfortable with more elaborate verbal debriefings, that include explanations of how effects of the experiment might persist and information about how to counteract such persistence, than we might be with simple written debriefings. But even that conclusion is not fully supported by evidence. We still need to make a subjective judgment call on these situations.

McFarland, C., Cheam, A., & Buehler, R. (2007). The perseverance effect in the debriefing paradigm: Replication and extension. Journal of Experimental Social Psychology, 43, 233–240. http://dx.doi.org/10.1016/j.jesp.2006.01.010

Miketta, S., & Friese, M. (2019). Debriefed but still troubled? About the (in)effectiveness of postexperimental debriefings after ego threat. Journal of Personality and Social Psychology. Advance online publication.
http://dx.doi.org/10.1037/pspa0000155

Ross, L., Lepper, M. R., & Hubbard, M. (1975). Perseverance in self-perception and social perception: Biased attributional processes in the debriefing paradigm. Journal of Personality and Social Psychology, 32, 880–892. http://dx.doi.org/10.1037/0022-3514.32.5.880

New Preprint on OSF

My collaborators and I have completed a new manuscript that is close to in-press for the Psi Chi Journal of Psychological Research. It is a meta-analysis of several replication attempts of a highly-cited study about how self-esteem is related to Facebook usage motivations and perceptions. We found that the replications clearly replicated two of the four findings of the original article, failed to replicate one of them, and there was mixed evidence for the other finding. This tells us a bit more about how robust the original study’s findings were, and helps people who are interested to begin looking for moderators that we did not study.

The process was great because I had four great co-authors and we all worked together to get it done and out the door. I learned how to do meta-analysis in the process! We earned journal badges for Open Data, Open Materials, Preregistration, and Replication.

The prepress manuscript is on PsyArXiv at https://psyarxiv.com/sx742/

Leighton, D. C., Legate, N., LePine, S., Anderson, S. F., & Grahe, J. E. (2019, January 1). Self-Esteem, Self-Disclosure, Self-Expression, and Connection on Facebook: A Collaborative Replication Meta-Analysis. Retrieved from psyarxiv.com/sx742

The original article by Forest & Wood (2012)  is at Psychological Science: http://journals.sagepub.com/doi/abs/10.1177/0956797611429709

 

Why Do So Many Studies Fail to Replicate? – The New York Times

Over the last two semesters, our lab, the Peace and Justice Psychology Lab, has tried to replicate a finding from a 2007 article (Sommers & Norton, 2007) that showed a racial bias in excluding jurors during jury selection. They showed that black potential jurors were struck from jury selection more than white jurors, a finding that generalized from college students to law students and to attorneys. This finding matched the findings of a report issued last year that documented race bias in jury selection in Louisiana over the last decade.

We contacted Dr. Sommers to ask about his methods, and had a good conversation with him. One of the comments he made was that we needed to pilot test our juror profiles extensively to find ones that were effective in making the jurors undesirable. We did that (but only tested four different profiles). We asked him for the images he used in his study, but it was a long time ago and he couldn’t find his exact materials.

So, we set about selecting juror images from a database of facial images – one black and one white – that were matched on gender, age, expression, attractiveness, etc. We settled on two images of middle-aged women.

The replication failed spectacularly: There was no difference between the rates of striking black and white jurors. If we want to interpret our finding optimistically, we might say that students at Southern Arkansas University are impervious to racial bias. Realistically, I believe the replication failed because the materials were not identical. That tells us that the effect has something to do with factors other than race. For example, it is possible that Sommers & Norton used images that were not matched on attractiveness, and the black juror’s image simply was less attractive. Sommers used male juror profiles and images, and there is a pervasive (replicated) effect that black men are typically viewed as more threatening than white men by research subjects, which may not have affected our participants’ impressions of the female jurors. If we replicated with male jurors, we might get the same effect.

The one finding we did get was that people high in Social Dominance Orientation (SDO) (which is a preference for hierarchical power structures, traditional social roles, and status quo among other things) struck the jurors (of both races) at a higher rate than those low in SDO. I believe that this reflects a gender role preference consistent with SDO. To wit, the jurors were women, middle aged, with no children, who had advanced degrees and a job as journalist. This may violate the gender roles that our culture typically assigns to women as homemakers, child bearers, etc., and those high in SDO might be more sensitive to this violation of social roles and want to “punish” the women. We have not directly tested this hypothesis, and I am not sure we will try.

I was motivated to blog about this because the social psychologist Jay Van Bavel wrote an opinion piece for this Sunday’s New York Times about the replication crisis. He addresses the difficulty with conducting direct replications of previous findings in social psychology because of the effect of context: there are lots of factors going on in the social environment that may be related to whether a replication fails or succeeds. But, I think that helps us understand that the effect is not as robust as only one, unreplicated, research finding might suggest.

Because it is hard to recreate the exact conditions of the original research.

Source: Why Do So Many Studies Fail to Replicate? – The New York Times

Sommers, S. R., & Norton, M. I. (2007). Race-based judgments, race-neutral justifications: Experimental examination of peremptory use and the Batson challenge procedure. Law and Human Behavior, 31, 261–273.

Threat and Prejudice and Moral Exclusion, Oh My!

This is why I do the research I do on the power of perceived threat in prejudice and moral exclusion:

Ken Knight, 54, a heating and cooling technician from Florence, S.C., said he agrees with Bush and Cruz that only Christian refugees should be let into the country.”I wouldnt bring the Muslims. They cut your head off. You cant trust them. Im sure there are good ones, but theyre like the mob. Once you get in, you cant trust them,” he said after a church service

I want to understand that strange thought process.

And, of course, Christians and terrorists are mutually exclusive, aren’t they?

Link to the article: Cruz: ‘No meaningful risk’ of Christians committing terrorism – The Washington Post.

About the new Arkansas law potentiating discrimination based on religious beliefs

I don’t usually blog about political issues, but this one is related to my research on prejudice, discrimination, and moral exclusion from the scope of justice.

It has been an interesting week here in Arkansas. The state legislature, not known to be a bastion of acceptance toward difference from a Christian, Western, heterosexist, capitalist worldview, has now passed the law HB 1288, deceptively titled “the religious freedom restoration act,” also known as “Mary’s law.” You can read the (alarmingly brief) bill here.

Here’s what the law says: Any governmental entity in the state (e.g., state, county, city) can’t enforce laws that, even if “neutral” toward religion, “burdens” the exercise of an individual’s religious beliefs (even if those beliefs are not compulsory or central to the religion). Interestingly, prisons are exempted from the law. By “burden” the law means “to prevent, inhibit, or curtail religiously motivated practice consistent with a sincerely held religious belief.” There is an exemption for laws that are “essential” for a “compelling governmental interest” AND are the “least restrictive” to achieve that interest.

So what does this mean to most of us? Likely not much. But, what does it mean to some of us, who are among minority populations? Maybe a lot.

The major issue raised has been that of religions that denegrate and demonize the behavior of alternative sexualities: gay, lesbian, transgender. So, for example, if I was a Christian who owned the only event hall in town, I might be able to refuse to rent the space for a wedding between same-sex people. Similarly, if I was a radical fundamentalist Muslim who owned that event hall, I may prohibit rental to Christians as “infidels.” Either way, that would mean no reception unless I was willing to travel to another town.

A more insidious outcome might be the establishment of separate facilities and businesses that only catered to people who didn’t violate the owners’ “sincerely held religious beliefs” as the law calls them.

On a more practical level, the law moves Arkansas away from the growth economies that we need to supplement agriculture and mining. Companies who need to attract the best and brightest talent from around the country can’t move to a state that codifies laws that are regressive toward civil rights. A good argument about this is made by the former head of the Arkansas Economic Development Commission. Here is a link to a powerful open letter from him about these effects:

Former AEDC Director Grant Tennille: “We are watching a re-run.” | Arkansas Blog | Arkansas news, politics, opinion, restaurants, music, movies and art.

His letter is bolstered by the fact that many of the largest corporations in the country, including Apple, have called on Arkansas not to pass the bill. It is telling that even the CEO of Walmart, based here in Arkansas, and the nation’s largest employer, has denounced the bill as it “sends the wrong message about Arkansas.”

Well, we will know in a few hours if the governor will endorse a potentially economically damaging piece of legislation. Not to mention one that further tells people in Arkansas with alternative sexualities that their right to the pursuit of happiness is not worth protecting.

EDIT: The governor just announced that he wants the legistalive branch to recall the legislation and make changes to bring it in line with the widely-accepted federal legislation. That is a good step. He wants us to make clear to the world that Arkansas is a good place to do business. Sadly, I think much of the damage has been done. But, he did not explicitly veto it, perhaps to avoid an override vote. If the legislature thumbs their noses at him, there is not much he can do save for vetoing and risking an override, or not sign it in which case it becomes law 20 days after the legislature adjourns.

Getting involved in research as an undergrad

SPSP (Society for Personality and Social Psychology) published a brief primer on getting involved in research as an undergraduate. It’s worth checking out:

Link to the article at SPSP.org.

Excellent NYT Magazine piece on Stapel and his fraud

The New York Times Magazine has an excellent piece on Deiderik Stapel and his fraud (posted earlier here and here and here and here and here and here and here). It chronicles the days leading up to the accusation, his family and childhood. One interesting piece of local trivia: He briefly attended East Stroudsburg University to study acting.

Here is a link to the article at the NY Times Magazine: link

A great quote:

He insisted that he loved social psychology but had been frustrated by the messiness of experimental data, which rarely led to clear conclusions. His lifelong obsession with elegance and order, he said, led him to concoct sexy results that journals found attractive. “It was a quest for aesthetics, for beauty — instead of the truth,” he said.

and in the exposé of the first of his frauds…

In one experiment conducted with undergraduates recruited from his class, Stapel asked subjects to rate their individual attractiveness after they were flashed an image of either an attractive female face or a very unattractive one. The hypothesis was that subjects exposed to the attractive image would — through an automatic comparison — rate themselves as less attractive than subjects exposed to the other image.

The experiment — and others like it — didn’t give Stapel the desired results, he said. He had the choice of abandoning the work or redoing the experiment. But he had already spent a lot of time on the research and was convinced his hypothesis was valid. “I said — you know what, I am going to create the data set,” he told me.

Sitting at his kitchen table in Groningen, he began typing numbers into his laptop that would give him the outcome he wanted. …Stapel at first ended up getting a bigger difference between the two conditions than was ideal. He went back and tweaked the numbers again. It took a few hours of trial and error, spread out over a few days, to get the data just right.

He said he felt both terrible and relieved. The results were published in The Journal of Personality and Social Psychology in 2004. “I realized — hey, we can do this,” he told me.

How to woo a scientist

The Guardian (UK) has a hilarious article about how to find and pluck the heart strings of your nearest scientist. Here’s the link.

My favorite quotes:

“Scientists can be hard to locate. They rarely frequent sporting events, popular music concerts, fairgrounds, organised cockfights or wherever it is non-scientists choose to congregate. A typical scientist is usually found in the laboratory.”

“There are instances where you will encounter a scientist outside of the laboratory environment. They may be giving a lecture, or possibly standing in an exotic location looking wistful. In both of these instances, engaging in conversation is impractical, given the context. … If you’re lucky, you may encounter one in a pub or similar establishment. … if you see someone who is clearly under the influence of alcohol but still using words of 5 syllables or more, then they’re likely to be a scientist.”

“When attempting to talk to a scientist, be sure you don’t say anything that might be interpreted as a claim unless you are certain it has been peer-reviewed or subjected to rigorous statistical assessment.”

“Should the conversation falter or hit a lull, try asking the question “How is your grant application going?” This is likely to result in a very long rant about the problems, frustrations and possible illegitimate birth origins of those involved with the grant approval process.”

SPSP comments on the final Levelt report on the Deiderik Stapel debacle

The Society for Personality and Social Psychology just released a statement on the Levelt report. It is not (yet) on their web site, so I will reproduce it here. It arrived on the SPSP listserv. I am glad we are undertaking an assessment of where we have gone wrong and how our research practices can be improved to help avoid something like this in the future.

Society for Personality and Social Psychology Statement on the Levelt Report

The recent Levelt report from the Netherlands details the breadth of Deiderik Stapel’s fraudulent activities and offers reflections on the scientific culture that enabled this magnitude of deception to go (nearly) undiscovered for many years. It is a sobering read. Both the European Association of Social Psychology and the Society of Experimental Social Psychology have issued statements on the report. These statements expressed appreciation to the Levelt committee for its thorough investigation of the Stapel case and indicated reservations about the report’s indictment of the field at large for what the Levelt report considered “sloppy science” practices. In large measure, we support these previous statements, and our statement is meant to complement rather than reiterate the points made by our allied societies.

In the aftermath of the Stapel case and other recently discovered cases of fraud, it would behoove us to reflect on the steps we can take as we move forward to protect our science against incidents of fraud in the future and to repair the image of our science. The core foundation of any field of scientific endeavor is trust and integrity.

The Society for Personality and Social Psychology has consistently maintained the stance that we must work together as a professional organization and as individual scientists to promote a context in which good scientific practices are celebrated and are embedded into the training we provide to young scientists joining the field. Indeed, this was the theme of a recent letter I wrote to the Society regarding the Stapel case. It is a theme that merits repeating. These recent events provide an opportunity not only for constructive reflection but also specific action. Upholding sound scientific practices will insure that our science has integrity. We should not assume, however, that because we all believe in the principles of ethical conduct that this is sufficient. In this regard, we can all profitably discuss ways to accomplish these goals. These goals should include, but not be limited to:

  • Identifying effective ways to build discussion of ethics and good scientific practices into our course work and everyday discussions in our laboratories;
  • Developing safe venues for trainees and others to report concerns about breaches of ethics within universities and within the journal review process;
  • Establishing clear standards for what personality and social psychology papers should present in methods and results sections of articles;
  • Providing formal training in how to review articles;
  • Clarifying within our formal training acceptable practices for addressing, for example, missing data, eliminating cases from analysis, and providing clear detail on methods and measures;
  • Increasing opportunities and incentives for conducting and reporting direct replications of important findings; and
  • Evaluating the pressures that can lead to a careerist focus as opposed to a focus on true discovery among scientists.

These are but a few of the issues we, and all sciences, need to consider. Recent months have borne witness to a number of activities designed to address these issues. For example: replication issues have been the subject of a recent special issue of Perspectives on Psychological Science (http://pps.sagepub.com/content/7/6.toc), a forthcoming special target article in the European Journal of Personality, and recent issues of our own Dialogue (see http://tinyurl.com/auzkr73). Brian Nosek and Daniel Lakens are co‐editing a special issue on replication of important findings in social psychological research in Social Psychology (see http://tinyurl.com/asujp7s). Finally, SPSP commissioned a Task Force for Responsible Conduct, which outlined a variety of ways we could take positive steps to ensure the integrity of our science (https://www.spsp.org/?ResponsibleConduct) and the Task Force continues to work on these issues.

Our upcoming meeting in New Orleans provides a number of immediate opportunities to explore and discuss these issues with our community. Two formal symposia address issues related to good scientific practices.

The first is titled “Openness in Scientific Reporting: Potential and Reaction” and is scheduled for Friday, January 18 from 11:15 am to 12:30 pm (Rooms R03‐R05).

The second symposium is titled “False Positive Findings: Effect Sizes Too Large, Too Small, or Just Right” and is scheduled for Friday, January 18 from 3:30 pm to 4:45 pm (Rooms R03‐R05).

Finally, I suggested to you in a recent letter that the leadership of SPSP was likely to hold a Special session for the membership to come together to discuss these issues or any issues of interest to the membership. We have scheduled this session for Saturday, January 19 from 3:30 pm to 4:45 pm Room 203‐205. In attendance will be David Funder who is the 2013 President of the Society, Jenny Crocker who chaired the SPSP Task Force on Responsible Conduct, Jack Dovidio the current Executive Officer for the Society, Jamie Pennebaker the new President‐Elect of the Society, and me. We invite you to come to this session with questions and we will do our best to address these and other issues relevant to the Society and engage the membership in a productive discussion.

Yours Sincerely,

Patricia G. Devine for the SPSP Executive Committee

Past-­‐President, Society for Personality and Social Psychology