Treatment FAQ

"how conditioning on post-treatment variables can ruin your experiment and what to do about it"

by Juvenal Kovacek Published 3 years ago Updated 2 years ago

How does repetition affect belief?

Repetition increases belief in false statements. This illusory truth effect occurs with many different types of statements (e.g., trivia facts, news headlines, advertisements), and even occurs when the false statement contradicts participants’ prior knowledge. However, existing studies of the effect of prior knowledge on the illusory truth effect share a common flaw; they measure participants’ knowledge after the experimental manipulation and thus conditionalize responses on posttreatment variables. In the current study, we measure prior knowledge prior to the experimental manipulation and thus provide a cleaner measurement of the causal effect of repetition on belief. We again find that prior knowledge does not protect against the illusory truth effect. Repeated false statements were given higher truth ratings than novel statements, even when they contradicted participants’ prior knowledge.

What is experimental research?

Experimental researchers in political science frequently face the problem of inferring which of several treatment arms is most effective. They may also seek to estimate mean outcomes under that arm, construct confidence intervals, and test hypotheses. Ordinarily, multiarm trials conducted using static designs assign participants to each arm with fixed probabilities. However, a growing statistical literature suggests that adaptive experimental designs that dynamically allocate larger assignment probabilities to more promising treatments are better equipped to discover the best performing arm. Using simulations and empirical applications, we explore the conditions under which such designs hasten the discovery of superior treatments and improve the precision with which their effects are estimated. Recognizing that many scholars seek to assess performance relative to a control condition, we also develop and implement a novel adaptive algorithm that seeks to maximize the precision with which the largest treatment effect is estimated .

How does endorser effect literature?

The endorser effects literature expects the public to be swayed by cues offered by trusted or liked sources. A useful question for political marketing scholars is whether inherently nonpartizan cues impact public willingness to access nonpartisan political information. We use two social media-based field experiments and two survey-embedded experiments to test whether a randomly assigned visual marketing endorsement of political information by a known nonpartizan organization in paid ads (i.e., the League of Women Voters) encourages users to click on an information video about upcoming elections. We found an overwhelming subject response to the information video when the League of Women Voters (i.e., the endorser) cue is present, relative to the control group that received no cue in either the field or survey experiments (and controlling for partisanship and political interest in the survey experiments).

Is YCI a post-treatment covariate?

... If the YEOs reduce yakuza members, then the YCI may be affected by the YEOs and is considered as a post-treatment covariate, a variable that is affected by a treatment. From a causal inference standpoint, conditioning on a post-treatment covariate can induce post-treatment bias, which estimates treatment effects among a group, whose post-treatment covariate is not affected by treatment (e.g., Rosenbaum 1984; Montgomery et al. 2018). By defining the YCI in the said manner, we can avoid this problem. ...

Extract

Audit experiments are used to measure discrimination in a large number of domains (Employment: Bertrand et al. ( 2004 ); Legislator responsiveness: Butler et al. ( 2011 ); Housing: Fang et al. ( 2018 )).

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The data, code, and any additional materials required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network, at doi:10.7910/DVN/6NVI9C. I would like to thank Ariel White, Noah Nathan, Julie Faller, Saad Gulzar, and Peter Aronow for helpful comments.

Research

For a full list of all my publications and working papers, please see my CV.

Understanding visual messages: Visual framing and the Bag Of Visual Words

Political communication is a central element of several political dynamics. Its visual component is crucial in understanding the origin, characteristics and consequences of the messages sent between political figures, media and citizens. However, visual features have been largely overlooked in Political Science.

Estimating controlled direct effects through marginal structural models

The estimation of direct effects allows researchers to understand causal mechanisms. However, in political science research, the treatment often affects both the confounders of the mediator and the outcome. Under these conditions traditional regression methods typically lead to two types of biases: endogenous selection and post-treatment control.

Through the ideology of the beholder: partisan perceptions and polarization among the mass public

Scholars have documented increased polarization among contemporary mass publics but its causes and consequences are understood less well. In this paper, we study how citizens develop and apply perceptions of partisan groups.

Untouchable face

The second article occurred, as Marya Dmitriyevna (who is old-school) in Natasha, Pierre, and the Great Comet of 1812 shrieks after Anatole (who is hot) tries to elope with Natasha (who is young) , in my house. So I’m going to spend most of this post on that.

Once, twice, three times a lady

But last night, as the frat party next door tore through 2am (an unpleasant side-effect of faculty housing it seems) it popped up again on my facebook feed. This time it had grown an attachment ot that IBM story. I was pretty tired and thought “Right. I can blog on this”.

With one look

If you strip away all of the coverage, the paper itself does some things right. It has a detailed discussion of the limitations of the data and the method. (More later) It argues that, because facial features can’t be taught, these findings provide evidence towards the prenatal hormone theory of sexual orientation (ie that we’re “born this way”).

Behind the red door

Male facial image brightness correlates 0.19 with the probability of being gay, as estimated by the DNN-based classifier. While the brightness of the facial image might be driven by many factors, previous research found that testosterone stimulates melanocyte structure and function leading to a darker skin. (Footnote 6)

Back on top

But enough about methodology, let’s talk about data. Probably the biggest criticism that I can make of this paper is that they do not identify the source of their data. (Actually, in the interview with The Economist that originally announced the study, Kosinski says that this is intentional to “discourage copycats”.)

Pictures of you

First, we used the Facebook Audience Insights platform to identify 50 Facebook Pages most popular among gay men, including Pages such as: “I love being Gay”, “Manhunt”, “Gay and Fabulous”, and “Gay Times Magazine”.

Breaking the law

So what of the dystopian picture the authors paint of governments using this type of procedure to find and eliminate homosexuals? Meh.

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 1 2 3 4 5 6 7 8 9