But, like, isn't that being a bit 'cargo cult' about the fact the experiment was randomised?

If the control isn't otherwise similar enough to isolate the hypothesis that we actually want to test (e.g. 'art makes you smart'), then it might as well not be randomised, with respect to that specific hypothesis?

(I'm not commenting on the original research with which I'm not familiar - just on this specific point.)

> If the control isn't otherwise similar enough to isolate the hypothesis that we actually want to test (e.g. 'art makes you smart'), then it might as well not be randomised, with respect to that specific hypothesis?

No, the randomization still serves for causal inference, it just means the identified causes are a mix of potential causes like direct effect of art and expectancy effects. If one wants to make the claim that the net effect _d_=0.09 (IIRC) was purely the work of the art/tutoring and nothing else, that would only be partially supported by the results, yeah.

>If one wants to make the claim that the net effect _d_=0.09 (IIRC) was purely the work of the art/tutoring and nothing else, that would only be partially supported by the results, yeah.

To put that another way: If one wants to make the claim that the effect was actually casually related to the art/tutoring at all - as opposed to the effect of just going on a field trip, expectancy effects, etc. - then this experiment isn't useful.

Its interesting that you phrase it, though, as 'partially support by the results'.

Would you therefore say that the results of a non-randomised trial also 'partially support' the hypothesis under test?

Lets say that we didn't have a randomised assignment; that instead we simply identified existing populations whose parents had brought them on one outing last semester, and contrasted those who had visited an art museum, vs. those who had visited a science museum, and administered our tests to these two groups.

We could probably do some research with our data, but we'd always be worried that we couldn't account for selection-like biases, (e.g. maybe the kind of children who have parents that choose art vs. science museums are already smarter).

I understand that in the strictest bayesian sense, any data that doesn't could directly reject a hypothesis, but doesn't, 'partially supports' the hypothesis.

But, I'm not sure that, in practice, its meaningful to differentiate between which of the two methodological problems is the generally bigger or smaller issue.

That's where my comment is coming from.

If the hypothesis is as described in the news headline (which may not have being how the original study was motivated, which may be what you are more interested in, hence our disconnect), then isn't one problem as big as the other? Isn't [potential selection bias etc.] just as bad as [potential expectancy effects etc.]

Wouldn't it hence be a little misleading to say "well, yeah, they didnt control properly, but savour the fact they had random assignment"?

> I understand that in the strictest bayesian sense, any data that doesn't could directly reject a hypothesis, but doesn't, 'partially supports' the hypothesis. > > But, I'm not sure that, in practice, its meaningful to differentiate between which of the two methodological problems is the generally bigger or smaller issue.

I'd disagree there, you can do more than simply throw up your hands and say correlation!=causation. Some of the work stemming from Pearl's causality formalism has been about conditions under which one can make causal inference even in the absence of an explicit randomization step, and one can attack it from another direction by compiling correlations which have been later examined with randomization, and estimating how many of the correlations turned out to be causation in the same direction (the numbers tend to look like ~10%) and how much they are due to other factors.

> Isn't [potential selection bias etc.] just as bad as [potential expectancy effects etc.]

No. Expectancy effects can be manipulated and measured and one could arguably then adjust for it in other results where expectancy is mixed in. It's a fact about human psychologies, as measurable as anything else there. Selection effects are too wild and unpredictable to hope to do such a thing.