This is a follow-up on my previous post, “In which a skeptic accidentally proves astrology while trying to disprove it” where I discussed the naive and wrong-headed approach that a skeptic set up trying to “prove” astrology. Today I’d like to explain further why that test wouldn’t work. (And the third part of this series is here.)

It’s not that you can’t test astrology — you can — but you need to understand the basics of what you’re even working with first.

Just as a quick review of the case I’m referencing, the guy wanted to present astrologers with:

• an astrological chart, and

• a list of five people.

The astrologer would need to match which of the five people belongs to the chart presented. He wanted to do this 12 times. If the astrologer could get 10 out of 12 correct, they would pass the test.

I have seen several other very similar challenges thrown out there by skeptics in the past, claiming that if astrology were real, astrologers would be able to perform this kind of matching game.

I am, again, going to set aside all the reasons why this test is not even good *science*, because I want to focus on the astrology. But it is really tempting to rip into this guy’s very shallow and flawed understanding of basic research methods!

Here comes a tangent

May I just indulge myself and mention one thing, really quickly, though? Because it’s annoying. This guy claims to be a mathematician, but what the actual...!

He says someone would need to guess correctly 10 out of 12 times, with 5 options in each trial, to surpass random chance. My youngest child is in 7th grade, and I’ve been helping him with his math homework, and he’s learning about probability, and even though I didn’t major in a STEM field, I got A’s in math throughout school, and I can just tell you… Spencer’s number is not accurate! (I write run-on sentences when I get annoyed.)

If you don’t like math, feel free to jump to the next section. 😉

Each trial presents 5 options, with one correct answer. The chance of randomly guessing one correct answer from 5 options is one-fifth, or 0.2.

For every trial included in the test, the chance of an accurate random guess would be multiplied by 0.2. So if there were 5 trials, a random guesser would get 1 correct.

If there were 10 trials, the random guesser would get 2 correct. Spencer wanted to offer 12 trials, so the random guesser would get it right 2.4 times.

To beat random chance, the astrologer would need to beat that 2.4 number, but since there are no “partially correct” options in Spencer’s version of the trial, we will round up to 3. The astrologer would need to get more than 3 answers correct, not 10, to beat random chance. Why the heck did he say 10?

Ten correct answers would be over four times better than random chance.

What kind of standard are we setting up here, and why?

OK, enough math. Shake it off!

Moving on to the astrology.

What are we testing?

The test as presented seems similar to ones used for testing psychic abilities. For example, a common test for psychic abilities is that someone is presented 5 cards, and they need to pick which card has something on the other side.

Psi Arcade is a source of (actually good-faith) tests of psychic abilities, and I’ve been following them loosely for several years. (They used to be called “Got Psi?”) The card game was one of their first tests available, if I recall.

Spencer’s trial of astrology, and several other trials I have seen mentioned in the past, seem to be based on this card-match model. I’m not sure why they choose this model, when there are many others they could pick. But let’s talk about why a test for psychic ability is not a good match (pun not intended) for astrology. (I’m not even convinced that the card match test is a good match for psychic ability, itself, but that’s a different conversation.)

When you say you want to test a claim, you have to make sure your test is actually testing what the claim actually claims. So if I say “astrology does [X]” and you say, “great, let’s test astrology doing [Y]” you are not being logical. The test needs to match the claim. While astrology is a very large field, with a wide array of applications, choosing matches from a list of five options is rarely a task astrologers need to do. So if we are going to present them with something that is so out of context to their normal work, there needs to be an excellent reason.

So what is the claim we want to test, anyway? In the case I’m using as an example, sometimes Spencer implied that he wanted to test whether astrology was even valid or not. Other times he emphasized that this test was only a test of the astrologer’s individual skill per se, rather than a proof of the entire field as a whole. The fact that he was offering $1,000 for those who could pass the test made it seem more like an athletic competition, than a true test of scientific validity. In his previous attempts to test astrology, he didn’t involve astrologers at all. He compared results of personality tests with the Sun Signs of the participants, to see if there were correlations. He could have done the same kind of correlative test by including other planets, leaving the subjectivity of astrological interpretation out of the test altogether. Why did he suddenly jump to wanting to test the astrologers’ subjective skill levels, then?

I’m not trying to pick on Spencer alone here. A lot of tests devised by skeptics suffer from this ambiguity of purpose.

Let’s give the skeptics the assumption of good faith, though, and let’s say they truly just want to investigate whether astrology is valid or not. Let’s say we are working with people of integrity, and we successfully narrow the question to: “Do astrological charts correlate to personality traits and life events?” Is the card match test a good tool to answer this question? No, it is not.

The black-or-white fallacy

The matching test has a binary outcome—you either pick the card with a picture, or you pick a blank card. You either select the correct person or one of the incorrect people. There is no in-between. You are right or you are wrong.

But astrology is not binary; it is highly nuanced. Each chart has thousands of data points. People’s personalities are also not binary; they are also highly nuanced.

On top of that, people have a lot in common with each other. People are not as different from each other as a blank card is different from a card with an image. None of us are blank cards. To pull an example from the Big-5 model, every person involved in the test is going to fall somewhere on the Openness scale. It is quite likely that at least two of the options in the list will have similar Openness traits.

The way this applies to the astrology test is that an astrologer might look at the chart and pick the “wrong” person, who nonetheless has many similar characteristics as the “right” person. The person they select might be 80% similar to the person in the correct answer, but they still get a 0% score ❌ for that trial. That’s why this test does not reflect how astrology is practiced—it doesn’t allow for similar answers to receive partial credit.

One way to help demonstrate this point would be to change our hypothetical card test so that instead of four blank cards and one card with a picture, each of the 5 cards has a different picture on it. Let’s say the subject is told to pick the “correct” card, but they are not told which one is correct. The five cards have the following pictures:

• Cat (the correct card)

• Dachsund

• House

• Car

• Tiger

If the subject picks the tiger, they get that answer wrong ❌, even though a tiger is also in the feline family, with many similar physical features to a cat! If they pick the dachsund, they get the answer wrong ❌, even though a dachsund is also a pet with some similar features to a cat. If they pick the car… ❌. But wait, “Car” also starts with “C-A.” To write the test fairly, the wrong answers should be weighted somehow, based on how similar they are to the correct answer.

This is why a binary outcome test is a very poor test for astrology.

Well, I have several more reasons, but this post is long enough! I will write more reasons why the card-match model is poorly suited for astrology in my next post. Stay tuned!