A Torah scroll. Photo: RabbiSacks.org.

Creating accurate data driven survey results may sound easy. Just ask some questions and get some answers. However, there are many things that can go wrong in a survey, which can nullify results. These include issues related to sample size and population, survey questions, and data analytics.

The sample, or group of people selected to respond to the survey, is significant.

For example, consider a hypothetical study that takes place on how many Hassidic folks live in Borough Park, Brooklyn. The researcher decides to survey the first 100 people on a given street corner. Everyone responds in the negative and he concludes that Borough Park has no Hassidic Jews. However, unbeknownst to the researcher, on that particular street corner, there is a non-Hassidic (Litvish) synagogue.

Something similar happened in the recent study from a company called Resume Builder, about hiring discrimination towards Jewish people. The sample was selected by incentivizing the survey through perks on dating apps. This attracted a specific age group and skewed towards younger hiring managers, which is not representative of a general hiring manager demographic.

Adding to this, one of the more difficult aspects of research is creating survey questions that are completely objective and do not lead towards any one response.

For example, imagine a witness in a court case is asked the following question: How badly was the car smashed? The response to that question would differ greatly from the response to this one: What was the condition of the car? Even reading these questions might conjure up images: an image of destruction for the first, a neutral image for the second.

In the Resume Builder survey, the polling company, Pollfish, allowed their clients to create their own questions with a survey do-it-yourself tool. Here are some of them:

“Why are you less likely to move forward with Jewish applicants?” Suggested answers were “Jews have too much wealth,” “Jews have too much power and control,” “Jews killed Jesus,” “Jews are an inferior race,” “Jews are greedy,” “Jews are oppressors,” “Jews are less capable,” “Jews claim to be the ‘chosen people,’” and “Other.”

Another question asks, “Why do you believe your industry should have fewer Jewish employees?” Response choices given were the same as those listed above. Survey questions like this can inadvertently create more antisemitism by suggesting answers that the hiring manager may not have thought of on their own

Data analytics, the science of analyzing data and coming to accurate conclusions about the data, is another critical area to consider.

For example, think about a researcher who is trying to identify the side effects of a medication. He puts together a group of 100 people, and many have terrible gastric issues. He then concludes that the medication has terrible gastric side effects. However, what if the medication makes the patient more sensitive to certain foods? What if the medication has a bad interaction with other medications? What if it can’t be taken on an empty stomach?

Data needs to be handled by trained scientists who know how to determine why they are getting certain results when they do a study.

In the Resume Builder survey, clients created their own analytics by counting percentages. They were not taught or trained to ask the right questions, calculate what other variables might skew the results, or employ any other sort of analytics that data analysts perform with specialized software to give consumers accurate information.

Data driven research needs to be carefully evaluated before it can be trusted. The Resume Builder survey is troubling but has too many flaws, and, therefore no conclusive information can be derived from it. The next time you read a data driven article, you might want to question: Who did this research? Are they qualified? Did they have a skilled scientist to analyze results? The lesson? Buyer beware.

Dr. Alexandra Chana Fishman is a Research Manager at StandWithUs, an education organization that supports Israel and fights antisemitism. She is proficient in data and analytics and uses this expertise to create original surveys with data driven results.