With polls of U.S. Jews on Iran at odds, which to believe?

There have been three major polls of American Jews since the announcement of the nuclear deal between world powers and Iran — by the L.A. Jewish Journal, The Israel Project and J Street— all with significantly different results.

Two show U.S. Jewish support for the Iran nuclear deal outpacing opposition to the deal by 20 percentage points: 48-28 according to the Jewish Journal, and 60-40 according to J Street. The Israel Project survey showed slightly more opposition to the deal than support for it: 47 percent against versus 44 percent in favor. (By contrast, in a CNN poll this week of non-Jewish Americans, 52 percent said Congress should nix the deal, compared to 44 percent who wanted it approved).

With these wildly divergent results, which poll should we believe?

A close look at each poll’s questions, methodologies and even sponsors quickly makes clear which of the surveys attempt to gauge public opinion and which are meant to influence the public. It’s the difference between honestly trying to find out what people think versus trying to produce a certain result in order to bolster a particular argument.

L.A. Jewish Journal

Note how the three key survey questions below are phrased. The formulations are neutral in their wording, and the interviewer is instructed to rotate the “good idea/bad idea” choices in the last question to ensure a bias-free result.

“As you may know, an agreement was reached in which the United States and other countries would lift major economic sanctions against Iran in exchange for Iran restricting its nuclear program in a way that makes it harder for it to produce nuclear weapons.”

Support the deal: 47.8 percent
Oppose the deal: 27.6 percent
Don’t know: 24.6 percent

“Should Congress vote to approve or oppose the deal?”

Approve: 53.6 percent
Oppose: 34.7 percent
Don’t know: 11.7 percent

[Rotate verbiage in parentheses] “In retrospect, was it a (good idea) or a (bad idea) for the U.S. to conduct negotiations with Iran, or are you not sure whether it was a good idea or a bad idea?”

Good idea: 57.9 percent
Bad idea: 18.2 percent
Not sure: 23.9 percent

The Jewish Journal used the polling research firm SSRS Omnibus, which conducts national, weekly telephone surveys on a variety of subjects. For this survey, SSRS called only respondents in its database who had identified themselves as Jewish in previous unrelated surveys.

Demographer Steven M. Cohen of Hebrew Union College-Jewish Institute of Religion, who does all kinds of Jewish community surveys, designed the questions. In all, 501 Jews were interviewed. The Jewish Journal used a similar methodology to survey non-Jewish Americans’ opinions on the deal (I won’t get into that here). The margin of error on the Jewish survey was 6 percent. The survey was conducted July 16-20.

The Israel Project

After an initial question about whether the respondent is a U.S. citizen, the survey asks:

“Now, generally speaking, would you say that things in the country are going in the right direction, or have they pretty seriously gotten off on the wrong track?”

Right direction: 37 percent
Wrong track: 53 percent
Don’t know/no opinion: 9 percent

Ira Sheskin, a University of Miami demographer who has conducted dozens of Jewish surveys, called it a terrible question.

Instead of framing the alternative of “right direction” as “wrong direction,” it articulates the negative as: “pretty seriously gotten off on the wrong track.”

“I would read that question and dismiss the entire survey,” Sheskin said. “They clearly are looking for particular responses when they do that.”

Let’s take a look at another question from the survey:

“Recently, the United States and five other countries (known as the P5+1) reached an agreement with Iran regarding the lifting of economic sanctions on Iran in exchange for concessions in Iran’s nuclear program. Based on what you know, do you approve or disapprove of this agreement?

Approve: 44 percent
Disapprove: 47 percent
Don’t know/no opinion: 9 percent

Suggesting that Iran agreed only to “concessions” on its nuclear program rather than agreed not to produce nuclear weapons bolsters the argument against the deal.

Finally, the most compelling reason to dismiss the results of The Israel Project poll come from the pollster himself, Nathan Klein. He told JTA he designed the questions to “educate” respondents about the Iran deal over the course of the questionnaire in order to gauge how their views would change once they were better informed. The following question was asked three times over the course of the survey, with the responses shifting over time:

“Now that you have some more information, in your own opinion, do you think that Congress should vote to approve the deal and lift sanctions on Iran or reject the deal and NOT lift sanctions on Iran?”

Near beginning of survey: 40 percent approve; 45 percent reject.
Middle of survey: 35 percent approve; 51 percent reject.
Near end of survey: 30 percent approve; 58 percent reject.

The survey was conducted via email and the questions were designed by Klein, founder of Olive Tree Strategies, former pollster for the National Republican Senatorial Committee and former director of research and messaging at The Israel Project. Klein used a third-party company that relied on previous email surveys it had conducted to find self-identified Jews in its database. Jewish respondents were sent an email blast with a link to the online survey. Only those who affirmed their religion as Jewish were able to complete the survey. In all, there were 1,034 respondents, and the poll was conducted July 21-26. The margin of error was 3 percent.

Though a cheaper method of polling, online surveys are inherently less reliable than telephone surveys, according to Sheskin, the Miami demographer, because studies show that those who feel most strongly about an issue are most inclined to complete online surveys (by contrast, telephone respondents are more likely to cooperate and complete surveys at random). In this case, the online survey probably favors those who oppose the Iran deal, because while the deal’s opponents tend to be adamant in their opposition, many Jewish proponents are openly ambivalent.

J Street

Like The Israel Project poll, the J Street poll was conducted online and by an advocacy organization with a particular agenda. J Street is a lobby group that supports the Obama administration’s Iran policies, and the survey was designed by Jim Gerstein of GBA Strategies, a public opinion research firm that works for Democratic candidates. Unlike either of the other two polls, the J Street survey did not offer respondents the option of “don’t know/no opinion.”

Klein argued that the omission of this option introduces an inherent bias in favor of the Iran deal, because uncertain respondents are more likely to trust their government on a complicated foreign policy issue they don’t know much about. But Gerstein said Klein’s assumption is false, given widespread distrust of the government, and that in any case it’s standard practice for pollsters not to offer “no opinion” options. Plus, he noted, the 20-point margin in the J Street poll was identical to the margin in the Jewish Journal poll — the result the Jewish Journal poll would have produced if the don’t knows were split down the middle.

The J Street poll had only two questions that asked for respondents’ opinion on the Iran deal — one on the deal itself, the other on which way Congress ought to vote. The questions’ formulation was identical to that used in an ABC News/Washington Post poll released July 20. But one of the questions seems to be flawed:

“As you may know, the U.S. and other countries have announced a deal to lift economic sanctions against Iran in exchange for Iran agreeing not to produce nuclear weapons. International inspectors would monitor Iran’s facilities, and if Iran is caught breaking the agreement economic sanctions would be imposed again. Do you support or oppose this agreement?”

Support: 60 percent
Oppose: 40 percent

The second line (“International inspectors …”) offers more detail on the deal that appears to bolster the argument in favor.

Gerstein said it’s fair to ask whether the question is biased, but that he stuck with the ABC/Washington Post question word for word because he wanted to be able to compare his U.S. Jewish results to a general population poll that asked the same exact question.

As with Klein, Gerstein contracted with a third party — in this case, Mountain West Research Center — to administer the online survey by email invitation. The survey included Jews who said their religion was Jewish or who said they consider themselves Jewish. In all, 1,000 respondents completed the survey July 21-23. The margin of error was 3.1 percent. Gerstein acknowledged a telephone survey would have been better but said it would have been cost prohibitive.

So, what’s the upshot?

Though it only has half as many respondents as the other two polls, the Jewish Journal poll clearly is the winner here — on the questions (neutral), the methodology (phone rather than email) and the source (the newspaper isn’t pushing a particular agenda).