On the ANE-2 list, Ariel L. Szczupak points to preprints of articles from an upcoming issue of The Annals of Applied Statistics. The preprints include an article by Andrey Feuerverger, the statistician whose calculations played such a large role in Simcha Jacobovici’s Lost Tomb of Jesus program, as well as eight responses plus one more reply by Feuerverger.

I have insufficient statistical competence to comment on Feuerverger’s mathematics. However, I draw your attention to the final paragraph (except for the “Acknowledgements”):

Among the various assumptions made, perhaps the one that most ‘drives’ our analysis in the direction of ‘significance’ is the extraordinary inscription Mariamenou [η] mara. It has been speculated that Mary Magdalene was a principal driving force in the movement founded by Jesus but was later vilified in the course of patriarchal power struggles. While we are in no position to weigh in on any such theories, what we can say is that from a purely statistical point of view, this much is true: It is the presence in this burial cave of the ossuary of Mariamenou [η] mara, and the mysteries concerning the identity of the woman known as Mary Magdalene, that hold the key for the degree to which statistical analysis will ultimately play a substantive role in determining whether or not the burial cave at East Talpiyot happens to be that of the family of Jesus of Nazareth.

In other words, the statistical case for connecting the Talpiot tomb with the family of Jesus of Nazareth depends strongly on identifying “Mariamenou [η] mara” with Mary Magdalane. In my judgment, Feuerverger’s paper pays insufficient attention to criticisms of this identification, and in fact he makes this identification a presupposition of his analysis:

We assume that the full inscription Mariamenou [η] mara refers to a single individual and represents the most appropriate specific appellation for Mary Magdalene from amongst those known; we further assume that this inscription is sufficiently distinctive that it could only have applied to very few and/or very particular individuals within the generic Mariam name category. Our specific implementation of this assumption will be of the type to assume that essentially at most one out of every 74 Mariams could legitimately have been rendered in this way, and that Mary Magdalene was among those who could.

Among the respondents, Holger Höfling and Larry Wasserman reassess Feuerverger’s data using two other statistical methodologies. Using a Frequentist approach, Höfling and Wasserman

conclude that the observed event is not rare at all. The chance that an observer would find a tomb that could be said to contain interesting target names is large. This is due to the fact that the interesting names are common and that the many tombs provide many opportunities for apparent surprises.

Using a Bayesian analysis, they estimate that “[e]ven in the optimistic scenario, there is only about a 60% chance of the tomb belonging to the NT family. In the other two, more realistic schemes, the probability is only 22% and 3%.” Höfling and Wasserman commend Feuerverger for taking on an interesting problem, but in the end, they conclude that “Our analysis suggests that the finding does not lend support to the hypothesis that the find is indeed the tomb of the NT family.”

In another paper, Camil Fuchs, though self-confessedly biased in favor of Feuerverger’s methods and results, writes:

On a personal note, I confess that I would have been very pleased to be able to conclude my discussion with two positive statements: a) that I found the results convincing and we can second Prof. Feuerverger’s claim that the tomb is most likely that of the NT family, and b) that the new approach is preferable to the existing methods in deciding whether the tested object is the special one.

Unfortunately, to anticipate the findings detailed below, despite the initial excitement and the personal preferences, I find myself in disagreement with the results and the conclusions. As for the new approach, it may evolve and prove beneficial, although not necessarily preferable to existing methods. I believe its properties have yet to be investigated.

Basically, the a priori assumptions—equating Mariamenou [η] mara with Mary Magdalene and treating the Matya ossuary as irrelevant—hamstring the value of Feuerverger’s calculations.

Another respondent, Don Bentley, notes that “we can only accept the conclusion if we are willing to accept the assumptions.” Bentley argues that Feuerverger’s procedure presupposes that a family tomb of Jesus did exist, and moreover, that it was located in Jerusalem:

I was unable to find a definition of “historical,” but based on the assumptions and arguments presented in the text, I have been led to believe that by historical viewpoint is meant a strictly literal translation of the Gospels with one exception. The final paragraph of Section 1 states, “We remark that in assessing the evidence in any way, it is essential to adopt a strictly historical viewpoint, and thus to set aside considerations that a NT tombsite cannot exist. In fact, Jewish ritual observances prevalent at the time are entirely consistent with the possible existence of such a tomb.” This wording is equivalent to assuming a New Testament family tomb might exist, that it is possible there could be a family tomb which might even contain the remains of Jesus of Nazareth.

However, this is not the assumption that was used in the analysis. Instead, the analysis is conditioned on the assumption that, with probability one, there did exist such a tomb. And even beyond that is the assumption that this Jesus family tomb was in the vicinity of Jerusalem with probability one. In Section 14, the author states, “We are in fact now in a position to carry out a particular hypothesis test: Here H₀ is the hypothesis that all 1,100 tombs in the vicinity of Jerusalem arose under random assignments of names, and H₁ is the hypothesis that one unspecified one among these 1,100 tombs is that of the NT family.” However, the calculations do not seem to allow for an a priori probability greater than zero that there does not exist a family tomb in Jerusalem.

In other words, Feuerverger’s calculations do not even allow for the possibility that the “Jesus family” did not have a tomb in Jerusalem! Bentley also points out that Feuerverger’s statistics are based in part on estimates of the population of Jerusalem, without taking into account non-residents who might be buried in Jerusalem-vicinity family tombs; yet the conclusions reached depend precisely on non-residents of Jerusalem being buried there.

Bentley seems aware of the scholarly discussion about the interpretation of “Mariamenou [η] mara” (citing Francois Bovon, Stephen Pfann, and others). Bentley also points out that while Feuerverger’s assumptions follow Rahmani’s interpretation of “Mariamenou [η] mara” (not the Mary Magdalene identification, but the simple reading/translation of the name[s]), but those same assumptions reject other elements of Rahmani’s treatment of the Talpiot ossuaries. Bentley also sees through Jacobovici’s shell game:

The author [Feuerverger], in numerous places throughout the paper, makes comments such as the following which appears in section 1, the Introduction and Summary. “Our computations were carried out under a specific set of assumptions….” And in the first paragraph of section 13, A Statistical Analysis, we find the statement, “The assumptions A.1 – A.9 under which we carried out our analysis are by no means universally agreed upon. Furthermore, the failure of any one of them can be expected to impact significantly upon the results of the analysis.” There is a footnote to this statement which warrants particular attention. The footnote states, “These assumptions were proposed by S. Jacobovici, except for A.6 & A.9 which are due to the author.”

Simcha Jacobovici (2007) is coauthor with Charles Pellegrino of the book, The Jesus Family Tomb, as well as executive director of the Discovery Channel’s documentary, The Lost Tomb of Jesus. At the press conference held on February 26, 2007 at the New York public library Jane Ruth, the president and general manager of the Discovery Channel referred to the subject of the documentary as, “what might be one of the most important archaeological finds in human history.” After introductory remarks by Ms. Ruth and the documentary’s producer, James Cameron, the podium was turned over to Jacobovici to provide the facts about the find. Following his presentation and prior to allowing questions from the media that were directed to the panel of experts, Jacobovici made the following statement. “Before I turn it over to the experts, because I have to say again, I’m not going to say I’m not an expert. I’ve seen a lot of internet buzz on this. I am an expert. My expertise is investigative journalism. I’m not an archaeologist. I’m not a DNA expert. I’m not a statistician. I’m a filmmaker and a journalist.”

During the questioning by the media, the statistician [Feuerverger] who is the author of the paper under discussion had the opportunity to speak. He began, “The obvious needs to be stated; that I’m not a biblical scholar, not a historical scholar. I’m just a numbers guy. … As a statistician, I do the calculations based on assumptions given to me by the subject matter experts, in this case the historical biblical scholars.”

The above quotes from the press conference raise two concerns. First, the responsibility of the statistician as an umpire of the science cannot be fulfilled by just accepting a given set of assumptions. They must be checked to make sure they meet certain standards as identified at the beginning of this discussion. But beyond that, the author did not even satisfy his own standards of using assumptions provided by “subject matter experts.” By Jacobovici’s own admission, his areas of expertise are filmmaking and journalism, and not the substantive fields of the complex history and archaeology of first century Judaism. Yet, as stated in both footnote 32 and the acknowledgements of the paper, it was Jacobovici who provided the assumptions which served as the basis for the statistical analysis.

In their response, Julia Mortera and Paola Vicard mostly point to some odd decisions and omissions in Feuerverger’s analysis, and they ask after the causes of these omissions.

Randall Ingermanson notes that the “Mary Magdalene” assumption “is almost universally rejected by scholars in the relevant fields. Several other assumptions are extremely dubious, and each of them biases the result toward H1. Since all statistical biases in Feuerverger’s RR values accumulate multiplicatively, the net effect is an enormous bias toward H1.” In his article, Ingermanson proposes to

examine [the "Mary Magdalene" problem and] five other serious problems. By Feuerverger’s own account, eliminating two of these statistical biases (the two relating to Mary Magdalene) is sufficient to destroy the statistical significance of H1. But all six statistical biases should be eliminated from the baseline model of the problem.

After a succinct discussion of these six problems, Ingermanson concludes,

Feuerverger’s computation contains a number of statistical biases, each of which favors H1. One of these (the “Mariamenou” inscription) introduces an illicit factor of 1/44 to RR, which accounts for a very strong bias all by itself. But five other factors enter in with moderate statistical bias toward H1, and the net effect is to create the appearance of statistical significance where none actually exists.

Incidentally, Ingermanson cites Richard Bauckham’s guest post on Chris Tilling’s blog. Score one for biblicablogdom!

Sheila Macdonald Bird pays waaaay to much attention to the allegedly “missing” tenth ossuary.

Curiously, in his reply to the responses, Feuerverger claims not to have “passed judgment” on whether the tomb is that of the “New Testament family” or not—though he certainly has been represented as such by Jacobovici et al. Of course, he defends his statistical procedures against his peers’ criticism. But the most interesting point comes in regard to the identification of Mariamenou [η] mara. Feuerverger writes in his response (my emphasis in the final paragraph):

This brings us to the subject of the clarifications subsequently issued by Professor Bovon. There is no doubt whatever now that these were not retractions in response to pressures nor were they motivated by a recognition of the possible uses which might be made of such work. In fact Bovon’s clarifications are those of a serious scholar whose remarks — having inadvertently been misinterpreted by Jacobovici — were conveyed to me out of context. To quote from Bovon’s statement to the Society of Biblical Literature:

“I do not believe that Mariamne is the real name of Mary of Magdalene. Mariamne is, besides Maria or Mariam, a possible Greek equivalent, attested by Josephus, Origen, and the Acts of Philip, for the Semitic Myriam.”

“Mariamne of the Acts of Philip is part of the apostolic team with Philip and Bartholomew; she teaches and baptizes. In the beginning, her faith is stronger than Philip’s faith. This portrayal of Mariamne fits very well with the portrayal of Mary of Magdala in the Manichean Psalms, the Gospel of Mary, and Pistis Sophia. My interest is not historical, but on the level of literary traditions.”

Without benefit of the last element — i.e., (g) — of the itemization above, I do not regard the assumption A.7 — concerning the most appropriate name rendition for Mary Magdalene — as being equally adequately justified by the remaining elements (a) through (f) on that list. In particular, this means that we cannot (on the basis of our RR procedure) say that the Talpiyot find is statistically significant in any meaningful way. Readers who wish to form their own judgement on this should note that the germane question here is not whether or not Mariamne was the actual name of Mary Magdalene, but whether or not we are justified — on an a priori basis — to say that the rendition Mariamenou [η] Mara provides a better fit to the name of Mary Magdalene than any of the others, whilst bearing in mind that she is repeatedly referred to in the NT as having come from Migdal, and is not referred to there as Mariamne. We shall see below, however, that this matter is not yet closed.

The discussion “below” to which Feuerverger refers has to do with the Princeton conference. The “new” idea is that Rahmani misread “Mariamenou [η] Mara” and that the inscription should be read as “Miriam, also called Mara.” I cannot quite tell what impact Feuerverger thinks this would have; it certainly would not strengthen the case for an identification with Mary Magdalane, as far as I can tell.

In any event, I encourage you to take a look at the papers.