Comparing the van Oosterhout and Chybicki-Burczyk methods of estimating null allele frequencies for inbred populations

• Published in: Molecular ecology resources Vol. 12; no. 6; pp. 975 - 982
• Main Authors: Campagne, P., Smouse, P. E., Varouchas, G., Silvain, J.-F., Leru, B.
• Format: Journal Article
• Language: English
• Published: England Blackwell Publishing Ltd 01-11-2012; Wiley Subscription Services, Inc
• Subjects: Algorithms; Animals; Animals, Inbred Strains - genetics; Biostatistics - methods; Computer Simulation; Gene Frequency; heterozygote deficiency; inbreeding coefficient; null alleles
• ISSN: 1755-098X; 1755-0998
• DOI: 10.1111/1755-0998.12015

In spite of the usefulness of codominant markers in population genetics, the existence of null alleles raises challenging estimation issues in natural populations that are characterized by positive inbreeding coefficients (F > 0). Disregarding the possibility of F > 0 in a population will generally lead to overestimates of null allele frequencies. Conversely, estimates of inbreeding coefficients (F) may be strongly biased upwards (excess homozygotes), in the presence of nontrivial frequencies of null alleles. An algorithm has been presented for the estimation of null allele frequencies in inbred populations (van Oosterhout method), using external estimates of the F‐statistics. The goal of this study is to introduce a modification of this method and to provide a formal comparison with an alternative likelihood‐based method (Chybicki‐Burczyk). Using simulated data, we illustrate the strengths and limitations of these competing methods. Under most circumstances, the likelihood method is preferable, but for highly inbred organisms, a modified van Oosterhout method offers some advantages.