It has been proposed that as a result of human adaptation to different climates, mitochondrial genes have been affected by natural selection. To further study the selective pressure on human mitochondrial DNA, we have analysed polymorphism at the gene, domain and nucleotide site level in four geographic regions. The ratio of non-synonymous relative to synonymous substitutions is elevated for ATP6 in North Asia, ND3 in Africa and cytb in Europe relative to other mitochondrial genes. In addition, non-synonymous substitutions appear nearly five times more frequently outside core functional domains as compared with within functional domains. Therefore, a large part of the rate variation between mitochondrial genes is explained by differences in the length of the functional domain relative to the length of the gene. The accumulation of non-synonymous substitutions in the ATP6 gene among sequences from North Asia has been a gradual process over many thousands of years, consistent with relaxed purifying selection rather than recent strong selective sweeps. It is not necessary to invoke adaptive selection to explain the pattern of polymorphism in mitochondrial genes, when the most parsimonious explanation is relaxed purifying selection and the action of genetic drift.