Mitochondrial genomes (mitogenomes) are remarkably diverse in genome size and organization, but the origins of dynamic mitogenome architectures are still poorly understood. For instance, the mutational burden hypothesis postulates that the drastic difference between large plant mitogenomes and streamlined animal mitogenomes can be driven by their different mutation rates. However, inconsistent trends between mitogenome sizes and mutation rates have been documented in several lineages. These conflicting results highlight the need of systematic and sophisticated investigations on the evolution and diversity of mitogenome architecture. This study took advantage of the strikingly variable mitogenome size among different yeast species and also among intraspecific strains, examined sequence dynamics of introns, GC-clusters, tandem repeats, mononucleotide repeats (homopolymers) and evaluated their contributions to genome size variation. The contributions of these sequence features to mitogenomic variation are dependent on the timescale, over which extant genomes evolved from their last common ancestor, perhaps due to a combination of different turnover rates of mobile sequences, variable insertion spaces, and functional constraints. We observed a positive correlation between mitogenome size and the level of genetic drift, suggesting that mitogenome expansion in yeast is likely driven by multiple types of sequence insertions in a primarily nonadaptive manner. Although these cannot be explained directly by the mutational burden hypothesis, our results support an important role of genetic drift in the evolution of yeast mitogenomes.