To determine the importance of Lactobacillus in shaping the human gut microbiome, the microbial composition of stools from 1,602 children between the ages of 0.3 months and 37.2 months was analyzed in a general population cohort in the All Babies in Southeast Sweden study. Lactobacillus colonized only 32% of the total pediatric population at an average relative abundance of 0.29%. Lactobacillus was age-dependent, decreasing in prevalence and relative abundance over time. The main determining factor for Lactobacillus colonization was whether the individual was actively breastfeeding. Following cessation of breastfeeding, Lactobacillus prevalence rapidly declined. However, within the actively breastfeeding cohort, 45.6% of the population remained uncolonized by Lactobacillus. The presence versus absence of Lactobacillus was determined to be human leukocyte antigen (HLA) dependent. Individuals with HLA DR15-DQ6.2 were 3.4 times more likely to be colonized by Lactobacillus than those without the haplotype, and those with HLA DR5-DQ7 were more likely to have zero Lactobacillus despite actively breastfeeding. These results suggest that HLA genetics should be considered when designing Lactobacillus-based probiotics.