Superimposition of protein structures is key in unravelling structural homology across proteins whose sequence similarity is lost. Structural comparison provides insights into protein function and evolution. Here, we review some of the original findings and thoughts that have led to the current established structure-based phylogeny of viruses: starting from the original observation that the major capsid proteins of plant and animal viruses possess similar folds, to the idea that each virus has an innate “self”. This latter idea fueled the conceptualization of the PRD1-adenovirus lineage whose members possess a major capsid protein (innate “self”) with a double jelly roll fold. Based on this approach, long-range viral evolutionary relationships can be detected allowing the virosphere to be classified in four structure-based lineages. However, this process is not without its challenges or limitations. As an example of these hurdles, we finally touch on the difficulty of establishing structural “self” traits for enveloped viruses showcasing the coronaviruses but also the power of structure-based analysis in the understanding of emerging viruses