Sir2 is a central regulator of yeast aging and its deficiency increases daughter cell inheritance of stress- and aging-induced misfolded proteins deposited in aggregates and inclusion bodies. Here, by quantifying traits predicted to affect aggregate inheritance in a passive manner, we found that a passive diffusion model cannot explain Sir2-dependent failures in mother-biased segregation of either the small aggregates formed by the misfolded Huntingtin, Htt103Q, disease protein or heat-induced Hsp104-associated aggregates. Instead, we found that the genetic interaction network of SIR2 comprises specific essential genes required for mother-biased segregation including those encoding components of the actin cytoskeleton, the actin-associated myosin V motor protein Myo2, and the actin organization protein calmodulin, Cmd1. Co-staining with Hsp104-GFP demonstrated that misfolded Htt103Q is sequestered into small aggregates, akin to stress foci formed upon heat stress, that fail to coalesce into inclusion bodies. Importantly, these Htt103Q foci, as well as the ATPase-defective Hsp104Y662A-associated structures previously shown to be stable stress foci, co-localized with Cmd1 and Myo2-enriched structures and super-resolution 3-D microscopy demonstrated that they are associated with actin cables. Moreover, we found that Hsp42 is required for formation of heat-induced Hsp104Y662A foci but not Htt103Q foci suggesting that the routes employed for foci formation are not identical. In addition to genes involved in actin-dependent processes, SIR2-interactors required for asymmetrical inheritance of Htt103Q and heat-induced aggregates encode essential sec genes involved in ER-to-Golgi trafficking/ER homeostasis.