Abstract Background Apoptin, as a tumor-specific pro-apoptotic protein, plays an important anti-tumoral role, but its mechanism of autophagy activation and the interaction between autophagy and apoptosis have not been accurately elucidated. Here, we studied the mechanism of apoptin-induced apoptosis and autophagy and the interaction between two processes. Methods Using crystal violet staining and the CCK-8 assay, we analyzed the effect of apoptin in the inhibition of liver cancer cells in vitro and analyzed the effect of inhibiting liver cancer in vivo by establishing a nude mouse tumor model. Flow cytometry and fluorescence staining were used to analyze the main types of apoptin-induced apoptosis and autophagy. Subsequently, the relationship between the two events was also analyzed. Flow cytometry was used to analyze the effect of ROS on apoptin-mediated apoptosis and autophagy mediated by apoptin. The effect of ROS on two phenomena was analyzed. Finally, the role of key genes involved in autophagy was analyzed using gene silencing. Results The results showed that apoptin can significantly increase the apoptosis and autophagy of liver cancer cells, and that apoptin can cause mitophagy through the increase in the expression of NIX protein. Apoptin can also significantly increase the level of cellular ROS, involved in apoptin-mediated autophagy and apoptosis of liver cancer cells. The change of ROS may be a key factor causing apoptosis and autophagy. Conclusion The above results indicate that the increase in ROS levels after apoptin treatment of liver cancer cells leads to the loss of mitochondrial transmembrane potential, resulting in endogenous apoptosis and mitophagy through the recruitment of NIX. Therefore, ROS may be a key factor connecting endogenous apoptosis and autophagy induced by apoptin in liver cancer cells. Graphical abstract