Abstract Background According to the recent global cancer statistics, breast cancer is the leading cause of deaths among women with 2.3 million new cases globally. Likewise, cervical cancer is also among the leading causes of mortality among women. Polygonum hydropiper is traditionally known for its cytotoxic effects and several bioactive cytotoxic compounds were isolated from it. This study was aimed to isolate potential anticancer compounds from its most potent fractions and evaluate their anticancer potentials. Methods Based on our earlier studies, active fractions including chloroform and ethyl acetate were subjected to column chromatography for isolation of compounds. Chemical structures of isolated compounds were confirmed via 1H NMR, 13C NMR, mass spectrometry. Purified compounds were tested for cytotoxicity against breast cancer cells (MCF-7), cervical cancer cells (HeLA) and NIH/3T3 fibroblasts cells cultures using MTT assy. Anti-angiogenic potentials of isolated compounds were evaluated via chorioallantoic membrane assay. Anti-tumor studies were done using Agrobacterium tumefaciens induced potato tumor assay. Furthermore, to understand the binding modes of Isolated compounds, molecular docking was performed against EGFR, HER2 and VEGFR using MOE as docking software. Results Two bioactive compounds PH-1 (4-methyl-5-oxo-tetrahydrofuran-3-yl acetate) and PH-2 (methyl 4-hydroxy-3-methoxybenzoate) were purified from the active fractions. In cytotoxicity studies, PH-1 exhibited highest cytotoxicity against HeLA cells with 87.50% lethality at 1 mgmL−1 concentration and LD50 of 60 µgmL−1. Likewise, PH-2 showed 82.33% cytotoxicity against HeLA cells with LD50 of 160 µgmL−1. Similarly, PH-1 and PH-2 exhibited LD50 of 170 and 380 µgmL−1 respectively. Moreover, PH-1 and PH-2 were also very potent cytotoxic compounds against NIH/3T3 cells with 81.45 and 85.55% cytotoxicity at 1 mgL−1 concentration and LD50 of 140 and 58 µgL−1 respectively. Isolated compounds exhibited considerable anti-angiogenic potentials with IC50 of 340 and 500 µgL−1 respectively for PH-1 and PH-2. In anti-tumor assay, PH-1 and PH-2 exhibited 81.15 and 76.09% inhibitions with LD50 of 340 and 550 µgL−1 respectively. Both compounds selectively binds with EGFR and HER2 receptors with low binding energies. Both compounds exhibited stronger interactions with VEGFR through binding pocket residues Lys868, Val916 and Asp1046. Conclusions Both compounds cause considerable cytotoxicity against cancer cells. The anti-angiogenic and anti-tumor results suggests additional tumor suppressive properties. Docking analysis suggests that these compound not only has the ability to bind to EGFR and HER2 but also equally binds to VEGFR and may act as potential anti-angiogenic agents.