BTK inhibitor (BTKi) Ibrutinib carries an increased bleeding risk compared to more selective BTKis Acalabrutinib and Zanubrutinib, however, its impact on vascular endothelium remains unknown. In this study, we found that Ibrutinib induced stronger cytotoxic effect on endothelial cells than Zanubrutinib, however, Acalabrutinib cytotoxicity was extremely weak. RNA-seq, followed by KEGG analysis and quantitative RT-PCR validation, was conducted to identify the differential apoptotic target genes of BTKis, leading to their distinct cytotoxic effects on endothelial cells, which showed that Ibrutinib and Zanubrutinib dramatically modulated the expression of critical apoptotic genes, GADD45B, FOS, and BCL2A1, among which FOS and GADD45B were upregulated more significantly by Ibrutinib than Zanubrutinib, however, Acalabrutinib downregulated BCL2A1 moderately and was not able to modulate the expression of FOS and GADD45B. Next, we performed in vitro angiogenesis assays and found that Ibrutinib was more able to induce endothelial dysfunction than Zanubrutinib via stimulating more BMP4 expression, however, Acalabrutinib had no such effect. Especially, the capacity of Ibrutinib to induce endothelial dysfunction can be antagonized by targeting BMP4. Accordingly, Ibrutinib, as an angiogenesis inhibitor, inhibited ovarian and breast cancer progression in vivo. Collectively, our findings addressed a novel molecular basis underlying Ibrutinib-induced endothelial cell dysfunction and suggested the potential application of Ibrutinib to treat angiogenesis-dependent cancers.