One who takes pleasure in the speech of a heretic, even when it is not heretical talk, comes to thoughts of idolatry.

Sefer HaMidos, Faith, Part I §5