As a gay guy who’s definitely been in the room for gay jokes because they didn’t know I was gay: yes, you do. If you are only willing to call out bad behavior when you may get caught associating with it, then you aren’t actually an ally.