That’s a clever line of attack, but having an opinion does not constitute a conflict of interest. Otherwise there would be a whole shit-ton of recusal happening every day.
A conflict of interest usually involves some form of monetary compensation or other fiscal benefit.
This is sensible in the current social climate.