Those rules might not be unwritten depending on the community and/or instance the post is on.

I don’t personally disagree with voting however you want to while browsing, it’s just intentionally continuing to downvote a user or community when you could have blocked and moved on that I object to