This happens often, especially if you use a Twitter client like Tweetdeck. Someone tweets something worth fav-ing so you go to fav it, except for some reason, maybe the column streamed down or maybe you just clicked the wrong Tweet and hit fav without realizing it.
What do you do? Quickly un-fav and hope they didn’t notice? What if they have email notifications on? Then they’ll see it was fav’d, go to Twitter and see that you had un-fav’d it. Or worse, they could be Twitter addicts and catch everything that happens? What do you do?
The answer is simple - unless you absolutely 100% don’t agree with what was Tweeted (like racism), you leave it be.