Andy Johnson equalised with 10 minutes to go, though, the former Everton striker dispatching the ball beyond Wayne Hennessey just seconds after coming on as a substitute.
Hughes was incensed by Oliver's decision to book Brede Hangeland for a challenge on Fletcher, although he joked he may have been given his marching orders for his inability to take out his frustrations on a bottle, which he tried to kick but missed.
"I was disappointed because I felt Brede had won the ball," Hughes said. "The referee interpreted it in a different way and saw it fit to book my player for a challenge I thought was fair, so that is why I reacted.
"Probably the fourth official played a part in my dismissal and it was for missing the bottle I was aiming at three times - that was poor play from my point of view!"
Hughes subsequently made changes which paid dividends, with Bobby Zamora - on for Eidur Gudjohnsen - tussling with Christophe Berra for a ball into the box which fell to Johnson, teeing up him up to score. "I won't take any credit for that - I'm just trying to make a difference," Hughes said.