Yes, according to the Third District in Robert Gielchinsky v. Vibo Corporation (Gersten, Shepherd & Schwartz). In that case, all parties agreed to the appointment of a special magistrate to address discovery issues. Later, one of the parties withdrew that agreement to the special master.
According to Florida Rule of Civil Procedure 1.490(c), consent is needed to refer matters to magistrates. Once withdrawn, then “it logically follows that the matter is no longer appropriate for a special magistrate.”