Question: whom were you talking to?
Answer: The correct version is “To whom were you speaking?” It is incorrect to end a sentence with a preposition, so the “To” should come first. But speaking absolutely correct English has gradually gone out of style, and sounds artificial and overly-formal now.