After a preposition it should be "whom," no matter how formal or informal you want to be:
By whom ("by who" would be illiterate) was he given the new car.
This passive sentence is stiflingly formal though and gives me the shivers.
If you delay your "by" until the end:
"Who was he given the new car by?" is easier, but because of its passive voice is barely passable in colloquial speech.
"Who gave him the new car?" is usual.
"To whom did you give the car?" is formal and OK.
"To who did you give the car?" is illiterate.
"Who did you gave the car to?" is informal but OK.
"Whom did you gave the car to?" is again formal but OK.
Clue:
To be formal, write "whom" right after the preposition:
To/By...whom was he...
To be informal, say "who" and delay the preposition until the end: Who...to?/by?/for?
Try not to be anything in between.