It looks like the breeder you got him from doesn't care and isn't interested in doing the right thing and I'm not sure I would take him to back to such an unfit breeder in the first place. I'm not sure why he said "That is sad" when the breeder himself shouldn't have weaned him this early, as you know. That really is just terrible.
Did the vet give you any suggestions as far as feeding?
I don't have alot experience in this area but here are some links that may be helpful for your situation - one of the links suggests KMR (Kitten milk replacer you can buy at petstores)
http://www.mybunny.org/info/newborn.htm
http://www.rabbit.org/care/orphan.html
Keep us updated.