I have a few tips for you as I have also just got a holland lop that is 12 weeks old. I have had mine for almost 2 weeks now and we are just now starting to form a bond. All bunnies are different, some bond right away some take a long time! A few things you might want to try that have worked for me!
1. When you are sitting with him if he comes up to you ignore him. I have put some treats right in front of me or next to me and let my bunny walk up and eat them and turn and leave. I ignored him for probably 3 days while I did this. Then I worked up to him eating out of my hand but no pets again this took probably about 3-4 days. Finally now I can offer treats and get about 2 strokes in! Also it helps to offer irresistible treats. For example I tried to offer veggies as a treat and it was a huge no go. He likes his veggies but it didn’t work for bonding. But when he hears the bag of dried fruit he comes running! Also when I do this I usually lay on the floor so he can hop over me if he wants. It takes a long time to build trust!
2. I get ready in the morning in my bunnies room. I have a mirror on the floor and do my hair and make up while he has free time out of his cage. He seems to like to explore my makeup bag when he hears me digging in it he has to come check it out. Again he doesn’t get super close, but I let him look it over and then hop away.
3. Another thing (which maybe its just an odd thing my boy likes) but when I crawl around on my hands and knees he tends to follow me. He also likes to follow me when I walk around the room in fact hes starting to follow me throughout the house.
4. To start maybe just sit outside his cage with the door closed for a while, let him come up and sniff you through the bars (again just ignore him). When I did this I was just watching bunny videos on youtube about bonding.
Realize it takes a ton of patience! Its so hard to not pet them but you have to let them get over the “ok maybe the humans wont eat me, maybe im just a pet” phase. Your rabbit is super young so I agree he is probably super scared! It sounds like you are doing a great job so far just keep going! Also (not sure if you do this or not) but at first I was picking my bunny up and moving him out of his cage which cause a lot of mistrust. After realizing that this was his problem with me I have learned that I never pick him up unless absolutely necessary. His breeders before him were picking him up all the time and told me he really liked it (this was in no way true, he hates being picked up) so the first few days were rough for us because he thought I was going to pick him up . I hope this helps!