A younger man asked me out, then he has another plan. Why did this happen?

Ok. I just started working at a big place. I met this guy during orientation. I really liked him - aside from his looks, he seemed to convey things very well. He was very articulate and his brain seemed not to have any preconception or racist thoughts. He also prayed beautifully before lunch. I started saying "Hi" to him, sat next to him, asked him some questions, and felt very comfortable around him. Then, someone suggested we hang out together because we're both new to this area. I asked him out asking if he wanted to have some and crab cakes. He backed off, and invited me instead to a lunch with him team members at work. I backed off. I didn't have any romantic intentions though, because I didn't want to be the only non-team member in that lunch. So weeks after, he came by with a sad look on his face. I think he was lonely because he doesn't know anyone in this area. He asked me to have lunch with him. The next day, he asked me what I was going to do over the weekend, and if I wanted to go to this touristic place. I felt very charmed, and shouldn't have felt so. The reason I am saying I shouldn't have felt charmed is because I don't want to have a relationship at work, and secondly I'm 11 years older than him. He is very handsome, and I don't look that old, or so I am told. He asked what the plans were for this weekend, and if I wanted to go to this touristic place as I said, and then, he said in front of a co-worker and me, that he was going to Ocean City this weekend. I am saddened, because he backed out, and secondly, because I'm starting to think about him and a lot.