I don't know if Milestones is considered "too casual" by your standards, but my friends and I went to the one at 1210 Denman Street last year for a girl's birthday, and we had a blast.
If you guys can brave the cold, it's right by the water which can be gorgeous at night on a clear day, and it's also right beside Stanley park if you guys plan on driving.
Or if you prefer to be Downtown downtown there's one on 1145 Robson Street as well.
http://www.milestonesrestaurants.com/