I only say top five because I personally havn't played some of the awesome games from back in the N64 hay day like Banjo Kazooie or Perfect Dark. I am comfortable picking a top five based on the games I was able to (and am able to) play, others may have played so many games they can put a top 10 together. Here is my current top 5, may change it as I acquire more games I used to play.

1. Super Smash Bros. - this is number one simply because I have played it the most since so many friends in University wanted to play it, so I got a lot more involved in it lol. Also enjoyed every minute of playing it with 3 other people. I also first saw this game when all we owned was an NES, I wanted SNES, but was in total awe of the N64, and thought this was amazing when I saw my neighbor play it.
2. Super Mario 64 - this is the one that I remember playing at a neighbours house and being in awe every second as previously I had played a ton os SMB1, some SMB2, a solid amount of SMB1 and a bit of Super Mario World. I loved them all, and seeing Mario in 3D as an 8-10 year old was mind blowing.
3. Conker's Bad Fur Day - truthfully, I only played the Live And Reloaded verson on Xbox until acquiring it recently. If I played it back in the day it would probably be number one, but without the nostalgia, or exploring it as much as Mario 64, it can only be at 3. I love it as, for an older game, it plays quite well and feels less dated than many N64 games I've played. It beats the next game on the list simply because I find myself continuously drawn to playing it still, whereas the next one is great but I was able to walk away from it after a couple plays without feeling drawn to it (disclaimer, could be because I traded a bunch of games for it, and didn't pay too much for in box Zelda).
4. Legend of Zelda Occarina of Time. This would be number one on most lists, but I only ever got to play it myself as a kid for 15-30 minutes at place where you could pay a buck or so to play a game in store for 15 min. I saw my friend play the final boss battle with Ganon, and was surprised at the ability of the N64 compared to the games I'd played before. Seemed like a more hard core, sword and shield mario. I've played it since buying it a couple of years ago and you can still feel the quality; it just hasn't gripped me as tight as abraisive and surprising (for a nintendo game) humour of Conker, otherwise it would be ahead.
5. Mario Kart 64 - I grew up with NES and was very familiar with Mario, Luigi, Peach, Toad and that mean old Bowser. They through in a couple pals like Donkey Kong (played the SNES version on Arcade in my grandmas town, as well as original on NES bootleg cart), and good ol' Yoshi who I only got to see when playing Super Mario World at my aunt and uncles. Was never a huge fan of racing games, but getting to use some of my favorite characters, while adding in cool power ups and weapons (triple red shells ftw!!) made it an awesome game that I still play with friends today. Also, one of the only games girls I know actually seem to want to play.
HM: Star Fox 64 - hadn't really played the original when I first rented this as a kid, and man was I blown away. It took me a bit to make to past the first 3-4 levels, but I didn't care, I could have played the first mission for hours. It was the first time I really felt how powerful the newer systems could be, as I didn't think that was possible after personally playing stuff like Top Gun on NES (and seeing the original Star Fox at a friends) not long before.
HM: GoldenEye - Loved this game as a kid, played it probably more than any other when I finally had an N64 back in the day. The only reason it is not in my top 5 is because I bought it again a couple years ago and took it back, it can still be fun with friends, but having been spoiled with dual stick FPS's, I just found it to hard to want to play it for the money. I want to get it again, but these old FPS's just feel outdated and there are many other games that were good back then, that also hold up better for me today.

The reason I don't go up to 10 is I don't want to add arguably lesser games that I played extensively, simply because I was a late adopter of the N64 (Game Cube launched about 2 weeks to a month after I finally got one lol) and thus played a lot of games because my friends had them. I LOVED Road Rash 64 (pet rock is best last-place prize ever) and Turok 2 and 3, as they are arguably not great, but I had nostalgia for them.

I love seeing these lists because everyone's is different for different reasons, What are yours?

I had forgotten a bout some of the games that have been mentioned that I I had played as a kid. I may have to edit my list and make it a top 10.