I was sharing my screen with a friend the other day. I had just reloaded my computer a couple days before so I had to install some core gems, including Rails, to get spooled up on the project we were looking at. He noticed that my gems installed really fast.
The reason is not because my computer or Internet connection are faster. It's because I don't install rdocs. All you need is to make your .gemrc file look like mine by telling Rubygems not to install docs and you'll save yourself some time, especially with big docs like Rails.