Continuous effort to offer better user experience is what makes Google different from others. And that is the reason, Google is putting so much stress on website loading time. How much time your website takes to load is a factor Google is likely to consider to rank your website!
Just think: a user runs a Google search, Google displays the results, and then when the user clicks on a link, it takes several minutes to load the page! Why should Google list such a web page that annoys its user? Logical!
Slow websites are bad for your users also. When someone with dial up connection visits your website and the page takes so much time to load, chances are high that the user will leave your website. So, you are losing visitors because your website is slow.
Google is so desperate to enhance user experience that they are going to introduce a new internet protocol, name spdy (should be pronounced as speedy) which is supposed to be many times faster than http. Its own optical fiber based private internet is in beta or something. What else do you need? Get ready to type spdy://www.google.com/ in near future.