Wireless Router Troubles?

I have a WRT54GL, and have installed DD-WRT. It works fine when I’m the only one using it, however, I’ve noticed in the past little bit, with a bunch of people using it, sometimes the connection randomly drops out, or all computers start to refuse to connect it (they want the key again), and even a hardwired connection refuses to connect proplerly. It works fine when I unplug it and plug it back in. I really don’t have anything that unusual going on, just a WPA2 connection, and I have the wireless power turned up to 70dB. It seems to be better now that I switched to channel 3 from 9, and moved from WPA to WPA2 (Some windows computers had a hard time with WPA2).