Good question Arnie,
The important thing is too not kill the bacteria colony that lives in your filter. The two most dangerous things to the colony are temperature extremes and chlorine.
There are several good methods of cleaning sponges and filter material. One of them is to use the water that you take out of your tank when you do a water change and clean your sponges with it. You can use a bucket or just pour the water over the sponge as you genty squeeze the sponge.
Cleaning just parts of your filter set up at one time is a good way to avoid possible spikes in ammonia, but I don't think that it is as critical as avoiding chlorine and cold (or very hot) water.