Justice you have hit it on the head. But my advice is keep attacking the inactives only at this stage. Someone may only have light defences but they could still ninja you.
Search high and low for the inactive players, build Large Cargos and research comp tech so you get more fleet slots 😉
When your ready with colony ships make sure you plant them near other friendly planets. We have a large community out in galaxy 7, it’s fairly safe out there.