Most species can spare many of their males but few of their females, since one male can fertilize many females and isn't needed for childcare in most cases.

Colorfulness risks both predation and having more trouble catching prey as well.

However, there are circumstances where being colorful isn't risk--as in coral reef environments--and there, among fish small enough to hide in the reef's crevices, both genders are usually colorful.

The dangerousness of being colorful is shown in many birds where the male gets colorful plumage in breeding season and is camouflaged the rest of the year (ducks especially).

Offhand I can only think of one species where the female is bigger and more colorful: a benthic angler fish where the male is tiny and inconspicuous and when he finds a female he bites her, becomes a sessile dependent organism on her body, the nervous system degenerates and he becomes just a little sperm factory.