yeah should be better off to replace all sticks instead of just keeping some slow ones aswell.
do note it won't affect performance that much but you can reuse good ram for any future build else. (since ddr4 is still 1-2 years away and likely won't matter for gaming PoV aswell)