Thought I would post all the known crystal colours and where to get them, for reference.
Artifice
Blue – (+31 END, or CRIT) *Artifact* <LIGHT ONLY>
Red – (+31 END, or CRIT) *Artifact* <DARK ONLY>
Green – (+31 END, or CRIT) *Artifact* <LIGHT ONLY>
Yellow – (+31 END, or CRIT) *Artifact* <NONE>
Orange - (+31 END, or CRIT) *Artifact* <NONE>
Magenta – See Link (+33 END, CRIT, or POWER) *Artifact* <NONE>
PVP & PVE
Purple - Tier 2 Empire Columi Gear. They can also be obtained from PvP sabers. Not removable.
Black Blue - Republic Rakata Gear
Black Red - Empire Rakata Gear
Cyan - Obtained from Columi gear, not removable.
Special
Black Yellow - Pre-order Bonus (+4 END) <NONE>
Black Green - Obtained as bonus for Razer TOR hardware - http://www.razerzone.com/swtor <NONE>
Unconfirmed
White - So far unknown.
Thanks to WraithAscendant for gathering this information. Link