Mechanism 1: rare earth elements taking calcium's place
During crystal formation, some rare earth elements —yttrium, cerium, samarium, europium, among others— can slip into the structure and take the place a calcium atom "should" occupy. These elements aren't neutral guests: since their size and electrical charge differ from calcium's, they slightly alter how the crystal absorbs light, and that alteration translates into color.
What are "rare earth elements"? This is the name given to a group of seventeen chemical elements —including yttrium, cerium, and samarium— which, despite the name, aren't especially scarce in the Earth's crust, but almost always appear mixed together and in very small amounts within other minerals. In fluorite, they act as "impurities" that slip into the structure during its formation.
A landmark study on the subject, published in the scientific journal Physics and Chemistry of Minerals, established that yttrium combined with structural defects produces blue tones, the combination of yttrium and cerium gives rise to yellowish greens, and elements such as samarium are behind much of the green seen in specimens from Mexico or Namibia. The intense fluorescence of some fluorites —that blue glow under ultraviolet light— is, in many cases, due to traces of europium.