In most cases, the metals in the minerals from the mine/quarry dissolve into the soil and kill all but the hardiest plants, often disrupting the ecosystem in the area (though other human activities associated with mining, like building roads, are also quite disruptive). Zinc and aluminum are particularly common and very toxic to plants.

Sometimes you get enough metal ions to form strong acids, killing not just plants but also animals, fungi, and bacteria (e.g. the superfund site in Butte Montana: http://www.pitwatch.org/ )