By default, there is a 6.67*1/15 (6.67%) chance for a block in any of the 8 surrounding tiles to be destroyed. Of these 8 tiles, if any of them has no block below, they will have a 25*1/4 (25%) chance to be destroyed. The tick the original fossil is broken, the block above also has a 25*1/4 (25%) chance to break.
To get as many fossils as possible, players should try to mine horizontally from top to bottom and avoid creating holes, which can otherwise spread quickly. Explosives and especially Scarab Bombs can harvest whole swathes of the vein with minimum breakage.