Gems

Gems can be mined in veins rarely underground and are very valuable. With the exception of the Ruby, which is used in the crafting of the Gold Crown and the Robe, and the Topaz, which can be crafted into the Sandgun, their only crafting use is to forge a Phaseblade. <-- Not true. Jungle Armor. However, they are worth a large amount when sold to NPCs.

Gems are found mostly underground, inside Stone. Gems are scattered completely randomly, and the depth at which you dig has no effect on what rarity of gem you find. You are as likely to find diamond 300 blocks underground as you are 2700 blocks underground.