This difference is because of the difference in arrangement of carbon atoms both graphite and Diamond.
Carbon atoms in graphite are arranged in layered form in an infinite array of layers. These layers are held together by a weaker force of attraction called vander waal's force of attraction such that layer's can slip over one another. Whereas in diamond carbon atoms are arranged tetrahedrally. Each carbon atom is attached to four carbon atoms with a bond angle of 109.5°. It is strong rigid three dimensional structure that results in infinite array atoms. This accounts for hardness of the diamond.
