All perfect praise be to Allaah, The Lord of the worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His slave and messenger.
Zakah on fruits becomes due once they are visibly ripe. An-Nawawi said in Sharh Al-Muhaththab: "Ash-Shaafi‘i and his students said that: "The time when the Zakah of palm trees and grapes becomes due is when the fruits are visibly ripe and the time when the Zakah on grains becomes due is when they are solid. This is the correct opinion that is known from the early and late statements of Ash-Shaafi‘i. Its soundness is collectively agreed upon by Shaafi‘i scholars."
This is also the opinion stated by Hanbali jurists. Hence, you should know that, if the olives that your grandfather left were obviously ripe before his death, then Zakah is due on them before they are distributed because it was due when the olives were still under his ownership. However, if the ripeness of the fruits was not verified until after his death, then their Zakah is due on you since their ripeness was verified after its ownership was transferred to you. Each inheritor should take his share and pay its Zakah individually if it reaches the Nisaab.
Please note that there is a famous divergence of opinion as to whether Zakah is due on olives or not. The preponderant opinion is that it is due on olives since they are a kind of edible crop as stated by the Quran in the Saying of Allaah The Almighty (which means): {And [We brought forth] a tree issuing from Mount Sinai which produces oil and food for those who eat.}[Quran 23:20]