In a large bowl, stir together sweetened condensed milk, vanilla, and key lime juice.
In a separate large bowl, beat heavy cream with an electric hand mixer until it reaches stiff peaks.
Fold whipped cream into the sweetened condensed milk mixture gently until fully incorporated.
Pour mixture into a standard loaf pan and add graham cracker pieces on top. Freeze for 6 hours before serving.