bad rear calipers
If the piston in the rear calipers is stuck and not braking properly, the handbrake won't be able to stop the rear wheels.
Inspect calipers, if piston is super rusty replace it or buy new calipers.
worn brake pads or discs
Worn brake pads don't offer much stopping power, the same goes with a won brake disc.
Inspect them and replace them, especially if you didn't in the past 4 years.