Adam Smith says partner Janine Pitman and their six-month-old daughter Rosie had left their home in Godsey Lane, Market Deeping, just minutes before the Renault Scenic ploughed through their fence and into the front door.
The impact smashed the front door open.
The elderly couple in the vehicle were cut free by firefighters but police say they were not injured.
Mr Smith, 28, said: "I feel lucky that Janine had been out five minutes before. If they had gone to the toilet or had a drink and left a bit later, possibly they would have been in the property or on the path.
"The main thing I was worried about was the driver, property can be repaired.
The crash on Friday last week comes just two months after another car hit Mr Smith's vehicle on his drive.
The Renault ploughed into the house at 11.45am.
Firefighters cut the roof off to free the couple, both 77. It is believed the woman driver lost control of the automatic vehicle after leaving a car park across the road.
The first Mr Smith knew of the crash was when he returned home to find the road blocked by police.
The care worker said: "I immediately thought that there as a fire, I could see the fire engine and the firefighters in the garden but couldn't see the vehicle. I was worrying there were still people involved that could have been injured."
He says the cost of repairing the front door, fencing and a neighbour's fencing will be paid for by the driver's insurance.
Police re-opened the road at 1.15pm.