4 – James Madison Birthplace of James Madison Our 4th President, James Madison, was born here in Port Conway, Virginia in 1751. It is interesting that he was born here at Belle Grove Plantation instead of Orange County, where his parents lived. They were visiting Madison’s mother’s family at Belle Grove when she gave birth to him. The house Madison was born in no longer stands. Belle Grove Plantation is privately owned. The drive to Belle Grove. The rear entrance. The front (river facing) entrance. Rappahannock River looking upstream toward Fredericksburg. Out building Out building door Out building undergoing restoration. Out buildings Thanksgiving dinner inside the house.