Paget was the youngest son of Henry Paget, 1st Marquess of Anglesey by his second wife Lady Charlotte, daughter of Charles Cadogan, 1st Earl Cadogan.