Castle is an area in New Rochelle, New York. The Castle area includes New Rochelle's 'Downtown' area as well as select residential and estate sections located to the north, encompassing both the wealthiest and the poorest sections of the City.