Nobutora fought Hiraga Genshin at the Battle of Un no Kuchi in 1536. He was forced to retreat, but his son Harunobu defeated Hiraga and took the castle. Nobutora nevertheless wished to pass on his domain to Nobushige, and so Harunobu overthrew his father and exiled him to Suruga. Nobutora didn't return to Shinano until the death of Shingen in 1573, invited by his grandson Katsuyori, on that time Nobutora was in his 80's, some reported that even as an old man he still managed to strike fear to people around him.