It may be that Baritt balked at having Aliera assassinated (or having any other harm to come to her) because he had strong romantic feelings for her. Aliera, in her turn, may have been unwilling to believe that he had been involved in the Yendi Conspiracy because she was aware of these feelings, regardless of whether she actually reciprocated them.