You should really invest some time into exploring each kingdom’s soldiers, and recruit the ones who have potential. Good armies need good soldiers, not peasants and farmers. Before moving on to worthy fights, battles and sieges, you really need to think about increasing your party size.