Their scenes always were removed from all the other plots, and I think they boxed themselves into a corner due to the fact that Ezra was Aria's (and all the other characters') teacher. You could incorporate all the other love interests or secondary characters into the main plot more easily, because they were the four girls' peers at school — there was a reason for them to be in the same place, hanging out together. It was much more difficult to put Ezra in the same space as the other characters, experiencing the same things.

Toby's actor was the weakest of the bunch, which really didn't help, as you say, and it was even more obvious as he was mostly playing against Spencer's actor, who was probably the best of the four main girls.