Should grandfathers wear tuxedos or suits? They are not part of the bridal party.
They can wear suits. But if they are going to be in wedding photos, it would be nice if their attire complements the wedding party.
I agree that they should dress up a bit. They really shouldn’t be dressed as one of the groomsmen, but dressed slightly more formal than the guests.