It's not all by mail, btw. Many cities also offer secure dropbox locations, saving even the cost of the stamp to mail it.

About the only way I can imagine how it could be defrauded is if a partisan mail clerk risked federal prosecution by dumping a batch of ballots from a heavily partisan precinct. Not knowing the internal controls the postal service uses to avoid mail tampering, I can only guess it requires considerable risk with a small chance of success.