There is a lot of work being done on reproducible builds in the guix project and other distros. The idea being that you can be sure that a binary package is bit for bit the same, whoever builds it and on whatever system. This would be the first time you have complete traceability of what goes into your binaries.

On guix, you can for example install substitutes of packages which you could also build manually. Since the build environment and the dependencies are very tightly controlled, you have mathematical proof that the substitute is equivalent to the package built by the maintainer. You can thus be sure that no evil third party injected malware into the substitute binary, unless ot was done at source code level and the package maintainer has put it there (by accident).