Yes, 404 errors with high traffic can have a negative effect on page load speed. This is because most 404 error pages are not cached and therefore, the server is hit each time a page not found (404) is requested. The net effect is that a 404-error page with high traffic will make bandwidth consumption increase.
Low bandwidth means low page load speed.
Solution: If the 404-error page has high traffic, then 301 redirect it to a custom page.
NOTE: Too many 404-errors can also lead to your disk space being used up because of the increase in the size of the error log file.
Here is an interesting article you might want to read about the cost of page not found 404 errors