"The size of X" is a bit misleading. It's between 0.41% and 8.2% of the North Pacific Ocean. Earth has several of these patches. It's not a large humongous mass. Most of it are finer particles coming from several processes of human waste disposal with dumping waste being just one of them, one of the other being remains of washing clothes made of polyesters and other artificial materials as most sewage treatment plants are unable to filter smaller particles. The biggest danger isn't just plastics ending up in our food but also that most toxins released during breakup ends up as well the same way, DDTs, PCBs and PAHs are all known to be carcinogenic and linked to conditions such as diabetes. Now this does not mean the next fish you eat will be your last meal but you could eat someone else's clothes at least.