How to avoid crawling 𝐝𝐮𝐩𝐥𝐢𝐜𝐚𝐭𝐞 𝐔𝐑𝐋𝐬 at Google scale? Option 1: Use a Set data structure to check if a URL already exists or not. Set is fast, but

How to avoid crawling 𝐝𝐮𝐩𝐥𝐢𝐜𝐚𝐭𝐞 𝐔𝐑𝐋𝐬 at Google scale? Option 1: Use a Set data structure to check if a URL already exists or not. Set is fast, but