As another reviewer was saying, things may work better if the dup check is done first.
If you check broken links first, it may hang.