The addon attempts to keep track of the files it has already downloaded to avoid re-downloading them. This happens within the addon, regardless of your download folder's contents.
However, there is an attachment option that would serve files in a way impossible for the addon to keep track of, hence those files will be downloaded again with every visit. You will recognize them by their file name, consisting of the timestamp of download and a randomized number.