The addon has no way of knowing if the change came from you or from YouTube. If you want, you can always disable the addon (it has a quick menu, see screenshots) and turn it on later