Letting them go to other teams completely compromises the point of a draft league. Other teams would be getting players that went in the third/fourth round.
Sure they’re not 1,2 draft picks but they would still give certain teams an unfair advantage if they got them at this point