It may seem odd that you can't just delete someone from a mailing list, but there are good reasons why eNotify was designed so that recipients must be unsubscribed and not just deleted.
Unsubscribe information is preserved within the module so that the user is not returned to a mailing list they unsubscribed from. If you send someone an unwanted email, even accidentally, and that person complains to their ISP, you may be blacklisted and your mass emails will not be delivered to anyone else who uses that ISP for a long time.
For example, let's say email@example.com unsubscribes from your list, and you delete him from the list. Later, you're uploading new names from an outside database or spreadsheet and you accidentally upload firstname.lastname@example.org into the list again. If you'd kept the existing "unsubscribed" status, that email address wouldn't be added to the list. Since you erased that status, though, Bob is going to get an unwanted email. At best, he's going to be irritated at you. At worst, he might report you to his ISP as a spammer.
Keeping the list of unsubscribed addresses within eNotify enables the module to perform a cross-check at the time of adding new email addresses to the list and helps prevent such mistakes and protect your sender reputation.