When you delete an email in Horde, you may see that the email is still there. The email is greyed out and strikethrough. The email is only marked as deleted.
To permanently delete the email from your account, click on Other to get the drop-down list of options. Then click on Purge Deleted.
This is the default setup of Horde. You can change the preferences of Horde to move deleted email to the Trash folder. To see how this is done, click here.