The starting point for scheduling a meeting usually requires many options. Allowing the Organizer to delete the least popular dates easily would help attendees focus on the remaining most commonly available dates, and perhaps adjust their individual schedules accordingly. Simply adding a trash can icon next to the date option like you have for participants is ideal!
Also, when editing the existing options, the site wipes out all of the slots that were already listed and the Organizer has to start anew. It would be much better if it left the slots already identified, and allowed the Organizer to add/delete specific ones.