When I organised my recent meeting, I have received 288 responses from 155 attendees and many of those were duplicates.
So when accepting the invite and when they choose a time and date, the only option should be the email address, not the name. In my case, they have used their names as well as email addresses. Or you can ask for the name and email address so we can avoid duplicates.
Another thing is having their meeting time and date displayed when hovering the mouse over their name.So I don't have to scroll to the end to see the time and then to the top to see the date.