For the moment, only the Organizer / Creator can add or remove other attendees. 

 

In the future, attendees / participants will be able to request adding/inviting additional people to an existing hangout for Organizer approval, or request removing/disinviting existing attendees for Organizer approval.  Same goes for requesting time / location changes; at the moment, only the Organizer can change these details, but in the future we will introduce the ability for attendees / participants to request / suggest changes to these details as well.