I agree with you that it is difficult, but I don't think that means writers shouldn't try. People from the groups you mention often write about how happy it makes them when their stories are told, and told well, and it is possible to do so even when not belonging to the groups yourself. (The writers I single out for praise here are particularly noteworthy examples.)

As a woman, I notice when female characters aren't present, or aren't written well, and I can only imagine the need for representation is stronger, the more one is marginalised or ignored in mainstream fiction.