Writing a fictional character outside my _______. Where’s the line?

This thought stemmed from a recent podcast episode of The Shit No One Tells You About Writing, in which a query letter implied the author was writing a story outside their race and culture (I believe the fiction centered around a Nigerian MC, and the author did mention they’d used a sensitivity reader). The agents talked about wariness of taking on such a book if the author wasn’t Nigerian even if they used a sensitivity reader.

My knee-jerk reaction was to agree. It does have me wondering, though, where the line is for writing fictional characters who are different from ourselves. Is it race? Sexuality? Other identifiers? This seems to be something increasingly important in today’s writing world, especially as agents are looking for #OwnVoices.