Can a person be "in the world but not of it"?
Well, a writer must spend some time in the world to write about it, obviously. And I guess a writer such as J.D. Salinger, didn't have to spend too much time with people to write honestly and perfectly about the loneliness of being a fragile soul in a world full of loudmouths, blowhards, dullards and cretins. Seymour the damaged soul, traumatized in WWII, no longer able or interested in communicating with regular people, even his nail-lacquering wife. His characters all cry out for understanding in a loud, busy, cruel world that keeps on going on despite the fact that there are adults out there who never learned to be adults. They hurt like children hurt, that's why a lot of them like or deeply resent children. Jews can live like WASPs but they're still Jews and it makes life harder for complicated reasons.
I suppose a life of travel, luxe living in New York City, then getting hurled onto Utah Beach in 1944, he met a lot of different kinds of people. I guess he felt like he'd met enough.