In 1967, when Scott Walker was 24, he quit his band the Walker Brothers at the height of their fame–on their final tour the supporting acts were Cat Stevens, Engelbert Humperdinck, and the Jimi Hendrix Experience. Walker had always seemed uneasy with his role as a teen idol, and by tearing himself away from his adoring public to pursue an interior vision, he became a textbook example of the existential rock star.

He released four self-titled solo albums between 1967 and 1969, immediately gravitating to material deeper and darker than anything he’d been allowed to sing before. He began with English versions of Jacques Brel tunes, and for Scott 4 wrote all his own songs. When you first hear those records, his rich baritone, couched in pristine, layered orchestral arrangements, brings to mind hammy pop crooners like Robert Goulet, but once you understand what he’s singing about–isolation, madness, helplessness, hopelessness–the music’s profusion of interlocking patterns starts to seem outright claustrophobic. The longer you listen the stranger it sounds: lush and beautiful songs about desperate, miserable people, overwhelmed as much by their crowded psyches as their crowded apartments.

Of course, as Walker’s music grew increasingly complex, intimate, and harrowing, his commercial appeal dwindled. But these four are still the records for which he’s most widely revered today–after a decent fifth album in 1970, he released four more without any original material at all, including two forgettable country-flavored discs. By the time he reunited with the Walker Brothers for a few years in the late 70s, his chart career was on the wane, and since then he’s been putting out solo records at a rate of one per decade: Climate of Hunter in 1983, Tilt in 1995, and now The Drift, scheduled to come out later this month on 4AD.

The new album is the product of seven years’ work. The BBC recently broadcast Walker’s first TV interview in more than a decade, playing studio footage where he’s showing musicians exactly how to bang a metal pipe or slap a side of raw pork to get just the right percussion sound. The Drift is unmistakably the product of a powerful urgency, but it’s nothing like a teenager’s urgent desire to be understood, which is easily frustrated and just as easily spent. Instead it’s like a monk’s desire for transcendence, expressed in a steadfast commitment to work patiently, a little each day, toward a goal that’s hardly understood.

Walker is 63, but neither yields to the pressure to sound superficially contemporary nor revisits the feel of his canonized late-60s material. The Drift is so idiosyncratic that only his previous record can provide a meaningful context for it. With its sinister undercurrents and occasional eruptions of metal and industrial noise, Tilt is one the most shocking and unsettling records I own, and its often lurid surface can make it hard to appreciate the songs themselves. Walker’s new disc redeploys the avant-garde collage approach of Tilt in the service of his classic albums’ emotional impact.

The overall mood is of horrible suspense, noirish and futuristic. Field recordings create the illusion that each song is happening somewhere particular, not just in a studio, but the album mashes together those small ambient noises–and the tiny private sounds of the body, like the crackling of saliva as Walker whispers–with grandiose, sustained washes of discordant strings, subverting a healthy mind’s sense of scale. A rock band might try to get started, then give up, but one player at a time will carry on, as if to show the others the way. Layers of electronic whizzes, blips, and chirps erupt unpredictably, sometimes blending seamlessly into the mix and sometimes poking out grotesquely, like the background hum of a refrigerator or a computer accidentally exaggerated by a hypersensitive microphone. Brief, poetic radio-play-style dialogues enter and exit, suggesting a story but never providing any context for it. And then there’s Walker’s singing: he cavalierly stresses his words on the wrong syllables, deforming them to fit the odd, dilated melodies, but his voice is still the same rich, crooning baritone, almost operatic in its grandeur.

It feels like The Drift is only a record by happenstance. It could just as well exist in any other medium–say, as a wall-size painting or a dense experimental film. It’s impossible to process from the perspective of any musical genre, and tough to get used to even when approached on its own terms. After a few listens, the sonic shocks around every corner begin to seem inevitable–each one belongs exactly where it is–but they don’t get any less surprising, since the album’s amorphous structure makes it so hard to anticipate them.

With its restlessly shifting backgrounds, The Drift reminds me of Talk Talk’s last two records (without the hypnotic grooves) or John Cale’s Music for a New Society (without the clearly delineated pop songs). Though disorienting at first, its slippery pastiches eventually work to center the listener’s attention on the singing–a strange way of arriving at one of the key features of pop. Often the music seems like little more than a series of textures to resituate the vocal line in different spaces.

When you notice a guitar, it sounds distinctly like U.S. Maple (or like the Magic Band with all the blues boiled out), but it’s not as though there’s a recognizable rock lineup churning away beneath the obscure tangle of drones and effects. It’s often hard to identify any instruments at all, in fact, and in the few moments when a standard-issue band does take the lead, that sound is so pointedly just one of a vast assortment at Walker’s disposal that it seems more like a sampled track than a live group.

The accompaniment occasionally falls into an off-kilter burlesque bump reminiscent of the Get Hustle or Love Life, but in contrast to those bands’ discrete bursts, this is more a prepositional music–it’s always between states, on its way elsewhere, never settled. It’s as though Walker has written songs without verses or choruses, only long strings of bridges. When he loops a lopsided pattern for a few bars, the repetition is always a discharge of tension, a brief reprieve from the music’s unrelenting instability. It’s a sound that seems like a slightly sexy, slightly silly put-on when younger bands try it, but in Walker’s hands it’s truly heavy.

The violence and dream logic of the lyrics also reinforce the fractured, drifting, cubist aesthetic of the album. The lines are generally just a couple syllables long, easy enough to decipher one at a time but tough to piece together–Walker doesn’t seem willing to do any more for the listener than establish a range of possibilities and permutations, and refuses to impose ideas about what connections might exist between his images. Every song seems to pass fluidly from one perspective to another, so that even when you’re sure something terrible is happening it’s hard to tell who are the victims and who are the perpetrators. On the chorus of “Cossacks Are” Walker sings, “With an arm / Across the / Torso / Face on / The nails / With an arm / Across the / Torso / Face on / The pale / Monkey / Nails,” impalement imagery that’s echoed later in “Buzzers”: “Polish / The fork / And stick / The fork / In him.” Not even animals are safe: in “Jolson and Jones” two men try to out-bluster each other, taking turns shouting, “I’ll punch / A donkey / In the / Streets / Of Galway!”

Gone are the tales of particular, everyday people struggling to survive in dank 60s tenements–the people in The Drift could be living in the 10th century as easily as the 20th, staring up at the indifferent stars in the godless heavens and shaking under the weight of that terrible epiphany. The album’s press materials say “Jesse” is about 9/11 refracted through Elvis’s relationship with his stillborn twin, set to a demolished version of “Jailhouse Rock,” and insist that the lyrics to “Buzzers” conjoin the Balkan conflict of the 90s with the evolution of the horse, but you’d never know that if you were left to parse the songs on your own. “Hand Me Ups” links adulation to punishment by connecting hand claps and spanking, and Walker’s lyrics seem to be about celebrity–which, at least in the most obvious interpretation, makes the audience the culprit. But the same rhythmic claps also closely echo the steady footfalls from “Jolson and Jones” two tracks earlier, making it hard to tell which parts are born of which song or what the motifs are supposed to mean.

The Drift is as tightly packed with information as any record I’ve ever heard, but it still leaves much of its sonic space open and unstructured. Walker needs that space to generate suspense: Is the donkey about to start screaming, or is this where the distorted Daffy Duck comes scolding?

As a young man Walker connected with his fans by providing them the same sort of vicarious catharsis that every pop singer relies on. He told them how he felt, and they felt it with him–or more likely for him. He was all the more seductive because he was so clearly reluctant to comply with the expectations built into the role, acting instead as though he’d been somehow mystically anointed to be a pop star and had to play the part no matter how burdensome he found it. (Between them Bono and Michael Stipe have gotten a lot more mileage out of that routine than Walker ever did.)

But Walker now refuses that simple connection with his audience. His recent music is gnostic and ecstatic–qualities that arise from its meticulously chaotic form, not from his performances. There’s no spontaneity or improvisation in his singing, but his songs demand audience engagement in a way that’s more like sparse free jazz than any variety of pop.

The listener’s satisfaction no longer comes from identifying with Walker, but instead from an imposed alienation. In ordinary pop the performer’s moment of ecstatic release–the point of greatest intensity, often the chorus–is also a catharsis for the audience. Walker’s approach inverts this relationship, so that during his moments of ecstatic release the audience experiences sustained tension. Only when he comes down to earth or falls silent does the listener feel a sense of payoff or resolution.

This virtually guarantees that Walker will lose most of his potential audience, but it makes for a more powerful connection with the folks who stick it out. Even in the likely event that the full significance of Walker’s cryptic gestures eludes you, you’re eventually forced to concede that something profound is happening. Damn, this guy’s really going through something was my own thought. He doesn’t care if I understand it, and he might not even understand it himself. The level of trust Walker places in the unfiltered expression of his darkest inner corners–to the exclusion of conveying any tangible message–is what proves he’s achieved a truly monkish intensity of devotion to his art. He seems unconcerned with his audience, thinking more of how to get something out of his head than of whether anyone will pick up on it once he does.

Walker clearly has faith in the ability of music to exist at several different levels simultaneously, and in fact The Drift depends on that–alongside its immediate physicality, it has the primal depth of the songs an aboriginal tribe might’ve used to pass down its history. But even more important than this faith in music is his total respect for it: rather than approach it as a set of genres and categories, he treats it like a boundless force of infinite variability, malleable enough to fit whatever shapes his subconscious imposes on it.