Wes Bergmann

With twoChallengewins on his résumé,Wes Bergmannheads toAll Starsseason 3to try to pick up another victory.

“I signed up to go on a vacation, and then when I showed up it was all the best guys that had ever been on the show in one group,” Bergmann, 37, tells PEOPLE. “I was like, ‘This feels a little bit like a bait and switch.’ You look at the competition and I have an easier chance of just waiting until [The Challengeseason] 38 and winning it than I do of winningAll Stars.”

The new installment of Paramount+‘sAll Starsrequires past contestants to have qualified for or won at least oneChallengefinal. The cast, which Bergmann calls “very intimidating,” includes heavy hitters like Brad Fiorenza, Derrick Kosinski,Mark Long, Jonna Mannion, Veronica Portillo, Darrell Taylor andJordan Wiseley.

The producers delivered especially tough daily challenges for this fierce batch of competitors.

“This was as big of a budget and as creative of a shoot as they’ve ever done,” Bergmann says. “Every day that you show up is another day that you could die.The Challengenever rests.”

THE CHALLENGE: ALL STARS: ALL STARS, season 3

Bergmann considers season 3 participants Nehemiah Clark, Melinda Collins and Fiorenza, 41, his “squad” and thought he’d be off to a solid start heading into the game with a group of allies.

“But anytime you have a good start, the rest of the eyes are kind of looking at you,” Bergmann says. “I knew there was going to be somewhat of a target on my back, but I find out very quickly that there wasn’t a target on my back — I was the only one where I was in literally everyone’s crosshairs, including friends to a certain extent. If I knew on paper exactly how badly the mass majority of that cast didn’t want me there, I would have never come.”

Bergmann thinks other competitors target him out of “jealousy and insecurity.”

TheReal World: Austinalum doesn’t even think he should qualify as one of the show’s biggest threats. “I’m an average-size person that’s also aging. And as far as I’m concerned, I should be in the middle of the pack. Like, how old do I have to get before maybe you just consider me one of the people in the crowd?” he wonders. “I’m up there just because I’m wily.”

Wes Bergmann The Challenge 35

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Despite finding himself a major target even among a group of vets onAll Starsseason 3, Bergmann admits he didn’t try to get on anyone’s good side.

“I don’t think it’s much of a spoiler to say that I didn’t make amends with my rivals, but instead, just made it worse,” he teases. “I do that because I love making their lives miserable, but I also know that the fans love to watch me make their lives miserable. So I did it for everybody.”

Bergmann’s pestering goes beyond the show, too. Even though his top frenemyJohnny “Bananas” Devenanziodoesn’t appear on this season ofAll Stars, Bergmann promises, “I’m going to bully him for the rest of my life” because “we’re family.”

“I’m going to make his life miserable as much as possible, within the confines of the law, for the rest of my life,” continues Bergmann, the producer of the competition seriesThe Blox. “But no one outside of my family is allowed to bully him, because he’s my family. He’s just the most hated member of my family. There’s a lot of respect between me and him, but g– d—, he is so annoying.”

But Bergmann prefers to stay out of the online back and forth that takes place between cast members between filming the series.

Wes Bergmann

Even though Bergmann likes to distance himself from most of his costars and feels “I have more of an uphill battle on every season that I show up,” he says he plans to continue doingThe Challenge"when it lines up with my schedule."

The Challenge: All Starsseason 3 premieres on Paramount+ this Wednesday.

source: people.com