Jason Isaacs, the actor who played the villainous Lucius Malfoy in six of the blockbuster Harry Potter movies, has some news that he knows not everyone is going to like.

“It’s a terrible confession to make: they weren’t that much fun to make,” he said on a recent episode of BBC’s The One Show. “It’s quite boring making big special effects films; however, the pleasures all come afterward. I see and meet people for whom their lives were changed by it, and still people reading it and sharing it with their children. Some people say their lives were saved by it, and I believe it.”

Isaacs noted that settling into a theater seat with his popcorn was a different story, though. He was truly touched.

“Even though I was in the films, when I’ve taken godchildren or nephews and nieces to the [studio tour] and the thing comes up and suddenly you’re in the Great Hall,” said Isaacs, who currently stars in season 3 of HBO’s The White Lotus. “Every time, I burst out in tears. It’s incredibly moving and overwhelming. There’s some magic that happened in those stories.”

Based on the bestselling books by author J.K. Rowling, the Potter movies, about a coming-of-age young wizard embracing his heritage, were phenomenally successful. They collectively earned more than $7 billion at the global box office, ranking as one of the 10 highest-grossing franchises of all time, according to Business Insider. The series of films also spawned gobs of merchandise — more movies in the three Fantastic Beasts prequel films, a TV series in development at HBO, as well as clothes, jewelry, toys, games, and even theme parks — plus generations of die-hard fans. Isaacs is well aware.

Want more movie news? Sign up for Entertainment Weekly‘s free newsletter to get the latest trailers, celebrity interviews, film reviews, and more.

The actor was a part of six of the eight total Potter-focused, live-action movies, beginning with the second, Harry Potter and the Chamber of Secrets, in 2002, then appeared in Harry Potter and the Goblet of Fire (2005), Harry Potter and the Order of the Phoenix (2007), Harry Potter and the Half-Blood Prince (2009), and both parts of Harry Potter and the Deathly Hallows, which came out in 2010 and 2011.

“Something happened, who knows why, when those ingredients came together,” Isaacs said, “and the soufflé rose and it created just love around the world and a sense of inclusion.”

Isaacs now appears as Timothy Ratliff in the third season of The White Lotus, HBO’s popular anthology drama. Ratliff is a wealthy Southern business mogul confronted by past shady business dealings and fearful that his secrets will come to light and endanger his family, played by Parker Posey, Patrick Schwarzenegger, Sam Nivola, and Sarah Catherine Hook.