Jim Gaffigan Shares Thanksgiving Rule His Mom Enforced Growing Up (Exclusive)

Jim Gaffigan had to follow one strict rule when celebrating Thanksgiving as a child.

Ahead of hosting the Big Brothers Big Sisters Of America 120th Anniversary Benefit on Nov. 19 in New York City, the 59-year-old comedian revealed to PEOPLE that his mother “would not allow us to eat anything” before Thanksgiving dinner.

“Growing up, it was like you were starved by Thanksgiving,” Jim said. “Particularly if you slept in and you missed breakfast. She’s like, “Then you have to wait. You have to wait because you’re not allowed in the kitchen.’ ”

Now that he and his wife, Jeannie Gaffigan, are hosting Thanksgiving alongside their five children — Marre, Jack, Katie, Michael and Patrick — Jim made it clear that he wasn’t going to be anywhere near as strict as his mother once was.

“I don’t kid myself if there are going to be any rules that are going to be observed by children,” he quipped. “I guess what’s good about Thanksgiving is it does focus you to kind of contemplate gratitude. Because I think human beings are kind of dumb and we need these reminders, kind of like we need Mother’s Day and Father’s Day to kind of like, ‘Oh yeah, that’s right. Maybe I should thank these people.’ ”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

While he doesn’t dare implement any rules for Thanksgiving in his household, Jim recently shared further insight into life as a father of five during a panel at the Tribeca Film Festival.

“Children are evil,” he teased back in June. “I joke around and say, parenting is the most important thing I’ll fail at, and there is something to that. But, I think that if human beings remembered how difficult and complicated and impossible, teenagers are the species which cease to exist.”

“I think that we forget that parenting is similar to the entertainment industry, filled with enormous amounts of humiliation, and you’re biting your tongue a lot,” he continued. “When kids are younger, there’s a cuteness to it, but when your kids are teenagers, they’re not conscious of it, but there’s a cruelty that you’re exposed to.”

The following week, Jim told PEOPLE that even getting his kids out of bed can become a battle of its own.

“I feel like getting teenagers out of the house or out of bed on a weekend is a game in itself,” he explained. “So the game is like, do you blackmail? Do you motivate? Do you inspire? Or do you just go full crazy?”

“And for me, there’s the balancing act of, you don’t want to lose your s—,” he added. “But you also have some sense of self-respect. You know what I mean?”