And there was always a part of him that dreamed of playing in the same building those guys did, a short ride up I-77 from his hometown of Rock Hill.
Thursday, he walked into that building for the first time as a member of the Panthers.
"I always wanted to, growing up as a kid," Gilmore said. "Some things you can't control. But I always thought about it as a kid, and you always dreamed about it.
"But it happened, and I'm happy it happened."
So are they.
The Panthers were so eager to acquire the 2019 NFL defensive player of the year that they traded for him, sending a 2023 sixth-round pick just to secure his services. The Patriots were expected to cut him, but that word got out to try to flush out offers. And rather than risk seeing him land somewhere else, Panthers general manager Scott Fitterer was happy to pay the small price.
Another team could have always traded for Gilmore, but if he had gotten cut, the Panthers could have saved that future pick.
"I think so," Gilmore replied, when asked if he'd have signed here anyway if he made it to the market. "This is a place I always wanted to play. Looking at the team and what they've got, the potential. Obviously, you have to put in a lot of hard work to get to where you want to go.
"For sure, I think it was a great opportunity I would come here."
Of course, he knows the city. He lives here, a few doors down from Fitterer, actually. And with his background, he knows his Panthers history.
But he hadn't even met head coach Matt Rhule in person late Thursday afternoon when he talked to reporters, so he was short on specifics. He doesn't know what number he'll wear (his New England 24 and South Pointe High School and University of South Carolina 5 are taken). He even stopped short of saying with certainty that he'd be able to play in Week 7 against the Giants.
He's on the physically unable to perform list at the moment because of a quad injury suffered last year, but he said he was running and lifting and doing football drills.
"I'm just taking it a day at a time, and talking to the coaches, and go from there," he said.
Gilmore has always been low-key, but even though this is a cause for celebration for him and his family and friends, he knows he's here to work.
Asked if there were particular things about home he looked forward to, he took it back to football.
"I'm just looking forward to playing ball, doing something I love," he said. "Obviously, community work, I'm big on community work, but the number one thing is trying to compete and get on the field and do what I do and have fun."
He does have some insight into at least one of his new teammates. Rookie cornerback Jaycee Horn has been under his wing, and Gilmore said Horn visited his home before the draft. They've spoken in these hectic last 24 hours, and Gilmore said he's disappointed it will be some time before they get to play together (because of Horn's broken foot).
"He's a great player; it was unfortunate for him at the beginning of the season," Gilmore said. "But he's got a bright future, and I'm going to come in and try to teach all those guys what I know and try to come in and make plays myself.
"I talked to him. He reached out. We keep in touch here and there, talk about different things we see, different receivers. He's a student of the game, and I like that."
Gilmore was very matter-of-fact about the last day.
When asked how he learned of the trade, he said, "Bill called me and told me," referring to Patriots coach Bill Belichick.
"It's been a crazy 24 hours," Gilmore said. "From sitting in the meeting room, to talking to Bill, to getting traded, to flying down here today. It's been a crazy 24 hours, but it's over now, and I'm looking forward to the future."
About that future. He said he's not worried about the fact his contract expires at the end of this season. He was working toward a new deal with the Patriots prior to the trade, but said he's content to play things out here.
The Panthers have said they'd like to keep him (along with 25-year-old free-agent-to-be Donte Jackson), but that wasn't on Gilmore's mind. When it takes two years to go from being defensive player of the year to traded for spare change, it gives you a certain perspective.
"That's how it is," Gilmore said. "Every year starts over. You've got to earn it every year. That's how I look at it. I turn the page each and every year and try to earn what I want. That's how I look at every situation."
Still, you could tell Thursday, that there were moments when he was looking around his new surroundings, that the kid from Rock Hill was still amazed he ended up in this place.
"I've been playing ball since I was 6 years old. I still remember running around the cones with my dad and him throwing me the ball," he said. "A lot of memories here.
"I grew up watching the Panthers play. I was always a Panthers fan. I knew all the players that played before me, and I'm happy to be back."
View photos of Gilmore from his years with the Patriots (2017-21) and Bills (2012-16).