That is correct, I don't exercise frequently and I'm not particularly big or ripped. I stand 5'8" tall and my weight fluctuates around 150 pounds, sometimes up to 160, which gives me a moderate build. My body fat is probably around 10%, although it's been years since I bothered measuring it.
Relating to Jessica's comment though, I am very active. I don't own a car so walk pretty much everywhere, typically logging 3-5 miles per day between walking to and from my office and doing errands. When I need to cut trees for making bows it's common for me to walk the 3.5 miles to land I have permission to cut on then carry the 100+ pound log back over my shoulders, sometimes making two or three trips in a day if the tree I felled yields more than one usable 8-foot log. I also heat my home with a wood stove, so carry in 15-20 loads of wood in each week during the winter, with each load weighing 40-50 pounds. I have also been training in mixed martial arts since the mid-1990s, and continue training a couple days each week, for a little more activity. I also enjoy swimming, and try to swim in Lake Champlain near my home at least a couple times each week, even in winter. I also cold train (do a google search for Wim Hoff to learn about this), so that's part of my motivation for swimming throughout the winter.
I don't count anything in the above paragraph as 'exercise', since it all serves more utilitarian purposes than just working muscles. So when I say that I exercise about once each week for brief periods, please don't get the impression I spend the rest of my week sitting on the sofa scarfing pints of Ben & Jerry's. Not the case at all.