Swanson was born in Canon City, Colorado in 1927, the only child of Oscar and Beatrice Gail Swanson. He attended Bullis Preparatory School prior to the United States Naval Academy, where he was graduated in 1949. He received a degree in Naval Architecture and Marine Engineering from the Massachusetts Institute of Technology (MIT) in 1953. From MIT, he advanced to the Naval Submarine School in New London, Connecticut.

After serving two years as a lieutenant aboard the USS Croaker (SSK-244), he earned the dolphins of a certified Submariner. Service followed at the Portsmouth Naval Shipyard in Kittery, Maine, the David Taylor Model Basin in Bethesda, Maryland, and the Bureau of Ships in Washington, D.C. He was then graduated from the Polaris Command School at Dam Neck, Hampton Roads, Virginia. An additional degree in the Program for Management Development at Harvard Business School was received in 1968. Tours on the staffs of the Commander, U.S. Pacific Submarine Force at Pearl Harbor, the Superintendent of Shipbuilding at Groton, Connecticut, Puget Sound Naval Shipyard in Bremerton, Washington, and Portsmouth Naval Shipyard, in Kittery, Maine, led to his appointment as Commander of the Pearl Harbor Naval Shipyard in 1972. Swanson retired from active duty in the Navy in 1975, becoming Deputy Director of the Department of Transportation for the state of Hawaii for four years, following which he served fifteen years in the private sector as President/CEO of Dillingham Maritime, managing Hawaii Tug & Barge Corporation's Young Brothers. Returning to the public sector, he became Director of Transportation for the City & County of Honolulu, and served as Commissioner of Honolulu's City Transportation Commission. Additionally, Swanson served as president of the Outrigger Canoe Club. In reflection, Swanson considered his greatest achievement to have been the realization of the Kahala Nui Continuous Care Retirement Community. "If we're lucky in life," he said, "a challenge comes before us that engages our imagination. It speaks to us of possibility, of need, and of adventure. Kahala Nui was just such a challenge for me."

He spent 16 years working with a team of Island leaders to establish the $184 million center. He was honored for his efforts in 2013 by both residents and staff. The Greystone Development Company of Irving, Texas, ultimate developers of the CCRC, said, "During Greystone's 30 years in business, we have never worked with a man who demonstrated a greater selflessness in his passion to serve, whose perseverance was infectious to everyone he encountered and who could demand performance with the skillful art of a gentleman."

Acknowledging his wife's support in creating Kahala Nui, Swanson said, "I am always grateful to Anne for being a part of every adventure in my life. She has certainly been a big part of this one." Swanson and his wife, Anne Beverley Wright, met on a blind date in 1947. He was smitten. They were married in 1950, in Richmond, Virginia. They have four children: Charles Beckman, William Bradshaw, Anne Beverley, and John Westwood; and, six grandchildren. In addition to being an avid and enthusiastic tennis player for the last 50 years, Swanson was involved in scouting for much of his life. He was an Eagle Scout who received membership in the Order of the Arrow, and was awarded the Silver Beaver Award for exceptional character and distinguished service. He was an active supporter of Hawaii Boy Scout Troop 1 for many years, sang in numerous church choirs, and volunteered locally with Meals on Wheels providing meals for those in need.

A memorial service in celebration of his life will be held at Kahala Nui on Friday, January 22nd at 4:00 p.m. Contributions in his honor can be sent to the Kahala Nui, Charles O. Swanson Endowment for Education, c/o Kahala Nui.