James Ellis Mattox was born to Merle Hodges Mattox and Dr. Nathaniel Ellis Mattox on May 18, 1924 in Lead, South Dakota. He takes his middle name from his Grandmother, Mary Ellis Mattox. She married Nathaniel Shrewsbury Mattox, the Grandson of Valentine Mattox.
He served in the Army during WWII, then went to medical school at North Western University. He married Caroline Elise Isely on June 14, 1949 in Evanston and then returned to Lawrence County, South Dakota. We was a general physician. He raised four children. He was remarried to Olivia. He died of a heart attack in Lead, South Dakota on June 20, 1997.