The only ancestor I know of in the Civil War didn't have slaves either. My grandmother's grandfather walked from south Alabama to Montgomery to volunteer. From their he marched to Virginia, was taken as a POW in his first battle, stayed four years in prison camp (outside DC somewhere) and then walked back to south Alabama. My uncle found some papers on him, written on the bottom of his discharge from prison paper was a note that said he never got paid for the four years he was away from home.
Could have been Point Lookout; that's where my Gr-Gr-Grandfather had been imprisoned however unlike your ancestor he died and is buried there. It was an exceedingly horrible place; far worse than the much vilified Andersonville.
My ancestor was an older man when he volunteered for the Cavalry; already had a family and kids and a tobacco farm (but no slaves). I often wondered precisely what motivated him but humans being humans I'm sure that he had the same feelings and thoughts that I have about an increasingly intrusive and imperious FedGov.