Found the following, but without more info, this is just speculation. Could use more info on both James's so I would know I'm heading in the right direction.
in the 1850 United States Federal Census
Name: James H Miller
Age: 6
Birth Year: abt 1844
Birthplace: Tennessee
Home in 1850: Subdivision 20, Roane, Tennessee, USA
Gender: Male
Family Number: 604
Household Members:
Name Age
James W Miller 32
Hannah Miller 26
Terrence M Miller 8
James H Miller 6
Andrew P Miller 5
Martha J Miller 4
Polly Miller 0