Amanda Butterfield Genealogy

As I have progressed throughout my research, I have tried to find my families in every census year they would appear. So far, thanks to Ancestry.com, this has been quite successful. However, multiple families are missing from the 1860 census. George Madison's family, his wife's in-laws, and some of his brothers' families are not to be found. I am almost certain they are still living in Jackson County, but just to make sure, I have done national searches for them. What is curious is that multiple families are missing. They are farmers, so they live in rural areas, and from what I have gathered, they live next to each other.

My current theory is that their area or neighborhood was missed by the enumerator either due to unclear enumeration districts, or bad weather which has been known to prevent census enumerators from traveling to rural houses. Any other thoughts as to why these families would all be absent from the 1860 census?