What are the federal and Tennessee state laws on video taping others?

Today I was in my hometown in Tennessee and went to my old house, knocked on the door, and asked to see if I could see the inside, he declined in a rude manner, and left without any trouble. I visited an old neighbor on a culdasac behind my old house. They mentioned how weird this man was, but I didn't believe them. Sure enough, as we were leaving, we saw the man in his backyard filming us. He video taped us as we exited, following us for about 130 degrees. It was very clear he was videotaping us specifically. He was in his own yard, and we were never on his property following him declining giving us a tour. Just out of sheer curiousity, what are our rights in this case?