I recently finished Subversives: The FBI's War on Student Radicals, and Reagan's Rise to Power by Seth Rosenfeld. It's a fascinating non-fiction book, and I feel like it contains a valuable lesson for writers of fiction as well.
The book is about the FBI under J. Edgar Hoover, and how it basically went off the rails and dedicated enormous resources to hassling hippies in the late 1960s. (They have long hair! LOOOONG HAIR!!!) Now, I knew that that had happened--I've read Steal This Book and Soul on Ice and a ton of other 1960s counterculture "classics" that to be honest are for the most part incredibly boring to the contemporary reader. (Yes. You smoked weed and got laid. That is so fascinating.)
But I'd never really gotten any insight into why it happened, other than Hoover was The Man, and The Man can't handle having his mind blown by young people smoking weed and getting laid, man! (I feel obligated to point out here that Hoover was probably gay, and some of the people he worked with quite closely for a long time were most certainly gay, so I don't think his problem was Puritanism.) Rosenfeld spent more than 20 years fighting the FBI in court for access to its files, so he has a lot of letters and memos and whatnot that provide insight into Hoover's thought process.
And a fascinating thought process it is! For starters, Hoover surrounded himself with people who thought exactly like he did--proof that the echo chamber existed long before the Internet. So by the time the late 1960s rolled around, assumptions like "Democrat = Communist" were widely accepted within the FBI, because it's not like anybody in the Bureau knew any Democrats or, God forbid, actually was one. In other words, there were no reality checks taking place, and no speed bumps on the road from Legitimate Security Threat Land to Crazy Town.
What was the legitimate security threat? Well, the Comintern was indeed a real thing, and in the 1930s and 1940s most Communist parties in the United States were actively managed by the Soviet Union and supported Soviet interests over American interests. In the late 1940s, people working for the Soviets were spying on the American atomic-weapons program. So that sort of thing was a totally legitimate area of interest for law enforcement, which is why the FBI got involved.
Unfortunately, as the level of Communist/Soviet activity in the United States waned in the 1950s and 1960s, the FBI just assumed it was better hidden. The way the Communists operated really helped stoke the paranoia--remember, this was a political movement that developed under the totalitarian regime of the czars. So Communists had a policy of 1. lying about who they were, and 2. surreptitiously taking control of organizations that were largely not Communist by having a small number of Communists enter the organization and take leadership positions.
The secrecy meant that if a group had few or no people in it who were openly Communist, you still couldn't be 100% sure that it was not a Communist front organization that would promote the interests of the Soviet Union. And indeed, Hoover and his people were, in general, 100% sure groups that were making trouble were Communist front organizations, even if the vast majority of the people in those groups were clearly not Communists.
So when the free-speech movement started at UC Berkeley in 1964, Hoover did not see a bunch of college kids agitating to pass out flyers on campus (yawn). He saw a Communist front organization (!!!). I was surprised about how sincerely Hoover believed this--I'd always assumed that the people making these sorts of allegations knew they were pretty ridiculous. But Hoover's underlings obediently produced a report saying that the FSM was a Communist front organization, and then Hoover himself was genuinely quite surprised when that report was discredited.
At this point, Hoover's thought process went like this:
Q. Why are these kids acting so weird?
A. Soviet infiltration!
Within just a couple of years, though, even the FBI knew that the Soviets had SFA to do with what was going on at Berkeley. Unfortunately, at this point they didn't care. Hoover's thought process had devolved to:
Q. Why are these kids acting so weird?
A. Who cares? They must be destroyed!
And that's when the FBI became an instrument of straight-up political oppression--hippies, peaceniks, Democrats, they were all subversives and all the enemy.
Do you see how much more interesting that kind of thing is that the simple-minded "Hoover is The Man" or "Hoover is evil" or "Hoover can't handle freaks"? It's a story, and a tragic one--a guy starts out in trying to protect people's freedom, but thanks to certain flaws in his character (an unwillingness to associate with anyone who does not agree with him; an unwillingness to adapt to change; a willingness to break the law to pursue an investigation), he winds up becoming quite possibly the most serious threat to that freedom. The slow decline in his ideals, the gradual sense that there are no rules that apply to him, the creeping belief that anyone who disagrees with him is evil--you can see how it could happen. You can relate to it even if you think you would have handled things differently.
It's so much more engaging than just having a two-dimensional villain plopped before you, along with instructions to hate him. Bad guys who are just bad--you know, one day they just decided to become evil, as you do--are such a wasted opportunity.