Many useful counter-terrorism lessons were learned in the late 20th and early 21st centuries. First, there was the growing affluence and lower poverty rates throughout the world, especially after World War II. This brought with it more terror groups and a proliferation of Islamic terrorists. Violent rebels (also known as terrorists) in the past had some of the same needs of the modern version. This included motivation, leadership at all levels and resources. This meant the means to fight and supplies of food and other necessities. Defeating these movements has not changed; you have to eliminate enough leadership or resources to cripple the movement sufficiently to eliminate it. You don’t hear much about the losers (there were a lot off them) but the winners got more coverage in the history books remained memorable far longer. All religions have some zealots who can sometimes get violent. Islam was no different but the radical Moslems had more impact because the religious dispute included the radicals demanding that most tech and knowledge not developed by practitioners of other religions be forbidden for Moslems. This dispute has been disrupting Moslem (especially Arab) populations for over a thousand years. It has also led to the Moslem world losing out on the many benefits (not just more powerful weapons) new tech has provided to the non-Islamic world. After World War II a lot of Moslem nations found themselves suddenly very wealthy because of valuable natural resources on their territory. This resulted in a lot more ambitious and very religious Moslems organizing movements to defend Islam from all those wealthier and better equipped (and armed) foreigners Moslems were encountering. This meant more Moslems were killed (for opposing the disruptive radicals) as well as a small, but growing percentage of foreign infidels (non-Moslems). There were always young men attracted to these radical movements but all that sudden affluence meant there were a larger number of older, wealthier and like-minded Moslems who were willing to help with cash.
The infidels were better equipped to deal with this growing source of terrorism and the counter-terrorism techniques developed continue to be refined and do the job better.
Two of the most obvious examples this can be seen in the defeat of the Palestinian terrorists in 2005, and the Iraqi Sunni Arab terrorists in 2008, as both movements were crippled by going after the support organization (recruiters, planners, bomb builders, scouts, guides, bomber handlers, security people) that makes these bomb attacks happen. There were other factors as well. In Israel, Palestinians were largely barred from entering Israel. That, in addition to the destruction of so many bomber support personnel, halted nearly all the successful Palestinian attacks inside Israel.
In Iraq, the situation was a little different. The bomber support personnel were being found and killed or arrested with greater frequency by 2006-7, and the Sunni Arab minority were under increased attack by the Kurds and Shia Arabs (illegal death squads as well as security forces). At that point the U.S. brought in reinforcements (the "surge") which finally persuaded the majority of Sunni Arabs to stop allowing Sunni terrorists to hide among, and recruit from them. The American roundup of terrorist bomber support staff then quickly eliminated the remaining terrorists. Both these cases demonstrated how a sanctuary (a somewhat safe base) was for keeping any Islamic terrorist group going for a Arab leaders that they had to either switch sides, or face extinction (exile or death) in Iraq. This double blow to the terrorists quickly led to a 90 percent drop in attacks. Without the Sunni long time.
Success in Iraq put the Americans on an equal footing with the Israelis in terms of gathering intelligence among the pro-terrorist population. The key to Israeli success was an informant network within the Palestinian community. The Israelis have hundreds of police and military operatives who can pass as Arabs (their families came from Arab countries shortly after Israel was founded in 1947). These Israelis speak fluent Arabic (with a Palestinian accent.) These agents dress as Palestinians and enter Palestinian areas and, backed up by regular troops, grab suspects and hustle them off, or kill them if they resist. But these agents also move about and recruit and run Palestinian informants. Many of these Palestinian informants are doing it for the money. Israelis pay for information. They also use other inducements (help with the bureaucracy, medical care, and so on). If that fails, they use blackmail and threats. Palestinian terrorist organizations have been unsuccessful in their attempts to shut down the informant networks, and many innocent Palestinians have died simply because they were falsely accused of being informants.
Actually, the Israelis gain a lot of information on terrorists via electronic intelligence work and UAVs that are constantly in the air over Palestinian neighborhoods. They seek to make the terrorists think that it's the gadgets, not informants, that brings in the most information. To the Israelis, inducing paranoia among the Palestinians is seen as a successful weapon. All this has helped keep the terrorists out of Israel for the last three years, something no one thought was possible.
In Iraq, the U.S. had very few military intelligence people who could pass for Iraqis, and those who could were mostly to almost always Iraqis, or the children of Iraqis, who had earlier migrated to the U.S. Using Iraqis as informant recruiters has proved difficult because the enemy, often veterans of Saddam's security services, are expert at intimidating and terrorizing Iraqis. This has made it difficult to keep the identity of informant’s secret.
The Iraqis have been able to make the Israeli tactics work, or at least work better than for the Americans. The problem here is that, all too often, Iraqis are easy to bribe. Money has been in short supply in Iraq for over a decade, and too many people are willing to sell whatever they got in order to make a buck. Even when there is a family connection (which is why recruiting several members of a family as informants is so useful), people will get sold out.
And then there's the religion thing. Nearly all the former Saddam intel people are Sunni Arabs, and there's little trust between them and the majority Shia Arabs. The head of the INIS (Iraqi National Intelligence Service) is a Sunni Arab, selected by the CIA because he had worked for American intelligence while Saddam was in power, and knew how to run an intelligence organization. While the CIA trusted this guy, most Shia Arabs did not. The Shia Arabs, using their control of another government agency, set up their own national intelligence agency.
The problem here was that members of both organizations can be reached by bribes, or threats to their immediate families. Worse, too many people, once bought, don't stay bought. The two agencies did not cooperate with each other, and the Shias were reluctant to work with Americans. It took American intel specialists a few months to get used to the way these things operate in Iraq, and the fact that they were the only ones who could operate in both Shia and Sunni areas. But despite it all, the basic Israeli emphasis on lots of intel, then lightning raids, and the acquisition of more intel (prisoners, documents, laptops) has remained the one successful approach to shutting down terrorist operations.
An analysis of post-World War II terrorism confirms that terror attacks depend on support staff to succeed, and that most terrorist cells, after their first few attacks, can carry out one or more a month. Oddly enough, most groups do not carry out more than a hundred attacks before they are destroyed or disbanded. To stop these groups from getting anywhere near a hundred attacks, you want to find them early, ideally, before they carry out their first attack. On average, you have about six months between the first attack and the second. That's when you have your best opportunity to take down the cell, and put their leaders and technical staff out of action.
The success of al Qaeda (“the Base”) and its radical offshoot ISIL (Islamic State in Iraq and the Levant) is that both stressed the importance of finding a secure sanctuary. Some countries, including some non-Moslem ones, have found it useful to offer sanctuary for some Islamic terror groups, in return for not committing terrorism in the sanctuary country, and doing the occasional mercenary mission for the host country.