Think Tank Impact Assessment Tool

Clearly, assessing the impact of think tanks is not an easy endeavor to undertake given the various and conflicting actors, events, and politics involved in the policy making process. Despite the significant challenges in establishing a causal relationship between knowledge and policy, it is necessary for think tanks to understand and effectively respond to the growing chorus of questions being raised by donors, journalists, and the public about the role and influence of think tanks in civil societies and governments around the world.

Think tanks can employ a variety of metrics to assess their impact, including such measures as an increase in research and analysis they produce as well as to account for their contributions to the policymaking environment and civil society. McGann’s recent (2008) research has focused on developing a comprehensive assessment tool for evaluating a think tank’s impact. The impetus for this research, in part, was the apparent confusion that exists about the differences between outputs and impacts. In various studies and surveys that McGann has conducted over the years, researchers and think tanks responded curiously when asked about impact on public policy and how they measure it. The overwhelming response was to provide a list of research outputs (number of books published, conference held, web hits, media appearances, etc.). Outputs, however, are not the only way to measure impact.

The metric provided below is designed to serve as a catalyst for a discussion on how to effectively measure the impact of think tanks. It is provided here as background for the think tank ranking process in the hopes that it will help clarify the distinction between outputs and impacts. We ask that you consider the following indicators when contemplating the impact of think tanks:

· Resource indicators: Ability to recruit and retain leading scholars and analysts; the level, quality, and stability of financial support; proximity and access to decision-makers and other policy elites; a staff with the ability to conduct rigorous research and produce timely and incisive analysis; institutional currency; quality and reliability of networks; and key contacts in the policy academic communities, and the media.

· Utilization indicators: Reputation as a “go-to” organization by media and policy elites in the country; quantity and quality of media appearances and citations, web hits, testimony before legislative and executive bodies; briefings, official appointments, consultation by officials or departments/agencies; books sold; reports distributed; references made to research and analysis in scholarly and popular publications and attendees at conferences and seminars organized.

· Output indicators: Number and quality of: policy proposals and ideas generated; publications produced (books, journal articles, policy briefs, etc.); news interviews conducted; briefings, conferences, and seminars organized; and staff who are nominated to advisory and government posts.

· Impact indicators: Recommendations considered or adopted by policymakers and civil society organizations; issue network centrality; advisory role to political parties, candidates, transition teams; awards granted; publication in or citation of publications in academic journals, public testimony and the media that influences the policy debate and decision-making; listserv and web site dominance; and success in challenging the conventional wisdom and standard operating procedures of bureaucrats and elected officials in the country.

Beyond this qualitative assessment, an effective evaluation of impact should also involve NGOs, as well as members of the government and policymakers, to ascertain the degree to which they have utilized the grantee’s research output. This participation can be obtained through interviews, surveys, questionnaires, and focus group meetings, utilizing the Outcome Mapping which “moves away from assessing the products of an activity or a program to focus on changes in behaviors and relationships (outcomes) which can lead to changes.” Impact can be viewed as positive if it “changes the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly.”

Although this qualitative assessment is essential because it recognizes that policy impact can be successfully achieved even if policy prescriptions are not directly translated into actual policy, we recommend that this assessment should be translated into numerical rankings, thereby allowing comparisons with baseline data for effective monitoring and evaluation in the future.