Broader Impact Statements: Are Researchers Thinking Broadly Enough?

shutterstock_210478717

Shutterstock 210478717

I’m working on a project focused on helping Georgia Tech faculty access the information and support they may need to help write more effective broader impact statements for National Science Foundation grant proposals. I started this work by doing some research of my own to find out what resources currently exist, and if there are examples of best practices.

I came across an interesting masters thesis, written by Sarah Wiley of The Ames Laboratory. She took a deep dive into a stack of NSF grant proposals to see what researchers were actually proposing to meet the broader impact criterion. It seems many are simply listing the things they already do – like teach, present at conferences and publish papers. Some provide more creative suggestions for sharing their research far and wide, but they seem to be in the minority.

I wanted to know more about this, so I reached out to Sarah – who graciously agreed to answer the following questions about her thesis:


  1. What is the NSF Broader Impacts Criterion is, and why do researchers need to know about it. 

Intrinsic to their funding proposal process, the National Science Foundation requires that Principal Investigators (PIs), the person primarily responsible for the research project, not only defend the technical merits of their research but also address the project’s larger scientific and societal value under the Broader Impacts Criterion (BIC). Grant proposal guidelines now stipulate that proposals must address separately each merit review criteria in their project description or risk having their proposal returned without review (NSF, 2013).

As they currently stand, “broader impacts may be accomplished through the research itself, through the activities that are directly related to specific research projects, or through activities that are supported by but are complementary to the project. NSF values the advancement of scientific knowledge and activities that contribute to the achievement of societally relevant outcomes. Such outcomes include, but are not limited to:

  • full participation of women, persons with disabilities, and underrepresented minorities in science, technology, engineering, and mathematics (STEM);
  • improved STEM education and educator development at any level; increased public scientific literacy and public engagement with science and technology;
  • improved well-being of individuals in society;
  • development of a diverse, globally competitive STEM workforce;
  • increased partnerships between academia, industry, and others;
  • improved national security;
  • increased economic competitiveness of the United States;

and enhanced infrastructure for research and education” (NSF Guide)

In essence, BIC is one massive science communication and outreach exercise carried out by NSF, with each grant contributing to the BIC goals.

  1. What compelled you to research the types of broader impacts PIs are (actually) proposing?

2There is a lot of talk in the science communication field about BIC and the debate swirls around whether scientists can/should/should have to/are responsible for articulating and fulfilling the broader impacts criterion. Yet, regardless of the outcomes of these important and necessary discussions, PIs still have to include BIC in their proposals. So, what are they actually doing? How are they addressing this criterion?

I wanted to get a look at the conditions on the ground, so to speak to provide a more pragmatic framework. As the criterion is currently phrased, researchers must address how their work, for example, broadens the participation of underrepresented groups in STEM or benefits society at large. Those guidelines feel awfully broad to me. So instead, here I reoriented the focus to broader impacts activities, that is, the actual activity that was fulfilling the BIC.

In providing insight into how researchers conceptualize Broader Impacts, I hope to help PIs better understand and work through writing broader impacts sections and science communicators and outreach specialists who want to track the trends, preferences, and state of current broader impact activities as they relate to academia and the public.

  1. How did you go about the research?

It was important to me to create a more manageable framework, and I turned to program logic modeling, which cuts complex projects into bite size pieces. The approach separates broader impacts goals, or outcomes, from the actual activities researchers, propose.

Long-term goals (outcomes), and the activities that contribute to their fulfillment (activities) are not synonymous. For example, a single grant that includes mentoring graduate students will not singularly “advance discovery and understanding while promoting teaching, training, and learning” but instead the proposed activity will contribute to this broader goal. This seems an obvious point but it is one that is oddly missing from the BIC literature. By conflating activities, outputs, and outcomes, confusion arises about what exactly is required under BIC, for what is the PI going to be held accountable, and how it is going to be measured. In short, what exactly are researchers proposing to do to fulfill BIC? What specific activities are being proposed to address the larger outcomes laid out by NSF?

It is important to note here that the set of proposals ranged from 2009-2011. BIC at the time encompassed five core long-term outcomes: teaching and education, broadening participation of underrepresented groups, enhancing infrastructure, public dissemination, and other benefits to society. Improved national security, increased economic competiveness, and increased partnerships between academia, industry, and others were introduced in 2013.

It is important to note here that the set of proposals ranged from 2009-2011. BIC at the time encompassed five core long-term outcomes: teaching and education, broadening participation of underrepresented groups, enhancing infrastructure, public dissemination, and other benefits to society. Improved national security, increased economic competitiveness, and increased partnerships between academia, industry, and others were introduced in 2013.

To look for answers, I worked with a set of over 80 NSF proposals which were anonymously submitted by PIs as part of a BIC campus study. Strengthening the Professoriate at Iowa State University (SP@ISU), a fantastic NSF-funded initiative here on campus that works to connect scientists and resources to better address BIC, collected the data set.

From this set, I broke the broader impacts statements by each reference to an individual activity. (I defined an activity as a structured, pre-planned action that shares, teaches, promotes, communicates or otherwise engages an audience in science.) Any mentions of the audience type, the resources used, and other pertinent themes were also recorded. Meaningful subcategories and categories were made from the recorded list of activities.

  1. What did you learn about the frequency, types, and ranges of activities PIs propose as broader impacts?

Across 87 proposals, a total of 458 activities were extracted. Five main activity clusters emerged from this set: Disseminating, Teaching, Training, Facilitating and Researching. The majority of the activities fell into the disseminating cluster. Of those, attending professional conferences, publishing in scientific journals, or making data available to peers accounted for 96 (21%) of the proposed dissemination activities.

By far, the majority of broader impact activities revolve around duties intrinsic to a university faculty member. Teaching courses, enhancing curriculum, presenting at conferences and publishing in peer-reviewed journals, and researching and training the in the lab are all part of a profile of a university researcher. Moreover, two main types of audiences were mentioned – students and research peers/faculty. Whether in the classroom or being trained in the lab, college-age students constitute the main audience.

Table1

  1. It seems like researchers are staying in the ‘safe zone’ by proposing activities they’re already doing in the course of their job. What about unique, new, and creative ideas? 

In my sample, there were some more unique proposed activities. Some proposed working with museums to create interactive displays (3) or being part of science festivals and competitions as mentors or judges (2). Others proposed working directly with the community through community advisory boards (5) or stakeholder meetings and open forums (6).

The internet and social media have opened up a whole new way for researchers to reach out beyond the academic community. Internet dissemination activities, including developing an app, building a website, updating/creating wikis, and disseminating through social media channels accounted for 36 activities.

On the other hand, there has to be a balance with out of the box ideas. Ultimately, researchers are including broader impacts statements as part of a funding proposal. Researchers are looking to be funded so how many out of the box ideas would reviewers accept? Could one propose a live tweet session or a Reddit AMA (see http://www.reddit.com/r/science/ for inspiration) as a broader impact? What about a science fact or fiction movie night (screening a Hollywood movie with a science/sci-fi theme and answering questions)? These are all viable science communication events, but will the NSF reviewers reward these types of outreach?

  1. Do the activities being proposed really make sense when considering the broader impact outcomes NSF has outlined, or should they be doing more/different things?

They do.

I think the interpretation of BIC has been quite narrow, encompassing mainly scholarly publishing, classroom teaching, and mentoring. That does not mean that these cornerstones such as teaching (96 activities) or training the next generation of researchers in the lab (94 activities) should be disregarded. These are important.

Is there room for more creativity? Absolutely. There is always room to be more creative and some have proposed more out of the box activities. Having the time, skills, and resources to implement those ideas might be another thing.

  1. You point out that the main audience for dissemination activities is still other researchers. Should we be concerned that communicating research more broadly isn’t the focus? (only 71 of 458 proposed tactics right?)

Correct, only 71 of the 458 activities were directed towards a broader, non-academic audience.

Should we be concerned?

Yes. If the point of broader impacts is to ‘broadly impact’, then it could be argued that only reaching out to the scientific community is not broad enough. However, I think it is vital to remember that all solid communication – whether media, advertising, events, a clever tweet –takes a lot of background work (ok, maybe not a tweet, but you get the idea). Planning, budget, time, a keen understanding of the views and needs of the audience, among a myriad of other things, are all required. Currently, we are asking PIs (and their funded teams) to perform these roles in addition to carrying out state of the art research.

I don’t know what the answers are but it is something to think about. If we want broader impacts activities to be the most effective (actively engage an audience with meaningful science) is it fair to put that solely in the laps of researchers?

  1. It also looks like when researchers did interact with the public their activities were slanted toward the ‘deficit model’ instead of an ‘engagement model.’ Is it just easier to push out information than to truly engage?

My hunch is that if it is not easier, it certainly is more familiar. It would be hard to dispute the very strong precedence of lecture-style teaching in science (or most of academia, for that matter). I am going to tell you what you need to know as opposed to telling me what you want to learn. Already in the broader impacts, PIs are set up to tell about their work to others, hence the ‘deficit model.’ Given these factors, I would think it would more comfortable to just push out information than to engage.

That said, there is room to grow. Reaching out to a wider (or different) audience can be very enriching. Engagement is difficult. It can sometimes be combative or open researchers up to uncomfortable questions. On the other hand, it can offer an entirely new perspective on a topic; thoughtful, honest discussion on issues vital to this country; and a way to connect on a different level. Engagement can have a lot of pitfalls, but there are a lot of rewards too.

  1. Did you learn anything about researchers measuring the outcomes of their broader impact activities?

None of the 87 proposals in this study’s data set described in any detail a measurable output indicator for any proposed activity.

  1. What recommendations came out of your research?

1If researchers are expected to engage a broader audience then they need to be provided with tools, opportunities, and incentives.

First of all, researchers need to be provided with the tools to engage a broader audience. It is important to strive not only to provide researchers with outreach tools and opportunities but also to provide insight into the needs and desires of the public. Outreach not only should focus on communicating to the public, but also communicating findings back to the scientific community.

Opportunities for outreach are also helpful. It was encouraging to see that where there were resources and programs available, researchers were taking advantage. For example, 18 proposed mentoring and recruiting activities working with the Program for Women in Science and Engineering (WiSE) an organization started on campus in the 1980s to bolster the participation of women in the STEM fields. Others proposed working with programs to encourage K-12 teachers to gain experience in the lab (NSF’s RET program was mentioned 15 times) or engaging undergraduates to conduct research through the on-campus honors programs (8), and the Experience for Undergraduates (REU) (20).

Lastly, (and I am echoing class from many others), the university culture would need to be addressed. It is pertinent to be aware of attitudes about outreach at both the university and scientific discipline levels which can serve as both enabler and barrier to engagement and can be discipline specific. The pressure to publish in order to fill the tenure requirements as opposed to focusing on science outreach means that researchers have little incentive to focus their time elsewhere.

  1. What are researchers doing well? What can they do better?

In my proposal set, I saw a lot of activities where researchers were tapping into existing resources. Whether on a university level (honors and STEM-awareness programs), community level (museums and libraries), or national level (RET and REU, both NSF programs), when the resources were there for researchers they seemed to use them. Making them aware of these opportunities could be a goal.

I’d love to see researchers reach out more. For those who do not feel they are comfortable with direct engagement with the public, providing opportunities to network with science communication and outreach specialists would be great. Here researchers can include a specialist on their grant who would then coordinate the broader impacts activities. The partnership could address the lack of efficient use of science and public outreach experts and would be beneficial for both scholars. A bit of a plug for science communicators? Sure, but we have a lot we can bring to the table!

Likewise, science communication specialists, outreach coordinators, and program planners can reach in and offer a hand to researchers. Build an entire community. Leaving the onus on researchers alone is not fair. Reaching the public isn’t just for researchers writing proposals; this should be an aim for all of us in science communication as well.

Editor’s Note: I also asked Sara if she explored the broader impacts proposed in funded vs. unfunded grants. She said that wasn’t part of this particular project. I think it would be a great follow-up research effort. Knowing which broader impacts ‘scored highest’ or were part of grants that were funded would be valuable. 


About Sarah Wiley:
SWiley2015

Sarah Wiley

Sarah Wiley graduated from Northern Illinois University with a Master of Arts in History and an emphasis on Southeast Asian studies in 2011, where her curiosity for science was piqued in a history of science and medicine class. She pursued her interest in her second master’s degree program and graduated top of the class with a Master of Science in Journalism and Communication from Iowa State University in 2014. She was a Science and Technology Policy Intern with USAID Indonesia for two years for the Department of State Virtual Student Foreign Service Program. Sarah currently works as a science communication specialist for the Department of Energy’s Ames Laboratory in Ames, Iowa.

Research Paper (Thesis):

 

 

8 thoughts on “Broader Impact Statements: Are Researchers Thinking Broadly Enough?

  1. Thanks for this interesting interview Sara.

    I particularly related to this section:

    “….However, I think it is vital to remember that all solid communication – whether media, advertising, events, a clever tweet –takes a lot of background work (ok, maybe not a tweet, but you get the idea). Planning, budget, time, a keen understanding of the views and needs of the audience, among a myriad of other things, are all required. Currently we are asking PIs (and their funded teams) to perform these roles in addition to carrying out state of the art research.”

    Did you review any proposals that requested a small section of the budget dedicated to commissioning the work of a creative professional (artist, designer, musician, etc) to accomplish these BIC?

    As you say, a PI probably sticks to what they know because that is how they are trained. If the expectation is to go bigger, broader, and more creative, then that requires collaboration with other professionals who aren’t necessarily PIs (but should be compensated for their time/expertise).

    Like

  2. This is not my experience with programs in NSF IOS. As both grantee and panel member for many years, I have seen the following come to pass: teaching, mentoring grad students and postdocs, and undergrads in the lab no longer constitute a sufficient BI plan. At one time undergrad research made the cut, but no longer. The problem with NSF is uneven enforcement. Program officers are still going their own way with this.

    Like

  3. Thanks for highlighting Sarah’s work. She eloquently confirms many things those of us who think about BI at the institutional and national scale suspect and have learned through experiences.

    I think it would indeed be interesting to look at funded and not funded proposals too, as such information may be compelling to PIs. It would also be interesting to see reviews across NSF programs and how they address broader impacts. I would caution that practices at NSF are changing with regard to broader impacts so any insights from such work may only have a short term relevance. In any case it would be good to establish a baseline.

    Like

  4. Reading this, aside from my general bias that all broader impacts does is dilute science funding at the same time as it dilutes the attention scientists pay to science, is that the broader outreach seemingly desired in the context of the interview is the thing least likely to make it into a broader impacts section. Namely, “Should we be concerned that communicating research more broadly isn’t the focus? (only 71 of 458 proposed tactics right?)”. Well, the most common way this is done is sitting down with the university PR rep and writing a press release when the research is published. And, if the press release has a good hook, then spending time on the phone (or in person) with reporters trying to balance giving them something to write about against overselling the research. So this is good, right? Maybe scientists should add this to their broader impacts section?
    Maybe, maybe not. The PR culture enveloping science is leading to increasing exaggerations of the significance of research, which seems to have spawned more deceptions and, judging from what you find at Retraction Watch, more junk in the literature making it that much harder to do, ahem, science.
    An interesting question related to what was studied might be, how much potential scientific effort was diverted by the broader impacts requirement?

    Like

  5. Pingback: Weekly Good Reads: Blind Experiments, Broadest Impacts, and Writing Explainers | The UnderStory

  6. Wow, this is really interesting stuff, I never would have guessed that broader impacts is largely being defined so narrowly. Such an important study, thanks for sharing.

    Like

  7. This was a really interesting article that I never thought I’d read. Our own funding is mostly NIH based so when I applied for an NSF GRFP, I was told it was pointless and very generally, outreach and science communication is not encouraged in my department. It’s interesting to see this from the perspective of the PIs who DO apply for NSF funding and have to grapple with the “pesky” BI requirements but likely have the same attitudes that people in my dept do. “Why waste the time, let’s just do the science.”

    I think more than anything, we should be emphasizing (and communicating this in a way that scientists are sure to understand – data) the benefits of science communication. I have, as a lowly graduate student, already benefited so much from trying to learn to communicate my science. If I can explain it to the public, then I can EASILY see why it would be important for it to get funding – it forces me to boil it down to the “Why should we care” points. It also has provided new perspectives and focus for my research. I genuinely feel I am a better researcher now because of my outreach and communication – not in spite of it, regardless of the time it has consumed.

    It also allows me to retain that most vital to my research – passion for DOING it. When experiments go wrong and I’m thinking, “why do I DO this?!”, I am rooted by the passion that has been kept alive by talking to people outside of my field about it. If I didn’t have this aspect to my career, I may not persevere.

    I think that’s what everyone is missing – how it will help and why it’s not just some requirement they have to satisfy.

    Like

  8. Pingback: Broader Impacts: We’ve only just started – We the Scientists

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s