15

To give context for the question I have volunteered to do some work for a local charity. The work they do is literally lifesaving and almost certainly something the government should be responsible for, but the reality is that various charities are filling the breech.

What I will do for them is data processing, they have messy data and they want some estimates for things they didn't record at the time. They will use the results to apply for funding for further work.

In one of our meeting the topic of negative findings came up, for instance, what if their work isn't cost efficient compared to charity Y? The immediate response was a sort of joking "Oh we will bury it!".

I was too uncomfortable to say something at the time, but I absolutely do not want to bury any findings. I understand that other charities may be burying bad findings, and refusing to do so may put this charity at a competitive disadvantage. Having read about what this has lead to in the pharmaceutical industry I know that this is probably more serious than anyone in the charity realises. In the long term it could completely undermine the causes they work for.

Unlike scientists in the pharmaceutical industry this work is not my primary research topic, and I'm not being paid for it. My continued employment at my university is not threatened anything this charity do. So I could just read the them riot act, however I suspect if I do that they will say "Oh sure" and then ignore it later. I need to be diplomatic and convincing. They care about the work they do, but they also care about their own organisation, and asking them to do this is good for the work but bad for the organisation.

How can I best approach this?

  • 3
    What are you/they going to do with the findings (other than apply for funding)? Will they be published? It’s in their best interests to act on any negative findings and improve, but it may not be in their best interests to publicly broadcast their weaknesses immediately (before acting to improve on them). Publishing takes time and they can take action in the interim. – Pam May 2 '18 at 19:18
  • @Pam, they will be published in informal ways. Think info graphics, and funding bids. I guess I'm mostly worried about them convincing the counsel and other funding bodies that something they are doing is an excellent solution when the data says it's actually not that great. – user17415 May 2 '18 at 19:55
  • I think you are discounting the difficulty of doing useful research in this space! Are you a social scientist? What is your research background? I would be shocked if you could make any definitive conclusions from post-hoc data comparing impact or cost-effectiveness of two organizations. Because everything will be a grey area, you should have fewer concerns! – Dawn May 2 '18 at 21:47
  • What country are you in? – jpmc26 May 3 '18 at 2:03
  • "Academic integrity?" You probably haven't met many Academics, have you? :p – gented May 3 '18 at 9:55
11

Ask for another meeting. Explain politely that you were very concerned by the "bury it" remark. See what they say. With a little luck, it'll end there.

If you're not satisfied, you can ask for a letter explaining how they will handle any negative results and make your continued volunteer work contingent on getting a letter that's satisfactory to you.

If you don't get it, walk. The inefficiencies you're worried about will eventually come to light, and you don't want to be tarred with that brush.

Edit: The remark from Pam to the question has very good advice for conducting the meeting I suggested. A positive approach early-on is quite likely to lead to a satisfactory conclusion, without need for letters, etc.

| improve this answer | |
  • 3
    I think the most reasonable thing to request is that any negative findings are acted upon, not buried or publicised. They’re not academics but they have got one on board to provide rigour and objectivity, which is a good indicator of good intentions (or over confidence). But clarification is needed, if only to reassure the researcher that their findings (good or bad) will have some positive impact. – Pam May 2 '18 at 21:12
  • 1
    Pam is right. You might also look at the new book out by Gugerty and Karlan. It thinks about the multiple kinds of research that can be useful to nonprofits. It might help to clarify your objectives and concerns. – Dawn May 2 '18 at 21:48
4

Pam has a great point in the comments.

Different publications have different purposes - marketing materials, grant applications, etc. serve the purpose of advertising and attracting funding, not adding to the academic body of knowledge about best practices. They should focus on positives.

An internal report is an appropriate place to show negative results. It can guide strategy and planning, hopefully steering the organization to a more effective path.

Maybe they partner with another organization to adopt more effective practices. Maybe they do an internal review and make program adjustments. Maybe, after they've had a chance to course-correct, they make a peer-reviewed publication about the best practices, showing the negative results. Maybe they find confounding factors that weren't accounted for in your analysis and decide the conclusions weren't correct.

Do talk more with them and make sure there is a mutual understanding about appropriate courses of action for different results. Present possible negative results as learning opportunities, and indications of areas that need additional analysis. Maybe you help them set up automated regular metrics, so they can track their own performance and improvement.

I don't know the depth of work you are planning to do, but don't presume that (small) negative results are damning. The work and rigor of analysis is very different for an infographic vs a program evaluation.

| improve this answer | |
1

I have worked with and mentored recent hires from academia in both government and private organizations. They often have similar kinds of concerns. This is the kind of advice I typically give them:

Fulfill the Request Honestly

First, your stakeholder asked you to perform some analytical task. The most important thing is that (if possible) you return the kind of value they requested. This is basic customer service - give them what they asked for.

For example, if they asked you to estimate how long the average child spends in a foster home you should respond with:

"The average child spends X days in our foster home".

Feel free to flesh out your conclusion as much as you like (are there different sub-populations with interesting and different trends?)

Provide Appropriate Caveats

During your analysis, you will likely become aware of limitations in the data or your analysis. Disclose them to your boss - but only if they are material to the organization's decisions.

For example, the hypothetical foster care analysis might have been limited because data for an entire subset of children was unavailable (maybe all of March's records were wiped out, or the organization didn't record arrival times when kids were escorted in after normal business hours).

This sounds like a big deal! Would knowing that these records were not included in the analysis likely lead to a different conclusion than assuming they were? If so, disclose it.

Talk About Resolutions

Here's the thing with mentioning problems: you were probably hired to provide solutions, not problems. When you disclose limitations or problems you should focus on something actionable. How can the organization solve the problem? What would it matter?

"I ran the numbers on the average time kids spend in our foster homes. We have quite a few kids arriving after hours in emergency situations, and we don't record when they arrive. I'm concerned about this because those kids may spend far less time in our foster home than other kids. We could be over-estimating. Why don't we talk to the case managers and see if we can improve this?"

Data quality is likely only important to your stakeholders if it will improve their business process. If they push back, connect data quality (or any other concern) to their operations:

"I understand keeping track of these records may seem unimportant, but we use the this number when we build financial forecasts. Right now we are overestimating how long each kid spends here, so we are probably overestimating our costs also. We could tighten that up and save the Board a lot of stress if we improved our record keeping."

The Boss Decides What is Important

Finally, the function of executives or managers in an organization is to set the priorities of their organization. It is entirely up to them whether what you disclose is important or not. Respect that they have this authority and that you don't.

If you followed my above steps, you should have a pretty solid case. You would have given them the information they wanted, shown them the limitation, explained why the limitation harms their operations, and provided a solution.

They may decide that your solution is important and they will look into it. They may decide that the fix isn't worth the cost. Or they may disagree that it's really a problem. Respect their decision.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.