INTRODUCTION
It’s one thing for a foundation’s leaders to be able to say, “We fund programs that do great work.” It is a very different thing to be able to say, “Our money is having the impact that we want it to have.” How can a foundation better understand the true results of its grantmaking?
The Annie E. Casey Foundation The Annie E. Casey Foundation, headquartered in Baltimore, Maryland, is the 15th largest private foundation in the United States with assets of more than $3 billion. Founded in 1948, its mission is to foster public policies, human-service reforms, and community supports that meet the needs of today’s vulnerable children and families. It ranks 19th in the nation for charitable giving, and expects to distribute $229 million in grants in 2007. |
This was the question that the leaders of the Annie E. Casey Foundation (Casey) were asking themselves when, in 2000, they committed to measuring results across all of their grantmaking. Their progress towards this goal—and in particular, the efforts of Bruno V. Manno, senior associate of the Annie E. Casey Education Program—illustrates one organization’s efforts to become accountable for achieving the results to which it aspires.
BACKGROUND
The Annie E. Casey Foundation was established in 1948 by Jim Casey, founder of UPS. Originally created to fund a small camp for disadvantaged children in Seattle, Washington, Casey grew substantially in the 1960s and 1970s, becoming a significant supporter of foster care and neighborhood-based services for children and families through grantmaking and direct services.[1]
In 1983, Jim Casey passed away, leaving Casey additional resources. With these funds, the board of directors expanded the location and scope of the Foundation’s direct-service work. The board also expanded Casey’s mission, dedicating more than half of its endowment to grantmaking. Consistent with its legacy in child welfare and belief in the potential of neighborhood-based services, the Foundation sought out agencies and other organizations in targeted communities. The board’s theory was that supporting a suite of complementary services would have more potential benefit for recipient families than would investing in organizations across a large geographic area, or supporting just one or a few types of organizations.
The Foundation soon was making grants in support of a wide range of organizations and services to reach beneficiaries in multiple geographies. Some of this grantmaking was organized into “major initiatives”—multi-year, multi-million-dollar investments in specific geographies and populations within them. For example, the “Making Connections” initiative, launched in 1999, brings together government, local businesses, social service agencies, health care providers, community-based organizations, and other funders in designated areas throughout the United States to help disadvantaged families become more self-sufficient and successful.
Influenced by Jim Casey’s attentiveness to performance, and also steeped in the performance-driven culture of UPS (the majority of Casey directors are still UPS executives), Casey leaders monitored the core results of these major initiatives closely, setting and honing indicators that would give them a sense of how their grants to individual organizations were “adding up” to strengthen the target neighborhoods. In the case of “Making Connections,” for example, they tracked indicators including the percentage of households in initiative neighborhoods with employer-provided family health benefits, households with children that report earned income and one or more adults employed, and children reading at or above proficiency level in third or fourth grade.[2]
But in 2000, Casey leaders found themselves wanting to know more about how their efforts were adding up across all of their grantmaking. Outside of the major initiatives, the Foundation was mostly tracking process results. These data provided a big-picture look at the scope of its grantmaking and a catalog of general outputs (the number of people served by a given grantee, for example). But importantly, they did not establish a clear link between these outputs and outcomes for target beneficiaries: Were the neighborhoods Casey was investing in stronger, safer, and more economically healthy? Were the children staying in school? Graduating from school ready to succeed at college and in life?
Casey’s leaders would need a different approach if they really wanted to understand the results of their current grantmaking and be more deliberate about future funding decisions. As a result, they committed to adopting a results-based measurement approach across the organization. Importantly, when they made this decision, the Foundation’s senior managers also explicitly determined to include grantees in the development of the new measurement system. They believed grantee input would be critical to ensure that the Foundation was asking the questions required to receive meaningful data. They also felt that grantees from across Casey’s program areas would adapt to any new reporting requirements more readily if they were given a chance to contribute to the formulation of those requirements.
FIRST STEPS
Casey’s board and senior leadership were deliberate about pursuing a new approach, sending program officers to conferences about performance measurement between 2000 and 2002, and having staff attend training sessions about how performance measures could factor into their budgeting processes.The board also formed a performance measurement committee—a subcommittee of directors—which would be accountable for delivering the new approach.
It was important at the outset to reach agreement about what needed to be measured. Casey staff began this task by working with an outside consultant to solicit opinions from a small group of grantees. Subsequently, they engaged with consultant Mark Friedman to develop a framework for articulating strategy and measuring results.[3] Specifically, they identified the objectives they wanted to achieve, the barriers they faced, and the actions they could take to overcome those barriers. Based on their belief that there are three ways—impact, influence, and leverage—in which a single grant can contribute to overcoming barriers, they chose to classify results in these three categories.
A grant could have direct effect on beneficiaries (impact); it could affect the behaviors of people not directly touched by a grant (influence); and it could be used to build or attract additional support beyond Casey’s direct reach (leverage). Casey staff acknowledged that some grants could be expected to generate all three types of results, whereas others might be focused on only one.
At the same time that they were developing the framework, they also decided to adopt a “found pilot” approach inside the Foundation. Several Casey programs already had shown significant interest in measuring results. By identifying any lessons learned at the program level, decisions about measurement foundation-wide could be better informed. Additionally, the participating programs could benefit from sharing information and experiences across program boundaries. A “Results Working Group” made up of the officers of these programs and staff across all Casey functions was created for this purpose.[4]
As Patrick McCarthy, co-chair of the Results Working Group and a senior executive at the Foundation, said, “We knew the goal, and we knew that there were some pieces in place. We wanted to start building on what there was, as opposed to starting from scratch.”
Prominent among the programs tapped for the pilot initiative was the Annie E. Casey Education Program, led by Senior Associate Bruno V. Manno. Manno hadbeen investigating ways in which he could track the Education Program’s performance since first joining Casey in 1998, consulting with colleagues inside the Foundation, soliciting grantee input, and engaging with several consultants. (In lieu of internal staff, Manno works regularly with part-time consultants—roughly three at any given time—who serve as informal staff.)
AECF Education Portfolio Snapshot
The Annie E. Casey Foundation’s Education Portfolio invests nearly $6 million annually in K-12 education, providing financial support and other resources to nearly 50 grantees, with grants ranging from $10,000 to $350,000. Grantees include schools as well as national and community organizations that provide the supports and services needed to ensure that young people graduate from school prepared for adult success.
|
In part, Manno was driven by a desire to understand more fully the impact of his grantmaking. But he also wanted to use the Education Program’s results to garner support for education issues beyond the Foundation. As he put it, “Many consider education philanthropy the quicksand of the giving arena— once you’re in it, you’re under the control of forces beyond your power to influence things that will lead to a positive outcome. But it’s always struck me that if up front we had a keener and more precise sense of the results or outcomes we wanted to accomplish, we might actually reinvigorate in education philanthropy a sense of purpose, realism, and genuine accomplishment that is now missing.”
MANNO AND THE EDUCATION PROGRAM: STRATEGY FIRST
Manno’s starting point had been to articulate the vision and strategy of the Education Program: the goal towards which impact, influence, and leverage would contribute, and the ways in which these three categories of results would combine to achieve that goal. Exactly what did he want to accomplish along these dimensions and why? How would that occur through Foundation activities, specifically its grantmaking?
“In order to measure how we were doing, we needed to be as clear as we could possibly be about what we intended to do,” Manno said. “So it was a reverse-process of sorts, with the concept of measurement driving our thinking about what results should be.”
Manno’s grantmaking had always had an implicit logic, but he knew that he needed to make that logic explicit and to test it. To that end, he engaged in a year-long, iterative thought process informed by conversations with grantees, colleagues throughout Casey, and Mark Friedman. The resulting product was a formal expression of the rationale behind the results that the K-12 Education Program sought, using the framework that the Foundation had adopted. His logic began with his vision for a core result:
“One day, all young people in tough neighborhoods will achieve the aspiration their families have for them: to graduate prepared for adult success and well-being in the worlds of work, family, and citizenship.”
Manno and his colleagues also articulated the three critical barriers to achieving that vision, and their consequences. In their words:
“Young people are not ready for school, and families and young people in distressed neighborhoods have too few quality educational options that meet their needs. As a result, too many of these young people are often trapped in schools that have failed year after year to educate them for adult success and well-being.
“Families and young people in disadvantaged neighborhoods lack strong connections with schools and community supports and services. As a result, families and other community residents seldom come together to define the kind of schools they want; analyze why their local schools fail to provide such schooling; and work as partners to demand better outcomes and better futures for their children.
“Policymakers, civic leaders, and the public lack good information on effective policies and practices and possess insufficient political will to be effective advocates. As a result, there is not enough support for creating quality options for young people that foster robust connections between families and community supports and services that produce strong results so young people can graduate prepared for adult success.”
In expressing the Education Program’s specific role in overcoming these barriers, he articulated the results for which the program would hold itself accountable.
“First, the Education Program aims to generate an impact by investing its resources in ways that contribute to measurable direct improvements in school outcomes in the neighborhoods and jurisdictions in which it works. Second, the Foundation uses promising and proven results from its investments and the work of other organizations to influence the behavior of four key audiences who have the potential to help the many children and families not touched directly by the Foundation’s Education Program: other funders, civic and other education leaders, policymakers, and families. Last, the Education Program leverages its own investments, expanding the scope of change by influencing the sources of financial support for education improvements.”
At this point, Manno was confident that the Education Program’s strategy was complete. The task then became figuring out how to hold the program accountable. What information was he going to need from grantees? What could grantees report on? What would they report on? How could Manno use that information to answer his own questions about performance?
AN ITERATIVE PROCESS
Manno knew that asking grantees to document their results was likely to have an enormous effect on them. While he was confident that his grantees’ strategies were aligned with his own, he suspected that they had not explicitly stated their strategies in a fashion that enabled measurement of their results. Therefore, he suspected that they would likely have to go through a process similar to his own. Further, tracking and reporting on results—depending on the complexity of the requirements—might strain grantees’ administrative capacity. Finally, if grantees perceived new measurement requirements as pressure from Casey to focus in certain areas, they might redirect their program activities at the expense of their own missions. Manno did not want that to happen, believing that grantees were in the best position to articulate their own strategies and program focus towards achieving them.
With those concerns in mind, Manno began to engage grantees in informal discussions about the logic models that supported their own work and the information that they would use to measure their progress. As he had suspected, in many cases, it was clear that the Education Program’s focus on results would necessarily motivate grantees to clarify their own goals. In some cases, this focus would also cause grantees to initiate much more rigorous measurement programs than they had in place.
As Gail Hayes, a Casey senior fellow who works closely with some of the Education Program’s grantees, reflected, “Grantees needed to spend time thinking through the logic model, asking questions and answering them before putting any technical measurement framework in place.”
To Manno, this early feedback signaled that he needed to be proactive in educating grantees about results-based measurement. Grantees would need this general understanding to be able to comply with reporting requirements going forward. Moreover, this training would allow them to inform his own ongoing process, helping to clarify the data they could and could not provide. To this end, he invited several grantees to Casey headquarters to talk about the work he had been doing with Mark Friedman, and to get their input on how the Education Program could measure its results through the results generated by grantees. He also gathered grantees for several training sessions on the concept of results-based measurement.
Manno continued to engage other grantees in discussions about results-based accountability and measurement during site visits to their locations. Over the course of their regular interactions to administer and monitor grants, Manno and (at times) his consulting staff talked with each grantee about the topic, and asked them all to begin communicating more systematically on their results.
Specifically, they asked grantees to tackle the following questions in their Interim and Final Reports to the Foundation: What are you doing? How well are you doing it? What difference does it make in terms of impact, influence, and leverage? As Manno put it, “This was the performance mantra that we wanted people to start thinking in terms of.” But at this early stage, Manno did not impose much in the way of requirements. The idea was to asses the information that was available, so Manno left much to the grantees’ discretion.
The first round of results revealed that grantees were all “on board” in terms of their willingness—and even eagerness—to assess results in a meaningful, measurable way. However, the actual data Manno was receiving did not help him clarify his sense of grantee accomplishments or needs.
Further, it was nearly impossible to aggregate the data to create a unified statement of the Education Program’s results. With such a diverse group of organizations in the grantee pool—albeit all working towards goals that were aligned and often integrated—it would be a challenge to find the common reporting denominators. It would be even more challenging to process and present the data in a way that would be compelling to the sector at large and beyond. As Manno put it, “The big picture looked like the Tower of Babel. We needed to keep in mind the endgame. We needed to keep asking ‘What do we want this knowledge to do for us?’”
DEALING WITH APPLES AND ORANGES
At this juncture, Manno received an offer of help from Harvard Business School lecturer and senior researcher Stacey Childress, who had heard of his work through a colleague in the nonprofit sector. Childress, who was teaching a course on social entrepreneurship, offered four of her MBA students. Their work for him would be credited as a project for her course.
“These students were the ones who did the first version of the results template,” Manno explained. “Their work was preliminary, but it was an important step. We were considering several existing measurement tools, but I didn’t want to start with something that was already there and then be forced to fit the kinds of things we were thinking about into it. This was uncharted territory. We needed something that would be created to be uniquely suitable for our purposes. And I wanted it to be more organic and grow out of what we were doing. These students were able to provide an outside look—from a very different perspective—that helped us think about how to design a measurement template that would gather information in a user-friendly way and also support aggregation.”
During this period, Manno also began to work with the Bridgespan Group, asking them to help him craft an approach to engaging grantees more productively in measuring results. Part of the Bridgespan team’s work entailed identifying relevant experiences of other foundations. This effort revealed limited but valuable insights. Chief among these was the value of using templates to guide data collection in order to aggregate results, and the need to provide direct support—for example, possibly providing consultants to work with grantees on site—to ensure that grantees had enough time and capacity to comply. Few foundations had developed robust measurement processes. Even fewer could aggregate grantee results to articulate their program performance as a whole.
Combining his early rounds of data collection with observations of other foundation efforts and the work of the HBS team, Manno began to realize that he would need to drill down another level of detail with grantees, creating his own detailed template that would guide their responses. Understanding that their results would always be “apples and oranges” at some level, Manno concluded that detailed templates were necessary if information were to be collected in a way that would enable aggregation.
Manno’s continued work with Bridgespan supported the creation of this template and an approach to helping grantees implement it. To this end, Bridgespan staff conducted a series of in-depth interviews with grantees, assessing their abilities to provide detailed reporting on their organizations and activities, and also collecting additional feedback for the process Manno was creating.
The interviews revealed a strong independent desire among grantees to create results-based cultures in their own organizations. However, some of the interviewees also expressed concerns about what regulated reporting would mean. As one grantee leader said, “Giving us specific metrics to measure would make the results less relevant. In order for them to be relevant, we would need to determine which metrics to use.” Another said: “It is horrible trying to fill out a matrix if nothing applies to you.” And, as a third explained, “We’re a think tank, and a small organization, without the resources to invest in results measurement. To ethically and efficiently reflect our results, we would need the Foundation to invest in a third-party assessment of the public policy environment.”
The interviews also revealed a desire to learn more about Manno’s philosophy on measurement and performance, and the desire to receive more direction with regard to reporting results.
Finally, they revealed a high degree of grantee satisfaction with their relationships with Casey and Manno. Being included in the process of pursuing results-based accountability had strengthened their ties to Manno and to the Foundation.
Manno responded in several ways. For one, he revised the Education Program’s Letters of Agreement to include increasingly specific reporting guidelines. Broad questions such as “What are you doing?” were followed with specific parameters designed to guide grantees’ answers. For an example, please see the appendix entitled “Sample Letter of Agreement.”
Manno also began to work with his regular consulting staff to finalize a template that grantees could use to report results. They wanted to be as disciplined as possible with requirements—to allow for aggregating results—while also being responsive to the uniqueness of individual grantees. To this end, they returned continually to the issue of what would be nice to have, versus what they needed to have, against the “cost” to have it. They also solicited feedback from select grantees; Manno made a concerted effort, however, to shield grantees as much as possible from the internal iterative process. As he said, “We’ve tried to keep the chaos associated with developing a new process inside the four walls of the Foundation.”
The first iteration of the template, in use as of this writing, consists of a toolkit and a spreadsheet. It includes a “menu” of reporting choices, from which grantees can select depending on their activities. Selections at the highest level trigger an additional set of inquiries. For an example, see the appendix entitled “Excerpt from the Washington Scholarship Fund 2006-2007 Results Report.”
Manno also wrote a document entitled “A Road to Results,” which explains the Education Program’s strategy, commitment to results-based accountability, and approach to measurement.[5] As Jon Schroeder, coordinator at Education Evolving, a grantee, noted, “The documentation that Bruno shared with us along the way, including various drafts of ‘A Road to Results,’ has ensured that grantees understand the ‘why.’”
At this time, Manno also considered the possibility of an annual convening of grantees. The rationale behind convening was the idea that grantees would learn from each others’ experiences with measurement, documentation, and (potentially) subsequent program adjustment. Grantee interest was there, but offsetting that interest was a concern about returns on the time investment; grantees expressed the thought that “sharing experiences” wouldn’t be enough of a motivator. As Manno said, “I came to believe that the conversation we would have on the results approach would be much richer if we waited a bit. We could then talk not only about the theory and framework of what we were doing, but also about the implementation issues and problems individuals had encountered in trying to make the results framework useful to our partners and to the Foundation.”
On this topic, the Foundation’s leaders also weighed in, ultimately expressing the thought that they wanted to support the work of other program areas in refining their approaches to measurement before pursuing such an event, and agreeing that waiting might allow for a more valuable exchange of information down the road. The convening idea was placed on the back burner.
EFFECT ON GRANTEES
A series of follow-up interviews with several grantees, conducted in late 2006 and early in 2007, confirmed that Manno’s approach towards results-based-accountability has been well received and grantees have appreciated the fact that they have been included throughout the process. As one grantee said: “It’s an honor to be asked to contribute to the process. It reflects the fact that Bruno values my input, that he thinks I’m a leader, and that my opinions matter.”
Grantees indicate also that Manno’s efforts to include them without over-burdening have been successful. As Sally Sachar, former president and CEO of the Washington Scholarship Fund, noted: “I participated in two rounds of testing and feedback, including going to Baltimore and providing feedback on the toolkit. The sessions were very straightforward and grantee focused, designed to reach us where we were, as opposed only to piloting something for a larger project. And the process of my involvement was relatively orderly and efficient—it felt like good use was being made of my time.”
Importantly, the Education Program’s efforts appear to be having a ripple effect on grantees’ own efforts to hone their strategies. With regard to the grantee Education Evolving, for example, as far as Manno can see, “The process has helped them to clarify what they’re doing. Their most recent proposal to me reflected a much clearer sense of what their theory of change is. I’m not going to claim that we’re the only ones who have helped them do that, but I hope we’ve contributed.”
And as Bob Pavlik, assistant director of the Institute for the Transformation of Learning at Marquette University, put it, “We are focusing our thinking on the end in mind, and how we can modify our activities to better achieve that end. Now we set honest, possible goals.”
Grantees have also said that as a result of this new approach, they have a better sense of how they fit into the “bigger picture” of organizations in their area that support the same target populations. As Jennifer Brown, chief program and operating officer of the Washington Scholarship Fund, explained, “As we think about services to provide to our families that were maybe outside of our core mission but were necessary for their success, as a result of this process, we’re starting to look outside of our own organization to other nonprofits that can provide those services—at no cost to us—and that can enable us to leverage the dollars we have. So one of the benefits—maybe not a planned benefit—has been the new connections we’ve made with other organizations that are locally serving the same populations we are. This approach has a ripple effect, then, because we not only gain, but other organizations do as well. They’re also walking away with that heightened perspective.”
Kimberly Cambridge, director of Beacon Technical Assistance at the Youth Development Institute of the Fund for the City of New York, has a different take on the “bigger picture” perspective. As she explained, “What the approach has done for us is to help us look more effectively across the sector and see how we fit into the larger public policy context. We have more of a sense of where we’re contributing on the larger scale. We’re funded specifically for technical assistance that’s related to organizational capacity building for the New York City ‘Beacons’ [school-based community centers]. And as an intermediary, we were always asking ourselves narrow questions, such as ‘What are we doing with our sites?’ Now we start with the broad outcomes. We’re asking ‘What do we want the ultimate effect of the support we provide to be? What about the staff who participate in meetings and training events?’ Interestingly, our ability to look at the big picture has improved our work. We’re better, now, at anticipating need and following up with support.”
Regarding the administrative impact, grantees report a learning curve, in terms of complying with the reporting requests. Predictably, the process gets easier each time. As Sally Sachar explained, “Now that we’ve done it once, it does not require a significant amount of time. Every six months, several members of our staff spend a portion of a day to collect our measures. One person aggregates this data. What’s interesting to note—and important—is that we are able to leverage this time spent on measures, because we are able to use the product (a table with clear indicators and activities) with other funders and partners.”
And, in some cases, grantees have been able to use Casey’s reporting requirements to improve and inform their relationships with other grant-makers. As Sachar explained, “We use the report that we generated for Bruno in other venues. When we reported to the U.S. Department of Education, for example, we used the same measures. We also use it with other grant applications. The document is simple, concrete, and speaks very clearly to what we’re trying to do.”
EFFECT ON THE EDUCATION PROGRAM
Reflecting on the changes that have resulted from his five-year focus on strategy and results measurement, Manno said, “Perhaps the greatest change to our grantmaking has been driven by the thinking that we did on influence. As a result of thinking through the specific audiences that we wanted to reach and ultimately influence, the actions that we wanted them to take, the messages that they would have to ‘hear’ and the messengers that would have to carry them, we were able to be much more intentional about the organizations and projects that we would fund. Even further, I began to think about the value of being even more proactive than we had been in the past in guiding these influence projects to ensure that they align with these audiences, messages, and messengers.
“This thinking about influence also came together with our thinking about impact to result in a renewed commitment to focusing in a select number of geographies. As we articulated the importance of neighborhoods in achieving our vision for kids in tough neighborhoods, and also our desire to influence others to do the same, we realized that we would need to be able to provide evidence of the validity of this approach. We could only expect these success stories to emerge if we focused our resources in supporting a select set of geographies with our resources and attention.”
Of his own relationships with grantees, Manno believes that the process has changed the locus of discussion for the better. “Our conversations used to be activity related, or process related,” he said. “It was ‘Oh, what work are you doing.’ It was about what the activity looked like, and that’s where it stopped. Before, we would be vague, saying that we hoped a grant would contribute to some broad agenda. But it’s absurd, for example, to think that a $50,000 grant is going to improve test scores. What it might do is keep kids off the street in an after-school program and there’s nothing wrong with that. Now the conversations are about results. We can all be more positive about what we can accomplish versus what we want to accomplish. We can also be more explicit and realistic.”
EFFECT ON CASEY
With regard to the Foundation at large, Manno’s work within the Education Program has contributed to several important shifts. As Donna Stark, director of leadership development and co-chair of the Results Working Group explained, “I think people throughout Casey were hungry to know the difference we were making. We were looking and ready to develop methodologies and tools to answer that question for ourselves. Now we ask, ‘Am I spending this money in the right way? Am I making any sort of difference for the people we want to be making a difference for?’ Bruno’s work is affecting how the organization does its business. It is transforming our budget, and the way we work.
“What this process has gained for us is increased clarity on what success can and should look like. It has introduced a discipline into our thinking that was not previously evident in any kind of way. Six years ago, I might have thought this way, but no one would have known. I could have been operating on a hunch. I could have been making grants with good people I’ve simply known forever. This discipline means building a case for the work that we’re doing. It means making things visible as they never were before, and that is transformational. It is creating a collective vision for the organization.”
McCarthy agrees. As he put it, “One of the things we’ve begun to see is a much more explicit description of the targets that people are trying to achieve—tying more closely to the strategies that they’re funding. So the disconnect that you often see between aspirations and what people are funding has been narrowed.”
Stark notes that the example set by the Education Program and other pilots have quelled concerns that results-based measurement in general could stymie creativity, or limit risk-taking. “Our own behavior quieted concerns of that nature because nobody was stopped from trying innovative practices. The Foundation is deliberate in allowing for risk-taking. Sometimes the question is ‘Why do you think this will make a difference,’ and not ‘What is the difference you are going to make.’ The important thing is getting the expectations right.”
McCarthy also noted a visible difference at the board level: the directors’ ability to engage more deeply and effectively—in light of the new kind of information they’re being given—with a greater emphasis on strategic issues versus individual grant decisions. “The reports to the board from different areas of the Foundation are increasingly sharing a common language and style. They’re also shorter, and clearer,” he said, “making it easier to connect what we’re all looking at with the concepts of impact, leverage, and influence.
“I don’t want to overstate this,” he continued. “We always gave the board data. But before, when we’d go to the performance committee, we would spend time helping them understand what we’d been doing with regard to results measurement. And now we’re reporting on and discussing the results themselves.”
“You don’t need to review every grant,” McCarthy explained. “What you do need to do is get clear on what you’re trying to accomplish—say, with child welfare. And then you need to be able to look at child welfare grantees as a whole, over time. We have found, now that we’re leading with results, that the board is better positioned to raise questions with us that help them with strategic issues. They can ask us why something worked, or didn’t work. This approach has leveled the playing field for all of us.”
Administrative effects are also evident. In parallel and informed by Manno’s efforts and other “pilot” participants in the results working group, Casey leadership is honing its vision for the systems and processes necessary to support results measurement across all programs and aggregation at the foundation level. For one example, all Letters of Agreement to grantees will formally establish results measures. For another, each program officer will be expected to create and maintain a document—called a Foundation Investment Summary—that captures their investment strategy and measurement approach. Foundation leaders will use the document to assess strategy and results at the highest levels; program officers will use it to guide the development and maintenance of their own strategies and measurement programs.
Progress is visible. For the first quarter of 2007, 65 percent of the letters of agreement Casey sent to grantees contained explicit results-based performance measures, up from 21 percent in the second quarter of 2006 (and 13 percent for 2006 as a whole). And, as Patrick McCarthy noted, if the Foundation can report between 75 and 80 percent, “that will mean we’re about at ‘full.’” (For some grantees—for example, those for which Casey simply sponsors a table at a charitable dinner—performance measures make no sense.)
Going forward, the challenge is addressing the field. The next chapter is to improve the Foundation’s data and information systems so that the knowledge being gathered can be disseminated effectively, and contribute to the work that other nonprofits are doing outside of the scope of Casey and its grantees. As Donna Stark put it, “We want to understand what it would take to gather this information in the field and aggregate it internally. And we want to be able to talk to the field.”
AN EVOLVING APPROACH TO AGGREGATION AND STRATEGY
Manno has been “at this” for nearly five years, and the work to measure, document, communicate, and adjust to results continues. Grantee implementation of the new letter of agreement templates began in spring of 2007. Alongside of template implementation, Manno has been investing in other means to aggregate the results of the Education Program’s grantmaking.
Specifically, Manno worked with a consultant to create a report that detailed the characteristics of the 35 Education Program school grantees, so that he can compare performance over time and determine whether and at what pace the gap between white and non-white student achievement levels is closing compared with district and state averages. Detailed current and historical results data was not available from these grantees, so Manno used information found in publicly available databases and followed up with phone interviews with school leaders. Drawing on these data, he created a “dashboard” summary chart for each school showing performance trends; state summary pages compare these trends among schools, and also with district and state trends.
“This was our first attempt to think about how all of these schools are doing,” he explained. “That’s not to say that we’ve caused any changes in the schools, but these are schools in which we’ve made investments. We wanted to take the benchmark year for each school—the year in which we made our first investment—and see what has been happening. Clearly, the value of this kind of report will increase the more we do it. But it has given us a nuanced perspective of what is going on across our grantees in different states—and that is something we did not have previously.”
Additionally, Manno has commissioned a series of reports that will begin to aggregate results across grantees, and will also present what he is learning about investment strategies in targeted areas (schools in general, the use of vouchers, or technical assistance, for example). These reports will focus on results over the past two years, but will also consider the bigger picture of eight years of Education Program investing under Manno’s leadership.
Summary of Key Learnings The work that Casey has done towards creating and implementing a results-based approach to accountability suggest some lessons for other foundations seeking to be more purposeful about results. Chief among them:
• Leadership Commitment: A results-measurement system will have a greater chance at success if its leadership at the board, foundation and program levels is willing to invest significant time, energy, and money to support its design, creation, and implementation.
• Strategy Before Measurement: A clearly articulated strategy is critical groundwork for the creation of a results-measurement system.
• Grantee Collaboration: Strong relationships and significant strategic alignment with grantees can facilitate the difficult process of determining measures and measurement processes. Further, ensuring that grantees understand the logic supporting the system supports collaboration and compliance.
• Long-term, Concerted Effort: Building a robust and meaningful measurement approach requires years of effort at all levels of the foundation, in conjunction with grantees.
|
APPENDICES
Excerpt from a Letter of Agreement
October 26, 2006
Grant Number: 204.0556
Sally Sachar President and Chief Executive Officer The Washington Scholarship Fund, Inc. 1133 15th Street, NW Suite 550 Washington, DC 20005
Dear Ms. Sachar:
We are pleased to inform you that the Annie E. Casey Foundation (Foundation) has taken action to continue our support to The Washington Scholarship Fund, Inc. with a grant of up to $XX for the period beginning July 1, 2006 through June 30, 2007.
This grant is intended to support the administration of the District of Columbia Parental Choice Program (Program).
Bruno Manno is the Foundation staff person responsible for the management of this grant.
Description of Work and Products
As specified in the proposal submitted to us, we understand that The Washington Scholarship Fund, Inc. will:
- Implement the integrated case management system for families and students, including the Parent Empowerment Group;
- Undertake recruitment of new Program participants, inform them and other Program participants about available schools, and educate all of them in the application and admissions process, including assisting families with verifying financial and resident eligibility requirements and the family and student renewal process;
- Help families locate additional sources of information and support, including forming additional partnerships with other community organizations to do this; and
- Include data documentation and information collections processing systems.
Description of Accomplishments and Performance Measures
- What are you doing? Number (e.g., individuals, organizations, etc.) served through the activities and/or services (e.g., technical assistance, training), organized by audience(s) (e.g., families, students, educators, policymakers and funders);
- What are you doing? Number and/or types of products developed and disseminated (e.g., reports, toolkits) and/or types of services provided (e.g. training, technical assistance), organized by audience(s) (e.g., families, students, educators, policymakers, community organizations and funders);
- How well are you doing? Percent of target audience(s) evaluating products, activities, and/or services on dimensions of quality (e.g., is the product/activity/service accurate, timely, useful, accessible, affordable, helpful, responsive, effective, and respectful etc.?);
- What difference are you making in terms of impact? Number/percent of target audience(s) influenced by products and services, advocating recommended policies, adopting effective practices, changing beliefs, etc., including evidence;
- What difference are you making in terms of leverage? Percentage of Foundation funding/non-Foundation funding of total grant or project operating costs; and
- Other accomplishments and successes, including stories and anecdotes.
Reporting Requirements
The Washington Scholarship Fund, Inc. will submit Progress and Expenditure Reports to the Foundation according to the following schedule:
Report Name | Due On or Before | Covering the Period of |
Interim Progress Report | January 31, 2007 | July 1, 2006-December 31, 2006 |
Interim Expenditure Report | January 31, 2007 | July 1, 2006-December 31, 2006 |
Final Progress Report | July 31, 2007 | January 1, 2007-June 30, 2007 |
Final Expenditure Report | July 31, 2007 | January 1, 2007-June 30, 2007 |
One (1) original and one (1) copy of each report should be submitted to the Foundation using the enclosed forms. Please send the reports to Grants Reporting and include the grant number designated on the first page of this document.
Further specific provisions of this grant are described in the attached Publication Acknowledgement Standard, and Terms of Conditions for the Annie E. Casey Foundation Grants.
Excerpt from the Washington Scholarship Fund 2006-2007 Results Report
Following is an excerpt from the Washington Scholarship Fund’s report on 2006-2007 results to the Casey Foundation. The full report (which includes a prose document) also covers results achieved in areas including: Family Support, School Outreach and Recruitment, Financial Support and Oversight, General Program Communication and Outreach, Sector Communications, and Financial and Programmatic Leverage.
DC Opportunity Scholarship Program (DC OSP)
Performance Measures
Submitted to the Annie E. Casey Foundation – 8/15/07