Effect of Appeal Content on Fundraising Success and Donor Behavior

Transcription

1 Effect of Appeal Content on Fundraising Success and Donor Behavior Juncai Jiang Virginia Tech. Sriharsha Kamatham University of Texas at Dallas Parneet Pahwa University of Texas at Dallas Nanda Kumar University of Texas at Dallas

2 Abstract Many fundraisers, for instance, K-12 public schools and charter schools particularly in high poverty areas do not have the resources to reach out to donors nor do they have the time to strategize to achieve their fundraising objectives. Teachers and students in these schools have a variety of needs which are not met because of inadequate funds. Aside from requesting the local community for financial support, many teachers in these schools use platforms such as DonorsChoose.org to appeal to the broader community for financial assistance. These appeals typically provide a brief description of the students in the school, the needs that they have and how the fundraising effort will impact their learning objectives. The goal of our study is to uncover the elements of the appeal that drive both the success of the fundraising effort and donor behavior. We want to understand how the length of the appeal, the sentiment of the appeal impacts the success of the fundraising effort and the amount donated by the donors. We also want to examine how the information that is displayed on the website impacts the amount donated by donors. How does the information displayed on the platform impact donor behavior? 2

3 1. Introduction Americans donated more than $ billion to charity in 2015, which is an increase of 4.1% over the donations in 2014 (National Philanthropic Trust 2017). Fundraisers spend in excess of $2 billion per year on a variety of strategies to successfully raise their funding targets (Kelley, 1997). Despite this the optimal behavior of fundraisers remains one of the most under-researched areas (List and Rondeau, 2003). Many fundraisers, for instance, K-12 public schools and charter schools particularly in high poverty areas do not have the resources to reach out to donors nor do they have the time to strategize to achieve their fundraising objectives. Teachers and students in these schools have a variety of needs which are not met because of inadequate funds. Aside from requesting the local community for financial support, many teachers in these schools use platforms such as DonorsChoose.org to appeal to the broader community for financial assistance. These appeals typically provide a brief description of the students in the school, the needs that they have and how the fundraising effort will impact their learning objectives. The goal of our study is to uncover the elements of the appeal that drive both the success of the fundraising effort and donor behavior. Specifically, we want to understand how the length of the appeal, the number of positive and negative words or the sentiment of the appeal impacts the success of the fundraising effort and the amount donated by the donors. In addition, we also want to examine how the information that is displayed on the website impacts the amount donated by donors. In particular, the platform can choose to display the number of donors who have contributed to the project thus far, the amount needed, and the total amount raised thus far or the total amount remaining to be raised. How does the information displayed on the platform impact donor behavior? 2. Research Setting Our research context is DonorsChoose.org (hereafter DC ), a 501(c)(3) non-profit organization that helps public school teachers who are in need of classroom supplies to raise funds from donors using an online platform. Initially founded in 2000, DC serves all K-12 public and public charter schools in the 3

4 US. Since 2005, DC has received the highest rating every year from Charity Navigator, the largest independent charity evaluator in the US. As of May 2017, DC is reported to have funded 914,000 classroom projects and aided over twenty million students. These projects attracted approximately 2.6 million donors who donated a total of $540 million. About 74 thousand public schools have participated, which accounts for 75% of public schools. To start a project, a teacher must register on the website and establish his or her qualifications as an educator at a United States public or public charter school (employed full-time by the school or district and working with students at least 75% of the time). 1 Once the verification is complete, the teacher can post a project request, which specifies the classroom materials and resources requested, along with an appeal. In the appeal, the teacher introduces the students and talks about the project itself. The DC staff reviews the project, determines the cost of the requested materials from vendors, and displays the project on the website. Besides the teacher information, the project page also reports detailed school information (e.g., the name, the address, the district, the poverty level, to name a few) and student information (e.g., the grade level, the number of students impacted by the project, the primary subject area, etc.). In Figure 1, we present an example of a DC project. The project appeal (texts in the box) is displayed at a prominent location on the project page. Once a project is published on DC, donors can land on the project page, read the appeal, and donate to projects. A donor can donate $1 or more to any project, and contributions from all donors are pooled together to fund a project. Each project remains available for funding up to four months before it expires. If a project reaches its funding goal within the time limit, DC is in charge of purchasing necessary supplies and shipping them directly to the teacher. Otherwise, the project is either relisted or removed from the platform. For the purposes of our study if the funding goal is achieved within the time limit the fundraising effort for that 1 Source: 4

5 project is deemed a success. As stated one of the objectives of our study is to understand the drivers of success of a project. [Insert Figure 1 About Here] Data and Variables The data covers the detailed project and donation information over a thirteen-year period between January 1 st 2001 and Nov 10 th We eliminated donations made to projects that were live as of Nov 10 th Our final sample includes 1,001,751 donors making 2,291,534 donations to 496,689 projects. On average, each donor made $110 in donations while each project raised $506 in donations. To evaluate the impact of the content of the appeal on the fundraising success and donor behavior, we conduct analysis at both the project and donation level. At the project level, our dependent variable of interest is fsuccess j which is a dummy variable that indicates whether project is successfully funded or not before the deadline. At the donation level, we use as the 2014 US dollar value of the donation made by donor to project. Donationij as the dependent variable. It is defined Given that our central interest in understanding the impact of the content of the appeal on fundraising success and donor behavior we conducted text and sentiment analysis on the content. The content was parsed to construct the following appeal specific measures. Word Count. We define Word Count as the number of words in the appeal excluding the stop words. Stop words are words that occur frequently in the text but are deemed as irrelevant and unimportant for the appeal content. Examples of stop words include the, am, about, etc. We use the Terrier stop word list that contains 733 stop words. After removing the stop words, each project appeal on average contains 145 words. Sentiment. To quantify the teachers affective states in the appeal, we perform sentiment analysis on the text in the project appeal. To do so, we first compiled a list of approximately 6,600 words that are 5

6 classified as positive or negative by human readers 2. Then we count the number of positive and negative words in each project appeal, and measure the sentiment as difference in the number of positive and negative words (Berger 2010). Readability. Following Mc Laughlin (1969), readability is defined as the degree to which a given class of people find certain reading matter compelling and comprehensible. In this paper, we use the Fog Index as the measure of readability due to its wide popularity, which is mainly attributed to its ease of calculation and adaptability to computational measure (Sawyer, Laran, and Xu 2008). Two factors that determine the Fog Index are the average sentence length in words and the percentage of complex words (i.e., those words with three or more syllables). Specifically, it is given by the following formula: Fog Index = 0.4 (average # of words per sentence percent of complex words) (1) The Fog Index indicates the grade level that is required to understand the text with lower (higher) values suggesting higher (lower) readability (Gunning 1952). In addition to the content specific variables described above, we have several covariates specific to the donor, the teacher and the project, which are summarized in the table below. For the sake of robustness, we also constructed other measures of readability like Flesch-Kinkaid Index, Automated Readability Index that were found to be highly correlated with the Fog Index as expected. Consequently, the qualitative insights obtained on the impact of readability on the outcome measures are insensitive to which measure was used to operationalize readability. We summarize the variables used in the analysis in Table 1. 2 The list can be found on 6

7 Variable Specific to: Donor Variable Donation Timestamp Location Message Time Remaining for expiration of the project Cumulative Donations Cumulative Donors Giving Page Description This variable captures the time when a donor made a donation. Donor city, state and zip information Text of the message left by donors on the website Time left for project to expire at the time a donation was made Total donations made till the donation made by a donor Total number of donors till the donation made by a donor Whether donation was made through a giving page Teacher Gender Gender of the teacher posting the project Member of Teach for America 1, if teacher is a member of Teach for America New York Teaching Fellow 1, if teacher is a New York teaching fellow Teacher worked at a charter school 1, if teacher worked at a charter school Teacher Teacher worked at a magnet school 1, if teacher worked at a magnet school Teacher worked at a year round school 1, if teacher worked at a year round school Teacher worked at a NLNS public school 1, if teacher worked at an NLNS public school Teacher worked at a KIPP public charter 1, if teacher worked at a KIPP public charter school school Teacher worked at a charter ready promise 1, if teacher worked at a college-ready promise school school Funding Status 1, if project was successfully funded Funds Sought Total amount in $s that a teacher seeks to raise Primary Focus Area The primary focus area of the classroom materials requested such as "Applied Learning," "Health & Sports," "History & Civics," "Literacy & Language," "Math & Science," "Music & The Arts," and "Special Needs" Resource Type The type of materials the project requested such as "Books," "Other," "Supplies," "Technology," "Trips," and "Visitors" Grade Level Four grade levels: "PreK-2," "3-5," "6-9," and "9-12" No. of Students Reached The number of students impacted by the project upon being funded Fog Index Number that shows how easy or difficult a text is to read Project Sentiment Difference between number of positive and negative words in the text of the appeal Word Count Total number of words in the text of the appeal School Metro Four different metros: urban, suburban, rural, others Poverty Level The poverty level of the school measured by the proportion of students in the school who are qualified for free lunch. "low poverty": 0-9%; "moderate poverty": 10-39%; "high poverty": 40-64%; "highest poverty": 65%+ No. of donors who left a message Total number of donors who left a message Mode of Payment Payment method used by donor to fund the project PayPal, Credit Card, Amazon, Check Project is qualified to receive a corporate partner's fund that bring down cost below $100 to go Whether the project is eligible to receive entire funds requested except the last $100. If project receives $100 from other donors, then the rest is paid by a corporate partner Total Donations The total amount of donations the project received Table 1. Variable Descriptions 7

8 3. Analysis and Results In this section we outline the models that we use to study the impact of the project and school level characteristics and the content of the appeal on both the success of the fundraising effort and donor behavior. Fundraising Success To determine the factors that affect the success of the fundraising effort we use a standard binary logit model, where the dependent variable is whether project j was successfully funded or not. The model essentially is: Logit E fsuccess j X j X j j (1) In equation (1), X j denotes the project j specific control variables and represents the unknown parameters to be estimated. The results of the random effects model are summarized in Table 2. Not surprisingly given the large number of observations in the data set the standard errors of most of the estimates is small and most of the estimates are significant at the 95% confidence interval. In the last three columns in Table 2, we present the estimates, the standard error and their p-value respectively. Let us first focus on the role of the appeal on the success of the fundraising effort. Readability of the appeal has a significant and positive impact on the success of the fundraising effort the more readable the message the more likely a project is to succeed. In recent work, Small and Verrocchi (2009) conducted experiments in a lab setting to examine the effect of facial expressions on charity advertisements. They find that people are particularly sympathetic and more likely to donate when they see sad expressions versus happy or neutral expression. 8

9 Parameter Estimate S.E. Pr > ChiSq Intercept <.0001 No. of Students Reached Donation Amount ($) Sought to be Raised <.0001 Fog Index <.0001 Sentiment (Positive Negative Words) <.0001 Total Word Count Sentiment*Total Word Count Teacher Gender (base = Female) Male Unknown (Dr. etc.,) Applied Learning Health & Sports <.0001 Primary Focus Area (base = Special Needs) History & Civics Literacy & Language <.0001 Math & Science <.0001 Music & The Arts <.0001 Books <.0001 Other Resource Type for which funds were sought (base = Visitors) Supplies Technology <.0001 Trips <.0001 Grades Grade Level (base = PreK) Grades <.0001 Grades Teacher is a member of Teach For America (base=no) Yes <.0001 Teacher is a New York teaching fellow (base=no) Yes <.0001 School Metro (base=rural) Suburban Urban <.0001 High Poverty <.0001 Poverty Level (base=moderate Poverty) Highest Poverty <.0001 Low Poverty Teacher worked at a charter school (base=no) Yes Teacher worked at a magnet school (base=no) Yes Teacher worked at a year round school (base=no) Yes Teacher worked at a NLNS public school (base=no) Yes Teacher worked at a KIPP public charter school (base=no) Yes Teacher worked at a college ready promise school (base=no) Yes No. of donors who left a message <.0001 Paypal as the main payment method (base=no) Yes <.0001 Amazon as the main payment method (base=no) Yes <.0001 Qualified to receive a corporate partner's (base=no) Yes <.0001 Check as the main payment method (base=no) Yes <.0001 Creditcard as the main payment method (base=no) Yes <.0001 Table 2: Parameters of Logistic Regression modeling Odds of Success of a Project 9

10 Similarly, Fisher and Ma (2014) find that attractive children are attributed desirable characteristics related to social competence, yet in the context of raising funds for children in need (in an experimental setting) they find negative effects of attractiveness on empathy for children in need. Consistent with these behavioral findings we find that the relative sentiment of appeal as measured by the difference in the number of positive to negative words in the appeal has a significant and negative effect on the success of the project. In other words, if the number of positive words in the appeal increases relative to the number of negative words the project is less likely to succeed. We also checked for nonmonotonic effects by including a quadratic terms for both sentiment and total word count but they were not significant. Consistent with intuition, we also find that the length of the appeal does affect the success of the project with shorter appeals more likely to increase the odds of success. To the best of our knowledge, ours is the first study to empirically document the effects of sentiment, length and content of the appeal on the success of fundraising efforts. The gender of the teacher making the appeal appears to have no effect on the success of the project. DellaVigna, List, Malmendier and Rao (2013) study whether there are gender differences in generosity from the donor s perspective. They find that in door-to-door solicitations men and women are equally generous, but women become less generous when it becomes easy to avoid the solicitor. We find no such gender differences from the fundraiser s perspective on the success of the project. As noted earlier, when the teacher posts a project request, he or she specifies the classroom materials and resources requested, along with an appeal. The materials requested could span different subject areas and resource types. So for instance, the teacher could request books or technological resources for a math and science class or music and arts class. We find that both the subject areas and resource type have an effect on project success. The subject areas of the project are classified into one of seven areas Applied Learning, Health and Sports, History and Civics, Literacy and Language, Math and Science, Music and Arts and Special Needs. We have six dummy variables to denote the first six focus area with Special Needs being the base when all other dummies are zero. Relative to special needs, we 10

11 find that Math and Science followed by Music and Arts projects are most likely to succeed, while projects pertaining to Health and Sports and Literacy and Language are least likely to succeed. Resource types similarly are classified into six different categories which are listed in the data description with visitors as the base category. Relative to visitors, projects related to Trips and Books are most likely to succeed, while technology related projects are least likely to succeed. While we understand teacher s needs are specific and that they cannot always control what resources their class needs but these findings offer insights on the kinds of projects that donors prefer to support. We find that teachers who are members of Teach for America or a New York teaching fellow are more successful in getting their projects funded relative to teachers who are members of neither organization. There is no statistical difference in project success across schools located in rural and suburban areas but relative to schools located in rural areas teachers from schools in urban areas have projects that are more successful. Again, this speaks more to donor preferences rather than anything a teacher can control to increase the odds of successfully funding their projects. Interestingly, the mode of accepting donations has a critical and sizeable impact on the success of the project. Accepting donations through checks, credit card have the highest impact, while accepting donations through Amazon and Paypal have a smaller but still positive effect. Having a corporate sponsor to lower the amount to be raised also has a positive effect on the project success. This is somewhat interesting because in a recent study Karlan, List and Shafir (2011) conduct a natural field experiment using two direct mail solicitations to test the effectiveness of $1:$1 and $1:$3 matching grants on charitable giving and only find weak evidence that either match works. In fact, in their data match only increased giving after the match deadline expired. Counter to this we find strong evidence that a corporate sponsor has a significant and positive effect on the success of the project. To account for unobserved heterogeneity we also performed the same analysis with school city level random intercepts. For the most part our results are qualitatively similar. We present these results in Table 4 in the Appendix. 11

12 Donor Behavior We now turn our attention to understand the determinants of the factors that influence donor behavior, specifically the donation amount. In addition to the effect of the content and sentiment of the appeal on donor behavior we also wanted to understand how the information displayed on the appeal page affects the donation amount. Figure 1 depicts the appeals page for a typical project. On the top of the appeal the donor is presented with information on the number of donors who have donated to the project, the funds still needed and the goal. This information can be presented in alternative ways and we want to examine what information to present and in what manner to present to increase the donation amount. As noted earlier the model is a standard regression model with the amount donated by the donor as the dependent variable. As in the earlier model we include the content of the appeal, its readability, sentiment and word count. In addition, we include the total amount to be raised for the project, the number of students that will benefit from the project, the number of donors, the cumulative amount raised at the time the donor makes a donation and the percentage donation remaining to be raised at the time the donor makes the donation. We also include many of the project and teacher level covariates as in the earlier model. The estimates from the regression model are reported in Table 3 below. First note that the effect of different elements of the appeal readability, sentiment and length of the appeal directionally have the same effect on donation amount as they do on the success of the project. Let us now turn to the effect of the information presented on the appeals page on donor behavior. Interestingly, displaying the number of donors that have already donated to the project seems to have a negative effect on the amount donated. This was counter to our intuition as we thought the number of donors may serve as a signal that would foster more donation. It is possible that as the number of donors increase the amount remaining to be raised is lower which in turn lowers the donation amount. However, the direct effect of the donation amount sought to be raised is positive and significant. 12

13 Parameter Est. (OLS Model) Class Estimate SE Pr > t Intercept <.0001 No. of donors who donated so far (up to a donor's time of donation) <.0001 % Donations Remaining to Total Sought <.0001 Donation Amount ($) Sought to be Raised <.0001 Donation Amount ($) Raised So Far <.0001 No. of Students Reached <.0001 Time Remaining (Days to Expiration of Project) <.0001 Fog Index <.0001 Sentiment (Positive Negative Words) <.0001 Total Word Count <.0001 Sentiment*Total Word Count <.0001 Teacher Gender (base = Female) Male <.0001 Unknown (Dr. etc.,) Applied Learning <.0001 Health & Sports <.0001 History & Civics Primary Focus Area (base = Special Needs) Literacy & Language Math & Science <.0001 Music & The Arts <.0001 Books Other Resource Type for which funds were sought (base = Visitors) Supplies Technology Trips <.0001 Grades Grade Level (base = PreK) Grades Grades <.0001 Teacher is a member of Teach For America (base=no) Yes Teacher is a New York teaching fellow (base=no) Yes <.0001 School Metro (base=rural) Suburban <.0001 Urban High Poverty Poverty Level (base=moderate Poverty) Highest Poverty Low Poverty <.0001 Teacher worked at a charter school (base=no) Yes <.0001 Teacher worked at a magnet school (base=no) Yes Teacher worked at a year round school (base=no) Yes Teacher worked at a NLNS public school (base=no) Yes Teacher worked at a KIPP public charter school (base=no) Yes Teacher worked at a college ready promise school (base=no) Yes Donation was made through giving page (base=no) Yes <.0001 Table 3: OLS Model. Dependent Variable: Donation ($) 13

14 Again, to check for non-monotonic effects of the displaying the number of donors on the appeals page we included a quadratic term to check whether the relationship between the donation amount and the number of donors is an inverted U shaped curve. This would allow us to confirm perhaps that displaying the number of donors may initially have a positive effect on donation amount and then a negative effect once the number of donors displayed exceeds a certain threshold but we found no such effect. As noted earlier, both the total donation amount sought and the cumulative donation amount raised so far have a positive effect on the amount donated by donors. We found very interestingly, that displaying the Percentage Donation Remaining to total sought has a positive effect on the amount donated by donors. We feel this has important implication for DonorsChoose.org. Note rather than displaying percentage donation remaining one could very easily display the percentage of the total sought that has already been raised. Since, percentage of the total sought that has already been raised is simply (1 - percentage of the total sought that is remaining to be raised), if we had included percentage of the total sought that has already been raised as a covariate instead that would have the same estimate of but with a negative sign. Given that displaying the number of donors also has a negative effect on donation amount an implication for DonorsChoose.org would be to not display the number of donors and display the percentage of total sought that is remaining to be raised rather than displaying the percentage of total that has already been raised. The effect of the project level covariates on donation amount such as focus area and resource type are directionally different from their effect on project success. All focus areas, except History and Civics have a significant and positive effect on donation amount relative to the base focus area of Special needs. Other covariates, for the most part directionally have the same effect on donation amount as they do on the success of the project. 14

15 4. Concluding Remarks As noted earlier, in excess of $2 billion is being spent annually on fundraising efforts and yet the empirical research on the effectiveness of these efforts is scant. Our research attempts to fill this gap and contributes to the literature in multiple ways. First, to the best of our knowledge this work is the first to quantify the effect of the content of the appeal on fundraising success and donor behavior. The appeal is perhaps something unique to the platform which allows us to conduct this investigation and offer evidence from secondary data that is consistent to prior behavioral research by Small and Verrocchi (2009) and Fisher and Ma (2014). Second, we find that the success of the project depends critically on the mode of accepting donations. Consequently, from a policy standpoint DC should accept donations through checks and credit card as they have the highest impact on project success. It is not obvious whether the teacher or the platform make these decisions but the insights can be used to optimize and make the fundraising decisions more efficient. Finally, another policy implication for the platform from our analysis is on the nature of information that is displayed and its impact on donor behavior. We find counter to intuition that displaying the number of donors that have already contributed to the project has a negative impact on the amount donated by the donor. Consequently, it will be prudent for the platform to suppress this information. We also find evidence of framing effects our analysis suggests that the donors would donate more when presented with information on the percentage donation remaining to total sought relative to being presented with information on the percentage donation to total raised so far even though the content of the information is the same. In conclusion, we believe our research offers valuable insights both to the fundraisers and the platform. For the fundraisers, the teachers we offer insights on how the content of the appeal that they put up on the platform affect both the success of the project and donor behavior which can be used to optimize their appeal. The insights on the focus areas and resource types and their effect on project success provide teachers with information on aggregate measures of success of various genre of projects. For, the platform there are several policy implications that we outlined above. To the best of our 15

16 knowledge this is the first study to offer insights based on this specific platform that facilitates fundraising by teachers from public and charter schools, which examines the effect of the content of the appeal on project success and donor behavior. 16

17 Appendix Figure 1. An Example of DC Project 17

18 Parameter Estimate S.E. Pr > ChiSq Intercept < No. of Students Reached < Donation Amount ($) Sought to be Raised < Fog Index Sentiment (Positive Negative Words) < Total Word Count Sentiment*Total Word Count < Teacher Gender (base = Female) Male < Unknown (Dr. etc.,) Applied Learning Health & Sports Primary Focus Area (base = Special Needs) History & Civics Literacy & Language Math & Science <.0001 Music & The Arts <.0001 Books Other Resource Type for which funds were sought (base = Visitors) Supplies Technology Trips Grades <.0001 Grade Level (base = PreK) Grades <.0001 Grades Teacher is a member of Teach For America (base=no) Yes <.0001 Teacher is a New York teaching fellow (base=no) Yes School Metro (base=rural) Suburban Urban <.0001 High Poverty Poverty Level (base=moderate Poverty) Highest Poverty <.0001 Low Poverty Teacher worked at a charter school (base=no) Yes Teacher worked at a magnet school (base=no) Yes Teacher worked at a year round school (base=no) Yes Teacher worked at a NLNS public school (base=no) Yes Teacher worked at a KIPP public charter school (base=no) Yes Teacher worked at a college ready promise school (base=no) Yes No. of donors who left a message <.0001 Paypal as the main payment method (base=no) Yes <.0001 Amazon as the main payment method (base=no) Yes <.0001 Qualified to receive a corporate partner's fund (base=no) Yes <.0001 Check as the main payment method (base=no) Yes <.0001 Creditcard as the main payment method (base=no) Yes <.0001 Table 4: Parameters of Logistic Regression modeling Odds of Success of a Project with School City level Random Intercepts 18

19 References Berger, J. and Milkman, K., Social transmission, emotion, and the virality of online content. Wharton research paper, 106. DellaVigna, Stefano, John A. List, Ulrike Malmendier, and Gautam Rao. "The importance of being marginal: Gender differences in generosity." The American Economic Review 103, no. 3 (2013): Fisher, Robert J., and Yu Ma. "The price of being beautiful: Negative effects of attractiveness on empathy for children in need." Journal of Consumer Research 41, no. 2 (2014): Gunning, Robert. "The technique of clear writing." (1952). Karlan, Dean, John A. List, and Eldar Shafir. "Small matches and charitable giving: Evidence from a natural field experiment." Journal of Public Economics 95, no. 5 (2011): Kelly, Kathleen S. "From motivation to mutual understanding: Shifting the domain of donor research." Critical issues in fundraising (1997). List, John A., and Daniel Rondeau. "The impact of challenge gifts on charitable giving: an experimental investigation." Economics Letters 79, no. 2 (2003): Mc Laughlin, G. Harry. "SMOG grading-a new readability formula." Journal of reading 12.8 (1969): Sawyer, Alan G., Juliano Laran, and Jun Xu. "The readability of marketing journals: Are awardwinning articles better written?." Journal of Marketing 72, no. 1 (2008): Small, Deborah A., and Nicole M. Verrochi. "The face of need: Facial emotion expression on charity advertisements." Journal of Marketing Research 46, no. 6 (2009):