Six years ago After School Matters, the vaunted teen-activity engine spawned by Maggie Daley’s Gallery 37 art-in-a-tent idea, put itself under the microscope of a team of Northwestern University researchers. ASM—which had seen rapid growth, expanding from one program to five, and would soon be operating in 60 high schools—wanted a rigorous scientific evaluation from the professors.
Psychologist Barton Hirsch and statistician Larry Hedges took the project on, devising a million-dollar study, soliciting foundation grants, recruiting colleagues, and spending three years collecting data. Their 115-page report, After-School Programs for High School Students: An Evaluation of After School Matters, was published in June and recently posted on the Wallace Foundation website, where it landed with a great big ambiguous thud.
It turns out that a comparison of teens participating in ASM to a control group of teens yielded results so negligible the authors offered a take-your-pick pair of interpretations: a “positive perspective,” focusing on the few statistically significant outcomes, and a “skeptical view,” noting that effect sizes were “generally small” and—oops—that “testing a more representative sample of ASM instructors may well eliminate the few positive impacts that were found.”
That’s because participants were chosen randomly, but their instructors were not. To give the program its “best shot” at demonstrating impact, only the best were included.
So it’s possible that ASM—which had a budget of $28 million in 2010, has been the darling of the politically savvy philanthropy crowd, and is touted as a pioneering model for programs in other cities—isn’t any better than, say, a slot on the school volleyball team.
Time for a little background. Back in 1991, mayoral spouse Maggie Daley, then the mother of a couple of teenagers, and her pal Lois Weisberg, then commissioner of the Department of Cultural Affairs, noticed the city didn’t have much going in the way of cultural activities for teens. After-school programs were generally geared to younger kids who needed child care until parents got home from work. As luck would have it, the mayor was looking at the time for something (anything!) to mask a glaring eyesore he’d produced in the heart of downtown: an entire empty block on State Street, cleared for a development that was looking like it would never materialize. By the time summer rolled around, Gallery 37, named for the block’s designation on city zoning maps, was operating a summer art camp for teens under a cluster of big white tents across the street from Marshall Field’s iconic flagship store. Classified as a job-training program (though jobs for artists are notoriously scarce), it qualified for federal funding, which made everybody happy: the teens got a stipend for learning the intricacies of mosaics or watercolors, the perpetually starving artists hired to instruct them got paid, and the mayor turned his most visible white elephant into a commendable beehive of activity.
By 1996 Gallery 37 was operating in public schools, and in 2000 its apprenticeship program, offering hands-on training under professional instructors to create “job readiness,” was expanded beyond the arts. Now, under the nonprofit umbrella organization After School Matters, it consists of Gallery37, Tech37, Sports37, Words37, and Science37. It’s run by a full-time staff of 75 people, operating out of the city-owned Gallery 37 building on Randolph Street and in the Cultural Center, and offers programming in parks and libraries as well as schools. This year as many as 14,000 kids from Chicago’s most disadvantaged high schools will participate in ASM programs.
The NU team gathered a sample of 535 kids who’d expressed interest in one of 13 apprenticeships and randomly assigned them either to participate or be part of the control group. Using surveys, interviews, school records, and observation, they tested outcomes in four main areas: positive youth development, marketable job skills, academic performance, and problem behavior. They were met with a couple of surprises early on: though they’d been told to expect little attrition, they encountered a nearly 50 percent dropout rate among participants, and while they expected to be operating in an activity desert, almost all of the control group found other after-school options. “This changed our understanding of the experimental contrast in an important way,” the researchers wrote: instead of comparing ASM to no “treatment,” they would be comparing it to an alternative.
The study turned up three significant differences between the two groups. Kids who’d been in ASM were slightly less likely to sell drugs or participate in gang activities and also somewhat better at self-regulation. Marketable job skills and academic performance showed no significant differences. According to the report, the weakest outcomes, on average, “were in ASM’s two highest priority areas—positive youth development and marketable job skills,” while the strongest outcomes were in an area of great concern to society: problem behavior.
So is ASM working? “There’s some evidence that it is,” Hirsch says. “It’s not easy to get results that are statistically significant,” so any difference is “noteworthy.” And the worst didn’t happen: “There were no instances in which the control group did better.” On the other hand, he says, results were modest. “I think it’s a good start, but they need to make improvements.” Among those: Instructors should be better trained and supervised so that students will be “more focused.” And curriculum needs to “deepen,” especially after the first semester.
Hirsch says he included alternate perspectives because “I was trying to be very fair and balanced. If it was a boxing match, it would be a split decision.” But, he said, the study “was an important step in understanding what they’re doing well and where they need to improve, and that’s partly why you do evaluation studies. After-school programs for high school kids haven’t been around all that long, so it’s really useful to get that information.”
ASM’s top strategist, David Sinski, agrees it was “a learning experience” and says some changes have already been implemented. He notes the study’s criteria for dropout and completion rates differed from ASM’s criteria, and maintains that if they were to do a study today, the results would be different. He offers “impressive data that we just pulled two weeks ago from another project,” Chicago Allies for Youth Success. “They tracked our teens against the CPS general population,” Sinski says, “and found that our teens are in school on average five days more. If they’re in two of our programs, almost eight days more, and if they’re in three of our programs, nearly 12 days more.”
But a major rationale for Hirsch’s million-dollar effort was that it eliminated what he calls “the selection effect,” described in his report as “a situation in which youth more likely to improve over time are disproportionately located in the treatment group.” The selection effect, Hirsch writes, makes it impossible to know whether the program being tested had anything to do with the outcomes. The Chicago Allies data compares 10,850 ASM participants to 89,363 other CPS students. Is it any surprise that the kids who chose to participate in apprenticeships are also the kids who have better school attendance records?
The study isn’t the only recent news out of ASM. Sinski, formerly executive director, was bumped down to Chief Officer of Strategy and Innovation at the end of July, when the organization, whose board is still headed by Maggie Daley, got a surprising new CEO: Ray Orozco, her husband’s final chief of staff, a veteran member of the Chicago Fire Department, and, like Richie Daley, a second-generation Chicago political insider. Crain’s Chicago Business reported that Orozco’s salary is $185,000. (After School Matters, asked to confirm this figure, said “it does not release staff salary information.”) Another refugee from the mayor’s final days in office, Katherine LaMantia—who took over as cultural affairs commissioner after Lois Weisberg’s abrupt departure earlier this year—has also joined ASM in the newly created post of chief financial officer.
Meanwhile, in the face of a $3 million budget drop thanks to reduced government funding, Sinski says, ASM has pulled back from 61 to 45 schools, reduced the number of apprenticeship slots by 20 percent, and cut its stipends to student participants. A basic apprenticeship, which paid $450 per semester during the study years, now pays $100. v