Arts Education Community-Engaged Research

Exploring scalable systems for education nonprofits to manage information about their students

Overview

As primary researcher for my Master’s capstone team, I led our three-month research partnership with local youth arts education nonprofit Urban ArtWorks.

This qualitative research with Urban ArtWorks staff members helped my capstone team to understand their systems & goals for managing information about their students amid rapid organizational growth. We interviewed every member of their team at least once and also conducted field research to help us situate in their environment. Through an asset-based analysis, I derived insights pointing us to the need for systems specifically focused on generating and utilizing meaningful feedback data from students & teachers.

my role: Researcher

  • Decided on appropriate methods
  • Developed research protocol
  • Conducted asset-based analysis
  • Leveraged insights to propose our design criteria & direction

team

  • Kimmie Aralar, designer
  • Claire Yi, PM

research methods

  • 3 unstructured interviews to gain context
  • 3 pilot interviews to validate design probe
  • 6 semi-structured interviews with a design probe
  • 2 field observation sessions

deliverables

  • Interview Protocol
  • Research Report
  • Design Recommendations

timeline: spring 2022 (10 weeks)

timeline showing March-May is community-engaged research, and June-August is co-design

This case study covers our research process; read more about our co-design phase with Urban ArtWorks for details on our final product outcomes.


Background

This research gave insight into how staff members of a rapidly growing arts education nonprofit manage data about their students, highlighting an opportunity to bolster program improvements & grant funding by designing a centralized system for recurring student reflections.

community-engaged research approach

By conducting our research in partnership with Urban ArtWorks, we built a sense of shared accountability and set ourselves up to explore a topic that we mutually agreed could bring tangible benefit to their organization: student information management systems.

Urban ArtWorks is a Seattle-based youth arts education nonprofit that focuses on public murals. We collaborated with them throughout our 6 month capstone project to gain a deeply situated understanding of their organization's environment and systems.

Urban ArtWorks staff pose in front of a mural, courtesy of Urban ArtWorks

importance

Systems which provide access to well-organized information about students are crucial for ensuring smooth daily operation, informing programming changes, and bolstering grant funding applications.

Student information is used in everything from understanding programs’ impact to making sure staff know who to call in an emergency. Knowledge about which communities they are serving helps Urban ArtWorks leadership decide where to expand and demonstrates their commitment to supporting under-resourced areas in grant funding.

challenge

Urban ArtWorks’ existing student information management systems were facing growing pains as they scaled up their programming, geographic reach, and funding needs.

Although they’ve existed at a small scale for over 20 years, the organization is currently undergoing rapid growth in terms of number of staffers, programs, and even geographic area. This growth is stressing their technical systems, which presented an exciting opportunity for collaborative research & design on student information management.

Urban ArtWorks students collaboratively painting a mural, courtesy of Urban ArtWorks

methods

In order to understand staff member’s goals, capabilities, and pain points, I leveraged a mixed qualitative methodology consisting of interviews and field observations.

My teammates and I conducted 3 unstructured half-hour interviews with the Executive Director and Program Manager to understand their overall context and decide on a project focus, six 1-hour semi-structured interviews with all staffers, and two field research sessions.

research outcomes

My research insights highlighted an opportunity to support education nonprofits in collecting, organizing, and analyzing the holistic feedback they need for student-centered program improvements and grant reporting.

The research I led revealed that although Urban ArtWorks’ staff had already begun to leverage their nuanced understanding of teachers’ workflows & students’ needs, they still faced difficulties getting the meaningful evaluation data needed to tell the story of their organization’s impact.

This informed my team’s final design of a learning reflection platform called Mirror: a desktop & mobile app that streamlines the process of collecting recurring, relevant feedback from students and teachers. Learn more about my design process here.

Mirror is a learning reflection platform which empowers education nonprofits to center student perspectives through holistic feedback


phase 1: selecting a topic & creating a research plan

With my team’s focus on supporting structural changes in youth arts nonprofits due to COVID-19, I advocated for a community-engaged research approach in order to center practitioners’ expertise and ensure direct benefit to their organization.

Research Timeline

Research timeline highlighting Phase 1 out of 4, which is Initial topic selection & study planning

initial topic: youth arts nonprofits

Youth arts nonprofits, crucial community organizations that support students’ development & happiness, are undergoing structural change as a response to significant challenges due to COVID-19.

Question to Answer

Why are youth arts nonprofits important community organizations, and what common challenges do they face?

Methods I Employed
  • Secondary research on youth arts nonprofits' benefits to students & current difficulties in the field
What I Learned

As a safe environment for creative self-expression & reflection, youth arts nonprofits can bolster students’ self-esteem, promote emotional & mental health, and facilitate strong peer bonds (Source: 1. Ennis 2018).

Unfortunately, COVID-19 has created significant challenges for these organizations, engendering widespread change within the field. A study released in 2022 by ArtsFund, a Washington nonprofit advocating for the arts, found that arts organizations lost nearly $100 million in revenue in the immediate months following the COVID-19 outbreak. It will likely take years, and significant public investment, to ensure the survival of these organizations, the study concludes (Source: 2. ArtsFund WA 2022 COVID Cultural Impact Study).

Research timeline highlighting Phase 2 out of 4, which is recruiting & research focus

“We are not at the tail-end of a pandemic – we are at the beginning of a structural transformation.”

Source: 2. ArtsFund WA 2022 COVID Cultural Impact Study

framework: community-engaged research

By partnering with a local arts education nonprofit to do ‘research with’ instead of ‘research on’, we ensured practitioners’ perspectives and lived realities were front and center in our work.

Question to Answer

Because many youth arts nonprofits have been hit hard by COVID, how could we ensure that our work actually contributed back to the community and that we were fully honoring their experience navigating the past few years?

Methods I Employed
  • Advocated for Community-Engaged Research framework
What I Learned

Community-engaged research is most common in public health efforts, but can be similarly leveraged in HCI to “conduct research that can translate more easily to real world settings” and “design more culturally- and language-appropriate interventions” (Source: 3. Handley 2010). In these efforts, a researcher aims to act as a “social designer, facilitator of processes, and agent of change” rather than the sole arbiter of knowledge and expertise (Source: 4. ACM Interactions 2022).

study planning: managing risks

Through a careful workback schedule, we acknowledged and tried to account for the additional uncertainty introduced by taking a dependency on a partner organization.

Question to Answer

How could we set ourselved up to successfully meet our project goals on time, despite the potential for a complicated recruiting and research focus selection process?

Methods I Employed
  • Risk assessment for delaying scoping & interview protocol creation
  • Workback schedule with ample buffer time
Outcomes

Once we had created a detailed schedule, my teammates and I realized that we would be able to minimize the risk incurred by searching for an organization willing to commit to a 6-month partnership by 1) starting our recruiting efforts quickly, 2) preparing our field note-taking template in parallel, and 3) working efficiently to create an interview protocol once we had found a partner organization and collaboratively selected a specific research topic.


phase 2: recruiting & choosing a research focus

After agreeing on a community-engaged approach, my team concentrated our efforts upfront on recruiting a partner organization and collaborating with them to identify an appropriate research focus area.

Research Timeline

Research timeline highlighting Phase 2 out of 4, which is recruiting & research focus

partner organization recruiting: urban artworks

We successfully recruited our partner organization, Urban ArtWorks, by iteratively refining our project pitch and conveying our genuine desire for our research to center their perspectives & organizational needs.

Question to Answer

What types of youth arts organizations would be best suited for forming a co-design partnership with, and how could we recruit them on a relatively tight timeline?

Methods I Employed
  • Recruiting criteria that ensured alignment while casting a broad net
  • Recruiting spreadsheet with background & contact info for over 30 local youth arts nonprofits
  • Organized tracking of our communication status with each organization and when to follow-up
  • Standardized research elevator pitch and answers to common questions we encountered
Outcomes

Although many of the organizations we tried to recruit expressed that their staffing & resource constraints left them unable to participate in more than a one-off capacity, we were thrilled when Urban ArtWorks’ Executive Director was willing to connect with us. In our introduction, I explained why a research partnership can be worth the additional commitment, while also being clear about our roles and boundaries as researcher and designers. After learning more about their organization and sharing about our project and program, we agreed that a collaboration could be mutually beneficial.

research focus: student information management

Guided by two introductory interviews with Urban ArtWorks leadership, we mutually agreed to focus our research on their systems for managing student information.

Question to Answer

In what area might our partnership be most fruitful both in generation knowledge & design plans for their organization, and as a learning opportunity for us as students?

Methods I Employed
  • Unstructured introductory interviews with Urban ArtWorks leadership
  • Negotiating proposals, in which “the community tries to understand how the researcher's design knowledge could contribute to strengthening its activities” (Source: 4. ACM Interactions 2022)
Outcomes

During our initial conversations, we gave input around where we believed we could contribute most, but ultimately let Urban ArtWorks guide our project scoping. We agreed that an appropriate topic would be important for Urban ArtWorks, but not something they are currently able to fully address on their own (ex. due to resource constraints). Furthermore, it needed to be an open-ended enough space for our research & design expertise to be useful.

After careful consideration, Urban ArtWorks’ Executive Director proposed that a fitting topic to research & design around would be systems for managing information about students. We agreed!

research objective & questions

This research aimed to understand: ‘How do Urban ArtWorks’ participant information management systems impact their ability to achieve their organizational mission at a broader scale?’

Questions About Current Systems
  1. What systems does UA currently use for participant information management?
  2. What do UA staff members think is working well about these systems?
  3. What challenges have UA staff members faced with these systems as the organization has expanded, and how have they responded?
Questions About Future Systems
  1. How well do UA staff think their participant info management systems will hold up as the organization continues to scale?
  2. What are staff members’ goals and hopes around participant info management as UA grows?
  3. What changes do they anticipate needing to make to these systems in order to handle scaling?

phase 3: conducting primary research with our partner organization

To understand Urban ArtWorks’ student information management systems, we paired interviews to directly hear staffers’ perceptions with observations to build an in-situ understanding of their environment and processes.

Research Timeline

Research timeline highlighting Phase 3 out of 4, which is primary research with partner organization

method 1: interviews

I chose to conduct semi-structured interviews in order to hear staff members’ goals and perceptions in their own words, and included a timeline design probe to help us orient to staffers’ different levels of zoom.

Design Probe

We chose to include a design probe in the form of a timeline activity during our interviews. From our initial scoping conversations, we realized that we needed a mechanism to give us a deeper-level understanding of their systems to have enough context for our findings & recommendations to be useful.

With this design probe activity, participants were directed to write down each action they took and system they interacted with over the course of a given time period, in relation to student information management. This time period was agreed upon during the interview and selected to represent a cycle in which all key actions were performed. For some staffers, that meant a single youth program, while the Executive Director operates on a yearly scale.

Interview setup including completed timeline design probe activity

Interview Protocol

I led the creation of a 60 minute interview protocol that aimed to understand Urban ArtWorks staff members’ processes and perceptions around student information management systems. Once we had covered consent, we reiterated our research’s focus area, going into as much detail as each individual participant needed.

I then designated 20 minutes for our timeline activity, with the goal of understanding the specifics of how they interact with student information. With this as context, we moved into 15 minutes of discussing their perceptions, with the goal of understanding their feelings around their interactions with student information management systems.

Shortened version of interview protocol showing questions for a timeline activity

method 2: Observations

Observations served as a complement to our interviews, and enabled us to build an in-situ understanding of Urban ArtWorks’ environment and processes.

Field Note Template

I created a template to use when taking field notes based on the AEIOU format (Activities, Environment, Interactions, Objects, Users). This template also included space for questions & follow-ups, reflections, and sketches.

Shortened version of field note template


phase 4: analyzing data & proposing a design direction

Research Timeline

Research timeline highlighting Phase 4 out of 4, which is analysis & design direction proposal

our findings

While crafting insights based on our interviews & observations, I took care to ensure they would highlight opportunities in a way that was situated with Urban ArtWorks staff members in order to catalyze a successful co-design phase.

findings premise

By centering our findings around a plane metaphor, we reflected the way Urban ArtWorks staffers had talked to us about their organization’s systems.

During our interviews, I noted that multiple staff members had talked about Urban ArtWorks as a moving object that they’re trying to keep up with. References to visceral imagery such as stepping aboard a moving train evoked a strong sense of the pace at which the organization is growing, and the balancing act that staffers must manage of maintaining day-to-day operations while envisioning their longer-term direction.

“We're building the airplane as we fly it.”

- Paul, Urban ArtWorks Projects Lead

insight 1: Evaluation

For youth to help steer the plane, they first need the structure to climb aboard.

What’s Already Working:

Staffers have a deep focus on building relationships with youth participants, and are already using this nuanced understanding to start evolving their feedback collection practices.

Challenge Staffers Face:

Staff members spoke of difficulties getting complete and meaningful evaluation data.

Opportunity:

How might we ensure feedback systems are accessible and meaningful to both youth participants and UA staff?

insight 2: streamlining

Streamlined systems lessen the turbulence of bringing on new people.

What’s Already Working:

Staffers have started to make information management systems explicit through a programs checklist & onboarding document.

Challenge Staffers Face:

Systems that have developed organically over time are difficult to explain and teach to new staff members

Opportunity:

How might we streamline systems as well as the process of creating guidance that evolves alongside them?

insight 3: accessing information

An organized toolkit enables finding information without rummaging around in luggage.

What’s Already Working:

In order to improve organizational practices, staff members have recently adopted the project management tooling Asana, and are re-organizing existing data in Google Drive.

Challenge Staffers Face:

Because gradually accrued information has become dispersed over time, it's hard for staffers to find information on demand. This is further complicated by an array of tools that don’t automatically tie together.

Opportunity:

How might we develop a consolidated tooling plan to ensure staff members have access to the information they need, when they need it?


Sources

  1. Gretchen Marie Ennis and Jane Tonkin (2018). ‘It’s like exercise for your soul’: how participation in youth arts activities contributes to young people’s wellbeing. Journal of Youth Studies. Apr 2018, Vol. 21 Issue 3, p340-359.
  2. ArtsFund Washington (2022). COVID Cultural Impact Study. https://www.artsfund.org/wa-covid-cultural-impact-study
  3. Handley M et al (2010). Community-Engaged Research: A Quick-Start Guide for Researchers. Clinical Translational Science Institute Community Engagement Program, University of California San Francisco. http://ctsi.ucsf.edu/files/CE/guide_for_researchers.pdf
  4. ACM Interactions (September 2022). Assets and community engagement: A roundtable with HCI researchers and designers. https://interactions.acm.org/archive/view/september-october-2022/assets-and-community-engagement