Arts Education Community-Engaged Research

Exploring scalable systems for education nonprofits to manage information about their students

Overview

As primary researcher for my Master’s capstone team, I led our three-month research partnership with local youth arts education nonprofit Urban ArtWorks.

This qualitative research with Urban ArtWorks staff members helped my capstone team to understand their systems & goals for managing information about their students amid rapid organizational growth. We interviewed every member of their team at least once and also conducted field research to help us situate in their environment. Through an asset-based analysis, I derived insights pointing us to the need for systems specifically focused on generating and utilizing meaningful feedback data from students & teachers.

my role: Researcher

  • Decided on appropriate methods
  • Developed research protocol
  • Conducted asset-based analysis
  • Leveraged insights to propose our design criteria & direction

team

  • Kimmie Aralar, designer
  • Claire Yi, PM

research methods

  • 3 unstructured interviews to gain context
  • 3 pilot interviews to validate design probe
  • 6 semi-structured interviews with a design probe
  • 2 field observation sessions

deliverables

  • Interview Protocol
  • Research Report
  • Design Recommendations

timeline: spring 2022 (10 weeks)

timeline showing March-May is community-engaged research, and June-August is co-design

This case study covers our research process; read more about our co-design phase with Urban ArtWorks for details on our final product outcomes.


Background

This research gave insight into how staff members of a rapidly growing arts education nonprofit manage data about their students, highlighting an opportunity to bolster program improvements & grant funding by designing a centralized system for recurring student reflections.

research topic

My capstone team partnered with local youth arts nonprofit Urban ArtWorks to research scalable systems for managing information about students.

Systems which provide access to well-organized information about students are crucial for ensuring smooth daily operation, informing programming changes, and bolstering grant funding applications.

Student information is used in everything from understanding programs’ impact to making sure staff know who to call in an emergency. Knowledge about which communities they are serving helps Urban ArtWorks leadership decide where to expand and demonstrates their commitment to supporting under-resourced areas in grant funding.

partner organization

Urban ArtWorks is a Seattle-based youth arts education nonprofit that focuses on public murals.

They aim to 'break barriers, one wall at a time' through opportunities like their Mural Apprentice Program, an "arts-based employment training program for teens who face barriers to the arts, education, and employment".

We collaborated with Urban ArtWorks throughout our 6 month capstone project to gain a deeply situated understanding of their organization's environment and systems.

Urban ArtWorks staff pose in front of a mural, courtesy of Urban ArtWorks

challenge

Urban ArtWorks’ existing student information management systems were facing growing pains as they scaled up their programming, geographic reach, and funding needs.

Although they’ve existed at a small scale for over 20 years, the organization is currently undergoing rapid growth in terms of number of staffers, programs, and even geographic area. This growth is stressing their technical systems, which presented an exciting opportunity for collaborative research & design on student information management.

Urban ArtWorks students collaboratively painting a mural, courtesy of Urban ArtWorks

methods

In order to understand staff member’s goals, capabilities, and pain points, I leveraged a mixed qualitative methodology consisting of interviews and field observations.

My teammates and I conducted 3 unstructured half-hour interviews with the Executive Director and Program Manager to understand their overall context and decide on a project focus, six 1-hour semi-structured interviews with all staffers, and two field research sessions.

Screenshot from an interview with Eboni, Urban ArtWorks' Program Manager

research outcomes

My research insights highlighted an opportunity to support education nonprofits in collecting, organizing, and analyzing the holistic feedback they need for student-centered program improvements and grant reporting.

The research I led revealed that although Urban ArtWorks’ staff had already begun to leverage their nuanced understanding of teachers’ workflows & students’ needs, they still faced difficulties getting the meaningful evaluation data needed to tell the story of their organization’s impact.

This informed my team’s final design of a learning reflection platform called Mirror: a desktop & mobile app that streamlines the process of collecting recurring, relevant feedback from students and teachers. Learn more about my design process here.

Mirror is a learning reflection platform which empowers education nonprofits to center student perspectives through holistic feedback


phase 1: selecting a topic & creating a research plan

With my team’s focus on supporting structural changes in youth arts nonprofits due to COVID-19, I advocated for a community-engaged research approach in order to center practitioners’ expertise and ensure direct benefit to their organization.

Research Timeline

Research timeline highlighting Phase 1 out of 4, which is Initial topic selection & study planning

initial topic: youth arts nonprofits

Through secondary research, we learned that youth arts education nonprofits are undergoing structural change as a response to significant challenges due to COVID-19.

Youth arts education nonprofits are crucial community organizations that support students’ development & happiness. As a safe environment for creative self-expression & reflection, youth arts nonprofits can bolster students’ self-esteem, promote emotional & mental health, and facilitate strong peer bonds (Source: 1. Ennis 2018).

Unfortunately, COVID-19 has created significant challenges for these organizations, engendering widespread change within the field. A study released in 2022 by Washington nonprofit ArtsFund found that arts organizations in the state lost nearly $100 million in revenue in the immediate months following the COVID-19 outbreak. It will likely take years, and significant public investment, to ensure the survival of these organizations, the study concludes (Source: 2. ArtsFund WA 2022 COVID Cultural Impact Study).

Young students show off their drawings, courtesy of Urban ArtWorks

Young students show off their drawings, courtesy of Urban ArtWorks

“We are not at the tail-end of a pandemic – we are at the beginning of a structural transformation.”

Source: 2. ArtsFund WA 2022 COVID Cultural Impact Study

selecting a framework: community-engaged research

I advocated for a community-engaged research approach because partnering with a local arts education nonprofit to do ‘research with’ instead of ‘research on’ would enable us to ensure practitioners’ perspectives and lived realities were front and center in our work.

Because many youth arts nonprofits have been hit hard by COVID, my team wanted to do all we could to ensure that our work would actually contribute back to the community and fully honor practitioners’ experience navigating the past few years.

As my team’s primary researcher, I decided to use a community-engaged approach in order to build a sense of shared accountability and set ourselves up to explore a topic that we mutually agreed could bring tangible benefit to our partner organization. Community-engaged research is most common in public health efforts, but can be similarly leveraged in HCI to “conduct research that can translate more easily to real world settings” and “design more culturally- and language-appropriate interventions” (Source: 3. Handley 2010). In these efforts, a researcher aims to act as a “social designer, facilitator of processes, and agent of change” rather than the sole arbiter of knowledge and expertise (Source: 4. ACM Interactions 2022).

Selecting methods: interviews & observations

I decided to conduct semi-structured interviews in order to directly hear staff members’ perspectives, complemented by field observations to understand the environment and system they work within.

I drew from ethnographic methods in planning how to approach our research. Semi-structured interviews would not only give insight into staff members’ goals and perceptions in their own words, but also allow us to introduce ourselves and begin to develop trust. Similarly, field observations would enable us both to build an in-situ understanding of Urban ArtWorks’ environment and processes, and also to physically show up and demonstrate commitment.


phase 2: recruiting & choosing a research focus

After agreeing on a community-engaged approach, my team concentrated our efforts upfront on recruiting a partner organization and collaborating with them to identify an appropriate research focus area.

Research Timeline

Research timeline highlighting Phase 2 out of 4, which is recruiting & research focus

partner organization recruiting: urban artworks

We successfully recruited our partner organization, Urban ArtWorks, by iteratively refining our project pitch and conveying our genuine desire for our research to center their perspectives & organizational needs.

Our first main hurdle was to decide on the types of youth arts organizations which would be best suited for forming a co-design partnership with, and figuring out how to recruit them on a relatively tight timeline.

To this end, my team employed the following methods:

  • Recruiting criteria that ensured alignment while casting a broad net (e.g. youth arts education nonprofits we could regularly commute to who were able to commit to a 6-month partnership)
  • Recruiting spreadsheet with background & contact info for over 30 local youth arts nonprofits
  • Organized tracking of our communication status with each organization and when to follow-up
  • Standardized research elevator pitch and answers to common questions we encountered

Although many of the organizations we tried to recruit expressed that their staffing & resource constraints left them unable to participate in more than a one-off capacity, we were thrilled when Urban ArtWorks’ Executive Director was willing to connect with us. In our introduction, I explained why a research partnership can be worth the additional commitment, while also being clear about our roles and boundaries as researcher and designers. After learning more about their organization and sharing about our project and program, we agreed that a collaboration could be mutually beneficial.

research focus: student information management

Guided by two introductory interviews with Urban ArtWorks leadership, we mutually agreed to focus our research on their systems for managing student information.

After committing to a research partnership, we set out to agree on an area in which our work might be most fruitful both in generation knowledge & design plans for their organization, and as a learning opportunity for us as students.

We accomplished this through 3 unstructured half hour interviews with Urban ArtWorks leadership (two with the Executive Director, and a third which also included their Program Manager).

During these conversations, we gave input around where we believed we could contribute most, but ultimately let Urban ArtWorks guide our project scoping. After careful consideration, Urban ArtWorks’ Executive Director proposed that a fitting topic to research & design around would be systems for managing information about students. After careful consideration, Urban ArtWorks’ Executive Director proposed that a fitting topic to research & design around would be systems for managing information about students. We agreed!

research objective & questions

This research aimed to understand: ‘How do Urban ArtWorks’ participant information management systems impact their ability to achieve their organizational mission at a broader scale?’

Questions About Current Systems
  1. What systems does UA currently use for participant information management?
  2. What do UA staff members think is working well about these systems?
  3. What challenges have UA staff members faced with these systems as the organization has expanded, and how have they responded?
Questions About Future Systems
  1. How well do UA staff think their participant info management systems will hold up as the organization continues to scale?
  2. What are staff members’ goals and hopes around participant info management as UA grows?
  3. What changes do they anticipate needing to make to these systems in order to handle scaling?

phase 3: primary research

To understand Urban ArtWorks’ student information management systems, we paired interviews to directly hear staffers’ perceptions with observations to build an in-situ understanding of their environment and processes.

Research Timeline

Research timeline highlighting Phase 3 out of 4, which is primary research with partner organization

method 1: interviews

Our semi-structured interviews included a timeline design probe to help us orient to staffers’ different levels of zoom, and showed us staff members’ goals and perceptions in their own words.

Design Probe

Our first round of unstructured interviews made it clear that we needed a mechanism to gain a deeper-level understanding of their systems, so I advocated for including a timeline design probe in our semi-structured interviews.

With this design probe activity, participants were directed to write down each action they took and system they interacted with over the course of a given time period, in relation to student information management. This time period was agreed upon during the interview and selected to represent a cycle in which all key actions were performed. For some staffers, that meant a single youth program, while the Executive Director operates on a yearly scale.

Interview setup including completed timeline design probe activity

Interview Protocol

I led the creation of a 60 minute interview protocol that aimed to understand Urban ArtWorks staff members’ processes and perceptions around student information management systems.

After consent & introductions, I designated 20 minutes for our timeline activity, with the goal of understanding the specifics of how they interact with student information. With this as context, we moved into 15 minutes of discussing their perceptions, with the goal of understanding their feelings around their interactions with student information management systems.

Shortened version of interview protocol showing questions for a timeline activity

method 2: field Observations

Observations served as a complement to our interviews, and enabled us to build an in-situ understanding of Urban ArtWorks’ environment and processes.

My teammates and I engaged in two observation sessions, which gave us a feel for Urban ArtWorks’ environment and allowed us to build more familiarity with staffers. We also leveraged more directed observations in our co-design phase with Urban ArtWorks, which you can read about here.

I created a template to use when taking field notes based on the AEIOU format (Activities, Environment, Interactions, Objects, Users). This template also included space for questions & follow-ups, reflections, and sketches.

Shortened version of field note template


phase 4: analysis

I led my team in conducting an asset-based analysis in order to pinpoint the organization’s existing strengths regarding student information management, as well as opportunities to build upon them.

Research Timeline

Research timeline highlighting Phase 4 out of 4, which is analysis & design direction proposal

Analysis: asset-based approach

Through an asset-based approach to our analysis, we aimed to focus on identifying strengths which could be built upon in our co-design phase.

Whereas much of UX research begins with needs assessments that identify problems and focus on the weaknesses of communities, this deficit-centered models may reinforce a sense of inadequacy or cause distrust in researchers’ intent (and rightly so). This can make it difficult for communities to get motivated to make positive changes.

In contrast, an asset based approach fosters hope within a community by shifting the focus from "what's wrong with us" to "what's right with us." It focuses on “Discovering and affirming [...] underutilized assets and untapped potential” which can be put into use to improve current conditions (Source: 5. University of Memphis).

With this in mind, I created the following framing questions for our analysis:

  • What are some existing strengths & opportunities of Urban ArtWorks’ staffers, systems, and processes?
  • What are Urban ArtWorks staffers proud of?
  • What systems, processes, and strengths are present that we can build upon?

Analysis: affinity diagramming

I led my team in identifying themes around assets, opportunities, and challenges using an affinity diagramming process.

My team first started by cleaning interview transcripts and breaking them down into short chunks that contained individual pieces of information, each of which was placed on a Sticky note in Miro. We then filtered out chunks that were not relevant to our topic area.

Once we had our data prepared, I led us in our thematic analysis. I set up a structure of overarching categories that were based in our research questions but modified to help us answer our framing questions. Working within these broader questions, I led my team in multiple rounds of affinity diagramming in which we clustered similar data chunks (where any chunk could appear in multiple clusters), organized these to derive sub-themes, and then thoughtfully grouped them to create higher-level themes.

Screenshot of sticky notes representing data grouped into four overarching categories

Phase 4: findings

While crafting insights based on our interviews & observations, I took care to ensure they would highlight opportunities in a way that was situated with Urban ArtWorks staff members in order to catalyze a successful co-design phase.

findings premise

By centering our findings around a plane metaphor, we reflected the way Urban ArtWorks staffers had talked to us about their organization’s systems.

During our interviews, I noted that multiple staff members had talked about Urban ArtWorks as a moving object that they’re trying to keep up with. References to visceral imagery such as stepping aboard a moving train evoked a strong sense of the pace at which the organization is growing, and the balancing act that staffers must manage of maintaining day-to-day operations while envisioning their longer-term direction.

“We're building the airplane as we fly it.”

- Paul, Urban ArtWorks Projects Lead

insight 1: Evaluation

For youth to help steer the plane, they first need the structure to climb aboard.

What’s Already Working:

Staffers have a deep focus on building relationships with youth participants, and are already using this nuanced understanding to start evolving their feedback collection practices.

Challenge Staffers Face:

Staff members spoke of difficulties getting complete and meaningful evaluation data.

Opportunity:

How might we ensure feedback systems are accessible and meaningful to both youth participants and UA staff?

“How can we do not evaluation, but these kind of like formative assessment checkpoints throughout the program, to where at the end, when you do give [students] this post evaluation, they remember, 'Oh, we did do that' [...] or whatever the case may be so it's actually good data.”

Eboni, Urban ArtWorks Program Manager

insight 2: streamlining

Streamlined systems lessen the turbulence of bringing on new people.

What’s Already Working:

Staffers have started to make information management systems explicit through a programs checklist & onboarding document.

Challenge Staffers Face:

Systems that have developed organically over time are difficult to explain and teach to new staff members

Opportunity:

How might we streamline systems as well as the process of creating guidance that evolves alongside them?

insight 3: accessing information

An organized toolkit enables finding information without rummaging around in luggage.

What’s Already Working:

In order to improve organizational practices, staff members have recently adopted the project management tooling Asana, and are re-organizing existing data in Google Drive.

Challenge Staffers Face:

Because gradually accrued information has become dispersed over time, it's hard for staffers to find information on demand. This is further complicated by an array of tools that don’t automatically tie together.

Opportunity:

How might we develop a consolidated tooling plan to ensure staff members have access to the information they need, when they need it?


Outcomes

At the conclusion of our research, my team had the key components for a successful co-design phase: trust with our collaborators, opportunities to design around, and guidance for how to do so.

design recommendations

In addition to identifying three primary opportunity areas through our insights, I also leveraged our data and relationships with staff members to create a set of recommendations as to how our design could be successful within their environment.

Successful Designs Must:
  • Support sustainable growth by ensuring any longer-term visions come with manageable upgrades along the way
  • Center around an explicit, unified vision that leverages each staff member’s ideas and expertise
  • Develop a harmonized balance of formal and informal workflows that allows for both scaling and agile collaboration
Successful Designs Avoid:
  • Assuming that improvements to student information management systems will always be top priority
  • Creating more hats that staff members must wear, when they are already filling many roles
  • Needing to duplicate work, which is a current problem with student information mechanisms like paper forms

next steps

With our insights in hand, we were ready to kick off our co-design phase by checking our understanding with Urban ArtWorks and discussing proposed design directions.

Read more about our co-design phase here, in which we collaborated with Urban ArtWorks to design Mirror, a learning reflection platform that empowers education nonprofits to center student perspectives through holistic feedback.


Reflection

While a community-engaged approach is not appropriate for every research project, the depth of understanding and mutual accountability it generates is incomparable.

challenges

This project made it blatantly apparent to me that doing community-engaged research on a relatively tight timeline is no easy feat, and it demands that researchers show dedication, organization, and flexibility.

Our detailed workback schedule was crucial due to the delays from finding a partner organization and choosing a research focus together. In future community-engaged research efforts, I would either advocate for more time to recruit & develop a partnership, or build out contingency plans if recruiting efforts were going awry. For example, if an organization I contacted said they did not have enough capacity, I would try to recruit them for one-off interviews down the line.

joys

Having a research focus embedded deeply within an organization was immensely satisfying as a researcher who thrives on relationship-building and systems thinking.

In building real connections with our collaborators and attempting to have a more balanced power dynamic, this project continually challenged my preconceptions about my role as a researcher, forcing me to ask myself how I could add value without insisting my expertise gave me authority over our process and our collaborators.

From an analytical standpoint, I enjoyed the challenge of trying to grok Urban ArtWorks’ systems as a whole, and factor in relationships and roles in a way that I wouldn’t if I had been interviewing individuals from a wide range of organizations.

I am deeply grateful for all the support, kindness, and generosity from the staff and teaching artists at Urban ArtWorks who collaborated with my team for our Master’s capstone project.


Sources

  1. Gretchen Marie Ennis and Jane Tonkin (2018). ‘It’s like exercise for your soul’: how participation in youth arts activities contributes to young people’s wellbeing. Journal of Youth Studies. Apr 2018, Vol. 21 Issue 3, p340-359.
  2. ArtsFund Washington (2022). COVID Cultural Impact Study. https://www.artsfund.org/wa-covid-cultural-impact-study
  3. Handley M et al (2010). Community-Engaged Research: A Quick-Start Guide for Researchers. Clinical Translational Science Institute Community Engagement Program, University of California San Francisco. http://ctsi.ucsf.edu/files/CE/guide_for_researchers.pdf
  4. ACM Interactions (September 2022). Assets and community engagement: A roundtable with HCI researchers and designers. https://interactions.acm.org/archive/view/september-october-2022/assets-and-community-engagement
  5. University of Memphis (2019). Module 4 - Asset Based Community Engagement. https://www.memphis.edu/ess/module4