This study examines how district leaders use research in their instructional decision-making in four large urban districts. We focus on organizational routines, which are a central medium through which instructional decisions are made in school districts. Routines may matter for research use because they bring particular people together at particular moments in the decision-making process, shaping what and how decisions are made, and likely the role of research therein. However, organizational routines have received little attention in existing scholarship on research use.
For this report, we focused on 140 interviews with district leaders that related to district organizational routines around ELA professional development in four urban school districts.
NCRPP Study 1 Technical Report No 5 FINAL Dec 2020
Research Questions
- How, if at all, do organizational routines structure decision-making?
- What research and other forms of information do district leaders use in their decision-making?
- How does the structure of organizational routines shape the role of research and other forms of information in decision-making?
Key Findings
- In all four districts, district leaders accomplished the complex work of instructional decision-making around ELA professional development by using multiple, interrelated, routines that both divided up decision-making into different tasks and also connected decision-making between individual routines.
- Districts divided-up the disparate work of decision-making into three different types of routines: design, deployment, and diagnosis.
- All four districts drew on common sources for information, including data, research, individuals, and organizations. But, each district had distinct portraits of information use in their decision-making. They varied in the number of distinct information sources (range), the relative distribution of information types (balance), and the degree to which a fewer or larger number of district leaders invoked research and other forms of information (spread).
- The type of the routine, whether it be design, deployment, or diagnosis, influenced what information district leaders used.
- The presence of connections or lack of connections between routines shaped the degree to which research use in one routine influenced the work in other routines within a district.
- We uncovered a new form of research use, which we call latent use. Latent use occurred as district leaders in one routine embedded research in artifacts, which then guided the work of leaders in other routines in substantive ways.
This report describes a study of the activities and influence of three different types of research-practice partnerships (RPPs) that shared a common focus on improving mathematics teaching and learning. The three types of RPPs were a networked improvement community (NIC), a design research partnership (DRP), and a research alliance (RA).
The findings are derived from analyses of 63 observations of partnership meetings, 58 different products, 122 surveys, and 133 interviews across the three RPPs.
Research Questions
- How did the RPPs differ with respect to their organization and major activities?
- What kinds of research products did the RPPs produce?
- What influence did the RPPs have on partner districts’ decisions regarding policies, programs, and practices?
- What influence did the RPPs have on sharing research-based ideas with their district leader partners?
Key Finding
- RPPs overlapped and differed in their organization and major activities. All engaged in research activities, with two also engaging in design research practices, such as developing, testing, and iteratively refining an innovation designed to improve teaching and learning outcomes, and one engaging in improvement science practices, including Plan-Do-Study-Act cycles.
- The three RPPs produced multiple research products. The RPP using a NIC approach most commonly produced and shared research- based improvement strategies, while the other two RPPs most commonly shared their work and findings at academic conferences. It was common in two of the three RPPs for research and practice partners to co-author products together.
- RPPs influenced districts’ decisions and research use. Education leaders active in all three of the RPPs reported that their partnerships’ work influenced the design of professional development in the district. RPPs influenced other policies and practices in the involved districts as well, including providing new frameworks for thinking about mathematics, and integrating new practices modeled in the partnership into the district.
- Education leaders reported that they consulted their research partners in some key decisions. RPPs differed with respect to which decisions they were consulted on, however, across decisions related to topics including professional development design, scaling up a program, and program resource allocation.
Within these broad findings were important contextual differences related to the nature and structure of each RPP.
This report presents results from a study of the Council of State Science Supervisors (CSSS). CSSS is a professional association for state science leaders that works to sustain and nurture a dynamic learning community and to empower its members to be effective and articulate advocates for quality science education at the local, state, and national levels.
Given that many state science supervisors are working to support implementation of the Framework for K-12 Science Education, we sought to develop an understanding of CSSS members’ roles and responsibilities, as well as their use of research to inform state implementation decisions.
Key Findings
Findings Related to Roles and Activities
CSSS roles. The survey asked respondents about the roles they have assumed within CSSS. In response to these items, CSSS members most often reported serving as conference presenters or participants, at both their state science conferences and the CSSS annual meeting. As compared to other activities, they reported less frequent engagement in CSSS leadership activities, such as committee and board meetings. Overall, associate, honorary, and affiliate members were more often engaged in out-of-state or national activities than state members.
CSSS activities. Respondents were asked to report how often they participated in a range of CSSS activities over the last three years. Overall, respondents reported frequently accessing information from the CSSS listserv, as well as participating in CSSS-sponsored webinars, consulting with CSSS members, and collaborating with other states. State members were significantly less likely to report visiting other states and presenting at national meetings than associate, honorary, and affiliate members. This difference may be related to variation in the roles and responsibilities of state and non-state members.
State activities. Respondents also reported engagement in a variety of state-level activities in the areas of curriculum, instruction, assessment, and professional development. They reported being highly involved in state policy decisions related to science standards, yet they were less involved with the implementation of curricula that reflected those standards. They also reported playing key roles in assessment and professional development, although few respondents had authority to make decisions related to resource allocation or contract selection for assessment systems or professional development providers.
Professional development. Based on a request from CSSS leaders, the 2017 survey included items related to teacher professional development (PD). Respondents were asked to identify the PD offered in their state that afforded teachers the best opportunity to learn about the Framework. Findings revealed that these opportunities tended to be led by a state agency or local leader and funded by federal grant programs. These PD opportunities covered foundational concepts such as the three dimensions of science learning in the Framework, Next Generation Science Standards (NGSS), or instructional planning; however, they tended to pay less attention to designing three-dimensional assessments and developing students’ skills.
Findings Related to CSSS Member Research Use
Specific pieces of research found useful or shared with others. The survey asked respondents to name a specific piece of research they found useful for informing their state’s decisions related to implementation of the Framework, as well as a piece of research they shared with district or school leaders. The research respondents used to support implementation of the Framework most often focused on student learning and classroom assessment, while the research they shared with local leaders most often focused on classroom assessment and pedagogical practices. Few respondents named research focused on the needs or assets of particular student subgroups (e.g., by race/ethnicity, gender, socioeconomic status, or language). Overall, the pieces of research named were primarily research reports or policy briefs, particularly those published by the National Academies of Science, Engineering, and Medicine, as well as peer-reviewed journal articles.
Research trustworthiness. Respondents were asked to indicate why they found the particular pieces of research they named trustworthy. The most often cited reasons were if the research findings applied to their state context, or if the research gave them new ideas to support implementation of the Framework. A less commonly cited reason was that the research methods were rigorous.
Sources used to obtain research. The survey asked respondents how often they obtained research from a list of 13 sources. Of these, many respondents indicated obtaining research through CSSS or colleagues in their state departments of education. Far fewer respondents indicated seeking out research from the National Science Education Leadership Association (NSELA) or from the What Works Clearinghouse (WWC).
Efforts to acquire research. We asked respondents to indicate whether they would seek out research under different conditions. Although a majority said they would look for research to inform a new problem or decision, fewer said they would contact researchers directly under these circumstances, especially researchers they did not already know.
Findings Related to Research Networks
Research networks. The survey asked CSSS members to whom they have turned for research to inform their state’s implementation of the Framework. Findings from these social network questions revealed that associate, honorary, and affiliate members served as prominent sources of research in the areas of curriculum, instruction, assessment, and professional development. University faculty not affiliated with CSSS were also frequently named as sources of research. Additionally, new state members were more likely than veteran state members to name individuals from their own states as sources of research. And, while some respondents facilitated the exchange of research within the professional association, or between CSSS members in different states, others facilitated the exchange of research between CSSS members and researchers unaffiliated with the association.
Relations between activities and networks. Our Year 1 report indicated that an important next step in this study was to explore relationships between CSSS members’ roles and activities and their use of research. We used Year 2 data to examine these relationships, and found that participation in structured CSSS activities, particularly substantive meetings as compared to planning meetings or more informal interactions such as webinars, were important for facilitating the exchange of research among CSSS members.
Click the image above to download the full report and read more about findings and recommendations.