Matrix Overview of Focus and Program Criteria Standards

American Youth Policy Forum

Programs are categorized as Effective
(www.aypf.org/publications/mendel/MendelRep.pdf)

Mendel, R.A. (2001). Less Hype, More Help: Reducing Juvenile Crime, What Works - and What Doesn’t. Washington DC: American Youth Policy Forum.

The goal of the American Youth Forum is to enable policymakers, national and state governments, and organizations to develop, enact, and implement policies regarding the nation’s youth. The book describes programs dealing with reducing juvenile crime, asking what works and what does not work, what is commonplace in policy for reducing juvenile crime, and what changes need to be made to make sure the program is delivered with the best practice. Many programs are described based on a review of the scientific literature; however, no specific criteria for the inclusion of programs is provided.

Blueprints for Violence Prevention

Programs are divided into Model and Promising
(www.colorado.edu/cspv/blueprints)

Elliott, D.S. (Series Editor) (1997-2004). Blueprints for Violence Prevention (Vols. 1-12). Boulder, CO: Center for the Study and Prevention of Violence, Institute of Behavioral Science, University of Colorado.

As evident by its title, Blueprints for Violence Prevention’s main objective is that of violence prevention in children and adolescents from birth to age 19. Programs focus on violence, delinquency, aggression (including predelinquent aggression), and substance abuse. Criteria for Model and Promising programs include: evidence of deterrent effect with a strong research design (experimental or quasi-experimental) on one of the above outcomes. Other criteria that Model programs must meet include sustained effects for at least one year post-treatment and replication at more than one site with demonstrated effects.

Center for Mental Health Services, U.S. Department of Health and Human Services

Programs are categorized as Effective
(www.prevention.psu.edu/pubs/Mental_Health_pubs.html)

Greenberg, M.T., Domitrovich, C., & Bumbarger, B. (2000). Preventing Mental Disorders in School-Aged Children: A Review of the Effectiveness of Prevention Programs (Revised). State College, PA: Prevention Research Center for the Promotion of Human Development, College of Health and Human Development, Pennsylvania State University.

With help from the Center for Mental Health Services, the Prevention Research Center for the Promotion of Human Development at Pennsylvania State University reviewed different intervention programs dealing with the reduction of risks or effects of psychopathology in school aged children. Studies were excluded in which children were given diagnostic interviews and met criteria for DSM-III R or DSM-IV disorders. The age focus was restricted to children from ages 5 to 18. Programs that met the review requirements had to be evaluated using an adequate comparison group with either randomized or quasi-experimental design with an adequate control group. Studies had to have pre and posttest data and preferably follow-up data. They also had to have a written implementation manual. Universal, selective and indicated prevention programs were identified that produced improvements in specific psychological symptomology or factors directly associated with increased risk for child mental disorders. Programs showing reduction in psychiatric symptoms were also included in the review.

Communities That Care (CTC), Substance Abuse and Mental Health Services Administration (SAMHSA)

Programs are categorized as Effective
(ncadi.samhsa.gov/features/ctc/resources.aspx)

Communities That Care. (2004). Communities That Care Prevention Strategies: A Research Guide to What Works. Washington, DC: SAMHSA.

Communities That Care focus on preventing adolescent substance abuse, delinquency, teen pregnancy, school dropout, and violence as well as promoting the positive development of youth and children. Programs focus on the family, school, and community. The criteria include: (1) programs address research based risk factors for substance abuse, delinquency, teen pregnancy, school dropout and violence; (2) increase protective factors; (3) intervene at developmentally appropriate age; and (4) show significant effects on risk and protective factors in controlled studies or community trials.

Department of Education, Safe Schools

Programs are divided into Exemplary and Promising
(www.ed.gov/admins/lead/safety/exemplary01/panel.html)

Relevant outcomes are related to making schools safe, disciplined, and drug-free: reducing substance use, violence, and other conduct problems and positive changes in scientifically established risk and protective factors. Program criteria is carefully and thoroughly described on the website, and includes (1) evidence of efficacy/effectiveness based on a methodologically sound evaluation that adequately controls for threats to internal validity, including attrition; (2) the program’s goals with respect to changing behavior and/or risk and protective factors are clear and appropriate for the intended population and setting; (3) the rationale underlying the program is clearly stated, and the program’s content and processes are aligned with its goals; (4) the program’s content takes into consideration the characteristics of the intended population and setting; (5) the program implementation process effectively engages the intended population; (6) the application describes how the program is integrated into schools’ educational missions; and (7) the program provides necessary information and guidance for replication in other appropriate settings.

Mihalic and Aultman-Bettridge (2004)

Programs are divided into Exemplary, Promising, and Favorable.

Mihalic, S., & Aultman-Bettridge, T. (2004). A Guide to Effective School-Based Prevention Programs. In W.L. Tulk (Ed.), School Crime and Policing. Englewood Cliffs, NJ: Prentice Hall Publishers.

Programs are all school-based. Exemplary and Promising programs utilize Blueprints criteria and outcomes. Favorable programs broaden the outcomes to include factors relevant for school safety and success, such as school disciplinary problems, suspensions, truancy, dropout, and academic achievement. These programs may also have weaker research designs than the standard held for Blueprints, however, there is “reasonable” scientific evidence that behavioral effects are due to the intervention and not other factors. These programs all have experimental or matched control group designs.

National Institute of Drug Abuse

Programs are categorized as Effective
(www.drugabuse.gov/Prevention/Prevopen.html)

National Institute on Drug Abuse. (2003). Preventing Drug Use among Children and Adolescents: A Research-Based Guide for Parents, Educators, and Community Leaders, Second Edition. Washington, DC: National Institute on Drug Abuse.

The focus is on drug prevention and reduction. Although prevention principles are mentioned in the report, there are no specific criteria for program inclusion.

National Registry of Evidence-based Programs and Practices (NREPP), Substance Abuse and Mental Health Services Administration (SAMHSA)

Programs are listed as Reviewed (Users must use NREPP's database to find rating)
(nrepp.samhsa.gov)

NREPP, formerly the Center for Substance Abuse Prevention's Model Programs Initiative, is a voluntary, self-nominating system in which intervention developers elect to participate. NREPP is an online searchable database. NREPP is intended to serve as a decision support tool, not as an authoritative list of effective interventions. SAMHSA does not approve, recommend, or endorse specific interventions. Being included in the registry therefore does not mean an intervention is "recommended" or that it has been demonstrated to achieve good results in all circumstances. NREPP now provides ratings for each individual outcome targeted by an intervention, not ratings of interventions overall. This emphasizes that many interventions are designed to achieve multiple outcomes and can be more (or less) effective in achieving some of these outcomes than others. (The former system categorized programs holistically into "Model," "Effective," or "Promising" and did not provide information on interventions that did not receive one of these three ratings.) NREPP's ratings and intervention summaries synthesize and evaluate the research on specific interventions and in this way can help potential users of these interventions better understand their scientific evidence base.

In addition, NREPP now evaluates and reports on a new dimension called Readiness for Dissemination. This concept was not part of earlier NREPP reviews. Readiness for Dissemination is essentially a measure of the availability and quality of training and implementation materials. This measure responds to the fact that practical issues such as the quality of training can affect implementation fidelity (i.e., how closely the prescribed model and procedures or dosage are followed). This can play a significant role in determining whether a particular intervention will be effective in practice.

OJJDP Model Programs Guide (Title V)

Programs are divided into Exemplary, Effective, and Promising
(www.dsgonline.com/mpg2.5/mpg_index.htm or www.ojjdp.ncjrs.org/titlev)

Title V. Training and Technical Assistance Programs for State and Local Governments: Effective and Promising Programs Guide. Washington DC: Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice.

OJJDP has a focus on assisting communities with delinquency prevention strategies. The Title V programs are divided into categories of community, school, peer, and the individual. To be classified as Exemplary, the program required evidence of statistical deterrent effect using randomized treatment and control groups.Effective programs had evidence obtained with a control or matched comparison group but without randomization. Promising programs had evidence of a correlation between the prevention program (generally pre/post) and a measure of crime.

Sherman, et al. (1997)

Programs are listed as Effective
(www.ncjrs.org/works)

Sherman et al. (1997). Preventing Crime: What Works, What Doesn't, What's Promising (NCJ 165366). Washington, DC: National Institute of Justice, Office of Justice Programs, U.S. Department of Justice.

The main focus of the Sherman report is crime prevention. The methodological rigor of each program was rated on a scale of 1 to 5. In order to obtain a score of “3,” programs had to employ some kind of control or comparison group. If the comparison was to more than a small number of matched or almost randomized cases, the study was given a score of “4.” If the comparison was to a large number of comparable units selected randomly, the study was scored as a “5.” Programs were assessed as “working” if they had two or more evaluations with 3 or higher and statistical significance tests showed the program effective. Programs were assessed as “promising” if they had at least one evaluation with a score of 3 or higher showing effectiveness. For this report, all “working” and “promising” programs were classified as “Effective.”

Strengthening America’s Families

Programs are divided into Exemplary 1, Exemplary, Model, and Promising
(www.strengtheningfamilies.org)

The National Program Review Committee, the University of Utah, and CSAP reviewed the programs that focused on family therapy, family skills training, in-home family support, and parenting programs. Each program was rated on theory, fidelity, sampling strategy, implementation, attrition, measures, data collection, missing data, analysis, replications, dissemination capability, cultural and age appropriateness, integrity, and program utility and placed into the following categories:

Exemplary I: Program has experimental design with randomized sample and replication by an independent investigator. Outcome data show clear evidence of program effectiveness.
Exemplary II: Program has experimental design with randomized sample. Outcome data show clear evidence of program effectiveness.
Model: Program has experimental or quasi-experimental design with few or no replications. Data may not be as strong in demonstrating program effectiveness.
Promising: Program has limited research and/or employs non-experimental designs. Data appears promising but requires confirmation using scientific techniques.

Surgeon General's Report (2001)

Programs are divided into Model and Promising: Level 1-Violence Prevention and Level 2-Risk Factor Prevention
(www.surgeongeneral.gov/library/youthviolence/default.htm)

U.S. Department of Health and Human Services (2001). Youth Violence. A Report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Injury Prevention and Control; Substance Abuse and Mental Health Services Administration, Center for Mental Health Services; and National Institutes of Health, National Institute of Mental Health.

The primary focus of the report by the Surgeon General is violence prevention and intervention. The criteria the Surgeon General set were appropriately rigorous methods of inquiry and sufficient data to support the conclusions. Model programs have rigorous experimental design (experimental or quasi-experimental), significant effects on violence or serious delinquency (Level 1) or any risk factor for violence with a large effect size of .30 or greater (Level 2), replication with demonstrated effects, and sustainability of effect. Promising programs meet the first two criteria (although risk factors of .10 or greater are acceptable), but programs may have either replication or sustainability of effects (both not necessary).