PBA Home > Institutional Research & Analysis > Surveys > Alumni Survey 2003 > Method

Alumni Survey, Summer 2003

Method

Population and Sampling Frame

The population for the 2003 Alumni Survey consists of alumni who had earned a bachelor's, master's, or doctoral degree between August 1998 and May 2000. Only alumni with U.S. mailing addresses, received from the Alumni Foundation, were included in the sampling frame. Only about 60% of all alumni who earned degrees during this period had U.S. addresses available. Surprisingly, U.S. addresses were available for a majority (76%) of international alumni. Fifty-nine percent of bachelor's-degree recipients had addresses available. This percentage is smaller than those for master's (71%) and doctoral-degree recipients (74%). A diagram is provided to illustrate the population, sampling frame, the group of alumni who did not have addresses, and to provide additional information about the sampling design.

Alumni who had earned more than one degree between August 1998 and May 2000 were included only once in the sampling frame, based on the program/major with the smaller enrollment. For example, if an alumnus earned bachelor's degrees in anthropology and sociology, then he or she was included in the population for anthropology (but not for sociology), because anthropology typically has a smaller enrollment than does sociology. The rationale for this method is that allocating multiple degree earners to small programs/majors increases the likelihood of having sufficient numbers of alumni within those programs/majors for sampling and reporting purposes.

In large programs that offer several majors, small majors were aggregated. For example, many alumni in business administration majored in finance, marketing, and information systems. Relatively few majored in real estate, international business, and tourism management. Therefore, the latter three majors (along with several others) were aggregated. This increased the likelihood that a report could be prepared for them. The aggregation of small majors is illustrated in Table 1, which contains counts of the population and sample, by degree, college, program, and major. The variable denoting program/major combinations appears under the heading of "PBA code." In business administration, for example, the code 520201x denotes nine small business administration majors that were aggregated for reporting purposes.

Sample

The target sample size for this survey was 4,000. For bachelor's and master's degree recipients, a simple random sample was drawn from within each program/major combination. All doctoral recipients with U.S. mailing addresses were included, due to their relatively small numbers. For statistical reasons, a minimum of six respondents were required in order to produce a program/major report. The sampling plan reflected this, in that just enough alumni were sampled from large program/majors (i.e., those with 60 or more degree recipients who had U.S. mailing addresses) to ensure that we could produce a report for them even if response rates were as low as 10%. The sampling plan is illustrated in Table 2.

Table 2. Sampling Plan

Program size*	Sample size or fraction			Total
	Bachelor's	Master's	Doctorate
Large	60	60	100%
Medium	100%	100%	100%
Small	15%	15%	--
Total Sample	2,590	1,023	386	3,999
Overall fraction	55%	82%	100%	63%

*Large: 60 or more degree recipients with addresses
Medium: 11-59 for bachelor's, 7-59 for master's

For small bachelor's and master's programs/majors (those with fewer than 11 and 7 degree recipients, respectively), 15% of the degree recipients were selected, because there was little chance that at least six alumni per program would respond and that we could, therefore, produce reports. Although small program/majors could have been excluded for purposes of providing individual program/major reports to departments, it was essential to sample them so that the responses of alumni could be generalized to colleges and to the campus as a whole

Additional information concerning the sample, sampling frame, and population is shown for each degree level in Table 3. This table illustrates that the percentage of bachelor's-degree recipients included in the sampling frame (58%) is smaller than the percentages of master's (66%) and doctoral-degree recipients (67%).

Table 3. Sample, Sampling Frame, and Population, by Degree Level

Questionnaire

Data Collection

Data collection was originally planned for May 2003, but difficulties in the development and production of the questionnaires delayed it until July 2003. The initial deadline for returning questionnaires was July 26, 2003, but this was extended twice to accommodate some respondents who did not receive questionnaires on time due to inaccurate mailing addresses. Data collection ended in late September.

Many questionnaires were returned to CU-Boulder marked as undeliverable or otherwise showing indications of never having been received by intended alumni. When possible, the Alumni Foundation provided updated address information for the intended recipients of undeliverable questionnaires. Even after several attempts at correcting addresses and re-mailing questionnaires, there remained about 422 undeliverable questionnaires.

Response rates vary considerably across programs, ranging from 0% (e.g., Film Studies; MCD Biology, doctoral level; Fine Arts) to 100% (Ethnic Studies; Business Administration: Minerals Land Management, Transport/Distribution Management; Communication, master's level; MCD Biology, master's level; Mathematics, master's level; and Sociology, master's level). The median response rate across program/major combinations is 18%, which is similar to the overall response rate of 19%.

A diagram illustrates that a substantial number of Alumni (4,673) simply could not be included in the survey because we either did not have U.S. mailing addresses available, or had inaccurate mailing addresses. These alumni are referred to as the "non-covered" group in the diagram. If we consider only those alumni who actually received a questionnaire (3,577), then the overall (adjusted) response rate is somewhat higher (21%). This rate is lower than that of the 2001 Four-Year-Out Alumni Survey (48%). The difference in response rates for these two surveys might be related to the difference in incentives offered to participants in each. In the 2001 survey, a $2 bill was enclosed with each of the 450 mailed questionnaires. Although this was shown to be an effective incentive in the 2001 survey, it was not economically possible in the re-designed 2003 survey, given its much larger sample. In the 2003 survey, a chance at winning a single prize of $1,200 was offered to all alumni who returned a completed questionnaire.

Across degree levels, adjusted response rates range from 19% (bachelor's) to 28% (doctorate). Across degree levels and discipline college, adjusted response rates range from about 13% to 42% (excluding response rates for Journalism doctoral recipients, of which there were only four who received questionnaires). Adjusted response rates for all colleges, by degree level, are shown in Table 4.

Table 4. Adjusted Response Rates, by Degree Level and College

Reporting

In addition to the program-level results displayed on this site, paper reports were prepared and delivered to program chairs in November 2003. The paper reports consisted of tabled results in a similar format to those displayed here. The key to whether a program received a report at all is the number of its alumni who returned completed questionnaires. Results are reported here (and were delivered in paper format as well) for programs with at least six respondents at the bachelor's, master's, or doctoral level. If a program meeting this requirement had fewer than six responding alumni at any degree level, then the program still received a report, but the results were excluded for degree levels not meeting the minimum. For example, a total of 16 alumni in Instruction and Curriculum returned completed questionnaires, 14 of whom were master's degree recipients, and 2 of whom were doctoral degree recipients. A report was therefore prepared for Instruction and Curriculum, but results for its doctoral recipients were excluded.

All completed questionnaires were returned to program chairs, regardless of number of respondents, so that the responses to open-ended questions could be reviewed. Open-ended questions include Questions 3 and 10, the last part of Question 9, and the question about what one thing could be done to improve a person's degree program.

The statistics reported in the tables and graphs for individual programs are unweighted. In contrast, the percentages reported for the entire campus, for degree levels over the entire campus, and for colleges are weighted to adjust statistically for different sampling fractions and response rates over programs.

Last revision 01/10/08