Guidelines for Program Review:The first step in program review is the administrative review of programs and disciplines. After consultation with appropriate other divisional deans, the coordinating divisional dean is responsible for completing the administrative evaluation for degrees, certificates, career studies certificates, and disciplines that have been approved for review by the Administrative Council. The administrative evaluation consists of two parts:

1. Productivity Evaluation:

Degree programs must meet the productivity guidelines of the State Council of Higher Education (SCHEV). In Spring 2003, SCHEV agreed to delegate to the State Board for Community Colleges responsibility for the review and discontinuance of any associate degree program that is non productive, with the following stipulations:

• Each community college will systematically review each degree program at least once every four years;
• Based on CIP code and standards congruent with SCHEV's minimum standards for productivity, the VCCS ] will monitor FTE enrollments and numbers of graduates for all approved associate degree programs;
• For any program that does not meet standards, colleges will submit to the VCCS (1) a plan to phase out the program, (2) justification for continuing it with low productivity, or (3) strategies to enhance the program's viability.
• The VCCS will report to SCHEV at least once every four years the results of its program viability review and describe any proposed changes to its policies and procedures.

As of 5/1/03, there is some uncertainty as to the standards that will be required of VCCS programs. The standards listed in a SCHEV report are (for a college of more than 5000 FTES):

Degree Program
Transfer(AA or AS);		AAA or AAS in Agriculture & Natural Resources, Business, Arts & Design, Public Service Technologies		AAS in Engineering, Mechanical, and Industrial Technologies		AAS in Health Technologies
FTES	Grads	FTES	Grads	FTES	Grads	FTES	Grads
24	17	18	12	13	9	10	7

Standards: VCCS
Beginning in 2002, the VCCS provides to NVCC the Report on Program Review/ Viability using the following standards:
20 FTES for transfer programs
15 FTES for occupational-technical programs
15 FTES for diploma programs (of course, NVCC offers no diploma programs)
12 students for certificates

The System Office's productivity report identifies programs at each college that did not meet the standards. Note that when the VCCS runs these reports, it considers each program separately, rather than lumping all within a CIP code together they way SCHEV does. Thus, SCHEV would view the Fine Arts AA and the Liberal Arts AA as one program but the VCCS would not.

2. Administration Evaluation

Standards for the administration of academic programs and disciplines have been developed and approved by the Curriculum Committee. [See the Curriculum Procedures Manual, revised December 1998, page 5] The coordinating division dean, in consultation with appropriate other division deans, will also be responsible for completing the Program/Discipline Administration Evaluation Checklist and indicating those standards which are met by the administration of the program. Standards that are not met will be examined further by the Program Evaluation Committee. The coordinating division dean may also identify administrative issues that should be addressed by the Program Review Committee.

Both the Productivity Evaluation and the Program/Discipline Administration Evaluation Checklist will become a part of the report submitted to the Curriculum Committee by the Program Review Committee. See Appendix D for copies of the Productivity and Administration Evaluation Checklists.

Format for the Review Report:

Chapter One: Program/Discipline Description:
Provide a brief overview of the program including its purpose, its components, a profile of the current student population, critical issues in the program field or discipline, and emerging trends.

Chapter Two: Curriculum
List the goals and objectives for each award (degree, specialization, certificate) along with the rationale for any revision the cluster will make before proceeding with the review. Evaluate whether the curriculum includes the optimum combination of awards with the best courses in the best sequence with the most current course content. Describe any revisions that would improve the curriculum. You will revise the curriculum after you have gathered data about student learning outcomes. If you made significant changes in the curriculum of your program since it was last reviewed, you may wish to describe those changes in this chapter. Review all course content summaries and make appropriate revisions and note that in this chapter.

Chapter Three: Student Learning Outcomes
This is the most important chapter in your review report. Describe how well students are meeting program goals. Include the findings of your assessment of students' attainment of general education goals and objectives and core computer competencies. Use multiple measures to assess student learning. You must use at least one direct measure of student achievement. You may include standardized tests, employer surveys, student surveys, internship reports, pre- and post-testing in selected courses, portfolio assessment, performance-based assessment, departmental exams in exit level courses, externally juried recitals or exhibits, demonstrations, pass rates on certification and licensure exams, capstone projects, common questions on exams in selected courses, course assignments and other course-embedded assessments. Appendix A describes some assessment measures.

Chapter Four:Instruction
Describe the instructional modalities in your program and how effective they are in promoting student learning. Evaluate the effectiveness of courses offered in non-traditional formats (e.g., weekend courses) as well as thorough alternative delivery systems such as online or TV courses. Survey current students or conduct focus groups. You may combine the chapters on instruction and student learning if you wish.

Chapter Five: Resources
Review the resources that support student learning in your program. Recommend changes if, and only if, you provide supporting evidence that new resources will improve student outcomes. Include a report on faculty satisfaction with and participation in professional development activities.

Action Plan: The action plan for the continued improvement of the program includes all recommendations and suggestions made in the review report. Each recommendation and suggestion is followed by a rationale for the action, the specific action to be taken, the person responsible for carrying out the action, and a due date. Your cluster will be required to report on the implementation of the action plan one year after the review report is approved by Administrative Council and again one year after that.

Assessment Activities by Semester:

Below you will find suggested activities for the three semesters allotted for the program review. These activities are recommended to help you collect the information you will need to evaluate the effectiveness of your program or discipline in a manner that will yield useful results for you and for the institution. They are not intended to be comprehensive or prescriptive.

You will also find below a suggested schedule for writing the parts of the review report that can be done concurrently with the assessment of student learning. Keep in mind that the chapters need not be written in the order they will appear in the final report.

Semester One

A. Review Program Goals and Objectives to determine the degree to which they

• are derived from and support the NVCC mission and goals statement
• support the general education goals of the college
• are consistent with expectations of businesses employing students, transfer institutions receiving students, and the needs of the community served
• are consistent with the purpose of the program as stated in the catalog

Now is not the time to change or revise the goals. That should come at the end of the process after you know more about student learning outcomes in your program. Of course, if the program goals and objectives are inconsistent with the College's mission and goals statement, with the expectations of businesses, transfer institutions, needs of the community, or purpose of the program as stated in the catalog, you should certainly start thinking about what needs to be changed. If for some reason there are no existing goals for your program, write them now. You cannot begin to assess student learning in your program until you have goals for the program.

Now is not the time to change or revise the goals. That should come at the end of the process after you know more about student learning outcomes in your program. Of course, if the program goals and objectives are inconsistent with the College's mission and goals statement, with the expectations of businesses, transfer institutions, needs of the community, or purpose of the program as stated in the catalog, you should certainly start thinking about what needs to be changed. If for some reason there are no existing goals for your program, write them now. You cannot begin to assess student learning in your program until you have goals for the program.

You also need to begin to write program-specific general education objectives for each of the eight general education goals of the college. Keep in mind that some of the objectives will be achieved by students by taking program-required courses. Other objectives may be achieved by students by taking the required general education (or liberal arts) courses required for the degree awarded by your program. For example, students may achieve the general education objective you will write for Goal 8 (wellness) by taking the PED 116, a course required in all degree programs. As another example, since a SPD course is required in all degree programs, if you write an oral communications objective for Goal 1 (communication skills), that general education objective may be achieved by students taking the SPD course.

B. Review the Current Curriculum
Review your current curriculum. Consider whether it is logically sequenced, whether there are courses that are duplicative in content, whether the curriculum is current and takes into consideration the emerging issues in your field or discipline. Is the curriculum comprehensive in addressing the goals of the program, general education objectives, and core computer competencies required by the college? You will want to consider information on grade distribution, success of transfer students, graduate surveys, student satisfaction surveys, and employer surveys. You will also want to consult with your Curriculum Advisory Committee if your program has one.

In general, changes in the curriculum should not be recommended at this time. That will come later and be justified by the data on student learning outcomes that you will gather. Some recommendations may arise just in terms of courses that are redundant or have not been offered for a period of time and, of course, you want to make note of these. But you should refrain from revising the current curriculum until the work on student learning outcomes has been done.

C. Decide on Measures of Student Performance
You should consider course-embedded assessments first. What program goals are being assessed well in program courses and which are not? How are general education objectives to be assessed? Think about what you are already doing in the classroom to assess student performance. For example, reading assignments, essay exams, and oral presentations that you regularly require in your courses can be used to demonstrate student achievement in college-level communication skills, a general education goal. As another example, you may decide on a series of questions that will be asked in common on final exams to demonstrate student achievement of program goals and objectives. Field experiences, capstone courses, and class projects are rich sources for assessing student achievement of program goals. There are many classroom research techniques - some of which you are probably already using - that are useful in assessing student learning. [See Appendix B.] The assessment coordinator can provide you with suggestions and resources for classroom assessment of student learning and can help you get organized.

Then consider what additional assessment you will need to conduct. Review the existing assessment instruments which the assessment coordinator can provide. Decide on the assessment instruments that can be used and administered this semester and those that must be developed. Decide on how assessment instruments such as surveys are to be administered. Decide what other measures of student performance will be used. You will also want to look at transfer success data and employment/placement rates.

D. Begin Assessment of Student/Employer Satisfaction
Administer surveys and implement what other measures of student learning you can this semester so results will be ready in the next semester. For example, a student satisfaction survey to ascertain students' perception of the learning environment or the quality of their educational experience should be administered this semester. You will also want to analyze data from graduate surveys. For reviews of occupational/technical programs, employer satisfaction surveys should be developed and administered during this semester. Employers are notoriously bad about responding to satisfaction surveys. You might find it more useful to use structured telephone interviews of a good sample of major employers instead. If your program has a student club, you might think of ways for student club members to interview employers either personally or by telephone. The assessment coordinator can provide you with sample Student Satisfaction and Employer Satisfaction Surveys.

E. Write Chapter One: Program/Discipline Profile including

• degree, specializations, and certificates included in the program
• campuses that offer the program
• a narrative description of the curriculum
• courses that support programs other than the one being evaluated
• a description of your current students
• specific industries and businesses served by the program
• institutions to which your current students transfer
• significant developments since the last evaluation
• emerging trends in your field or disciplin

The introduction should include a summary of the current curriculum, not a listing of courses, which can be found in the catalog. When you describe your current students, summarize in narrative form the information about your students included in the Information Packet provided to you by the Office of Institutional Research.

Semester II - Collecting and Analyzing Student Learning Outcomes Data

A. Continue Assessment of Student Learning
During this semester, assessment of student learning outcomes will be your primary focus. You will administer the assessment techniques chosen last semester and begin the analysis of the data you have collected. As you gather data and analyze it, the recommendations will begin to emerge. It is not necessary nor is it recommended that you gather all the data first, then analyze it. If you analyze the data as you gather it, you will find that the analyses will point out other areas in which you will want to do assessments and gather information. The assessment coordinator can provide you with assistance in the analysis of the information you are gathering.

B. Revise the Curriculum
As the results of the assessment of student learning are analyzed, you may begin to see the need to revise the curriculum. Is the curriculum working for the students? Are they prepared when they enter the workplace or transfer to a senior institution? Occupational/technical programs should meet with their advisory committees to discuss the curriculum. If you conducted employer satisfaction surveys last semester, decide what curricular revisions are indicated by your findings. Transfer programs should carefully consider transfer data on the success of NVCC students at senior institutions.

C. Review Resources
Review the adequacy of resources for your program. In any case where you consider the resources that support your program to be inadequate, you must justify that judgment by evidence that student learning is adversely affected. Consider the support personnel working in the program as well as the instructional and institutional academic support services that are unique and critical to your program. Are the classrooms and laboratory facilities adequate? If not, what must be done to make them adequate? Is the library collection available adequate? If not, consult with the librarians to determine what must be done to improve the holding. What about equipment and supplies? Again, if the data indicates that they are inadequate, you must describe what must be done to correct the situation.

D. Evaluate Instructional Modalities
Describe the various instructional modalities that students experience in your program or discipline. Highlight or showcase those that have been particularly effective in producing student learning. Describe how you know that these modalities were effective in producing student learning. Consider how you meet the differing learning styles of students in your program. Describe how the instructional modalities are appropriate to the purpose of the program and the goals of the program or discipline. If students have access to your courses through various distance learning opportunities and instructional delivery systems such as web-based courses, computer-based courses, and the Extended Learning Institute describe how students learning is assessed in these courses.

Semester III

A. Summary of Student Learning Outcomes
The assessment of student learning outcomes is at the heart of the program review. Before you finalize your report, you should complete a summary of what you have learned about student learning in your program to include in your report. In the summary you should indicate the following:

1. what learning objective (either program goal or general education objective) was assessed
2. the method by which the assessment was done
3. the assessment outcome
4. recommendations based on your assessment
5. actions to be taken to improve student learning

This summary will form the basis for your action plan and will be included along with a copy of the executive summary and action plan sent to members of the Curriculum Committee, provosts, divisions chairs, deans of student development, and LRC directors. It will also be included in the annual assessment report to the VCCS. You may want to use the matrix format found in Appendix E. Below is an example of how a student learning outcomes summary might look.

Learning Objective	Assessment Techniques	Assessment Outcome	Recommendations	Actions taken or to be taken
Program goal 1Gen Ed objective 1:1Students will write in organized, clear, grammatically correct English appropriate for a specific subject and audience.	Written report in (list of courses) graded by program faculty for content and quality of writing appropriate to the program.	75% of the students received a satisfactory grade on their reports.	Students need a better understanding of the criteria for a satisfactory report.	A handout on the requirements for a satisfactory report will be provided to students and placed in the writing lab. Students will be provided with a n example of a satisfactory report
Pro gram Goal 2Gen ed objective 4:1Students will demonstrate the ability to work effectively on a team.	Faculty observationduring class-time devoted to group project work in (list course).Student written self-evaluation in (list course).	Faculty observed that 75% of the students demonstrate d the ability to work effectively on a team.90% of the self-evaluations indicated good understanding of effective teamwork	Students have some difficulty in putting the principles of effective teamwork into practice	Have students critique a video showing a team at work and have them indicate which principles were well-employed and which were not.
Continue in a similar manner with a summary of other program goals and general education objectives that were assessed.

B. Revise Goals
Now that you have assessed student learning and evaluated the curriculum, instructional modalities, and resources, you may find that your program or discipline goals need to be revised. Be sure to include your revised goals and objectives in your final report and include the general education objectives for your program.

B. Complete the Final Draft of the Review Report
It will be helpful to have an outside reader review your report at this time to help you strengthen it as well as to help spot areas where additional data may be useful. The assessment coordinator can serve in this role.

C. Action Plan and Summary of Student Learning Outcomes
As the results of the various assessments are analyzed, recommendations based on the findings will start to cluster. You should include a chapter in your report that describes actions taken to improve the program or discipline during the evaluation process and provides a plan to accomplish the recommendations and suggestions you make.

As the recommendations and suggestions emerge, so will the action plan. Resist the urge to write the action plan before all the assessments are completed and recommendations are made. Recommendations and suggestions should be placed throughout the report where the supporting data is reported. Make sure that each recommendation and suggestion is clearly supported by the evidence you present. It is very helpful (both to you as you write the report as well as the report readers later) to list all recommendations and suggestions made at the end of the chapter in which they appear. Each recommendation and suggestion that you make throughout the report must also appear as part of your action plan.

A recommendation describes an action that must be taken to achieve the goals and objectives of the program or discipline and to make the program effective. The recommendations must be tied to supporting evidence in your report and must be directly and clearly related to the goals and objectives of your program or discipline. Recommendations are to be related to program or discipline specific issues, not college-wide policies and procedures, and include an action to be taken, a rationale for the action, the name of the person responsible for taking the action, and a due date.

Your division chair will be required to report on the implementation of your action plan one year after it is approved by the Administrative Council and again one year after that. Therefore, recommendations should address those areas over which you have some control and influence. Do not, for example, indicate that an action must be taken by "the college," but name the person who will be responsible and accountable for carrying out the recommended action. Finally, remember this is your action plan. It will detail what you and your colleagues plan to do to improve the effectiveness of your program.

A suggestion describes an action that should be taken to improve the program's or discipline's effectiveness. Each suggestion must also define a plan for action in the same way a recommendation does. Obviously, suggestions are not as strong as recommendations and therefore should define areas that while not crucial to the effectiveness of a program or discipline would, if carried out, enhance it.

The following format is recommended for each recommendation and suggestion in your action plan:

RECOMMENDATION: Include the page number in the report where it appears.

RATIONALE: The evidence from your report that supports the recommendation or suggestion.

ACTION TO BE TAKEN: This should be specific in nature. Just what, exactly, is to be done?

RESPONSIBILITY Name a single person (or position) as the responsible party (and it should be someone over which you have influence since this is, after all, your action plan).

DATE BY: Be realistic here. The wheels of academe grind slowly.

D. Write the Executive Summary
The intent of the Executive summary is to allow readers from a variety of audiences to focus immediately on the most important finding of your review. Your summary should be concise and done by chapter and/or section allowing readers to find the part of your review that may hold particular interest for them.

E. Finalize the Review Report
After you have completed the report, send it to the assessment coordinator. Since each division chair and provost who has responsibility for the program being evaluated must be given an opportunity to review and comment on the report, copies will be sent to them along with a comment sheet. A copy of all completed comment sheets will be sent to the chair of your review committee.

In addition to the main body, the final report will include a title page listing the members of the review committee, provosts' and division chairs' comment pages, an executive summary, the student learning outcomes summary, the administrative evaluation, and any appendices you feel are essential to the understanding of the report. The review report is due at the end of Semester III. Be judicious in what you choose to include in the appendices and include only what is absolutely necessary for a clear understanding of your report. Overly large appendices are not encouraged due to storage space and in general are not circulated unless specifically requested by a reader.

Your report will then be read by two persons chosen by the associate dean for curriculum services. These readers will provide you with written comments on your report. [See Appendix E for a copy of the readers' guidelines.] The executive summary, student learning outcomes summary, and action plan will be forwarded to the Curriculum Committee. You will be invited to attend the meeting at which your report will be discussed. The readers will be asked to present their comments to the Curriculum Committee. The committee members may have questions to ask you about the report and may wish to amend the action plan. Finally the action plan is forwarded to the Administrative Council for approval. You will be notified by the assessment coordinator when your action plan has been approved by the Administrative Council.

The full report and action plan with amendments is sent to the director of each Learning Resource Center with a request to keep it on file and to share it with those who may have an interest in it. The deans of student development are sent the action plan and asked to share it with the counselors. The assessment coordinator will report your findings and your action plan to the VCCS in the annual assessment report.

Suggested Chronology of Program/Discipline Review Activities:

FIRST SEMESTER

ACTIONS TO BE TAKEN	BY WHOM?	DUE DATE?
1. Coordinating Division Chair completes the Productivity and Administrative Evaluations.	Coordinating Division chair	Two Months before the semester the review begins
1. Discuss methods of assessing student learning and decide on available measures.	Review Committee/assessment coordinator	Mid-semester
2. Begin development of additional measures of student learning.	Review Committee	Mid-semester
3. Review current goals and curriculum.	Coordinating Division chair	Mid-semester
4. Conduct initial assessments of student learning.	Program faculty	During semester
5. Write an overview of the program.	Review Committee	End of semester
6. Complete development of additional measures of student learning.	Review Committee/assessment coordinator	End of semester
7. Administer Student and Employer Satisfaction Surveys.	Program faculty/Review Committee	End of semester

SECOND SEMESTER

ACTIONS TO BE TAKEN	BY WHOM?	DUE DATE?
1. Summarize results of initial assessment of student learning and review additional measurement instruments .	Review Committee Chair	Cross-campus Day
2. Finalize format of Report.	Review Committee	Cross-campus Day
3. Continue assessment of student learning.	Program faculty	During semester
4. Analyze data collected first semester	Review Committee/Assessment Coordinator	Mid-semester
5. Write Chapters on Resources, Instruction	Assigned Faculty	End of semester

THIRD SEMESTER

ACTIONS TO BE TAKEN	BY WHOM?	DUE DATE?
1. Write summary of student learning outcomes	Review Committee	Cross-campus Day
2. Revise Goals.	Review Committee/ Cluster	Mid-semester
3. Write chapters on Goals, Student Learning Outcomes, Curriculum.	Review Committee	End of Semester
4. Final Draft of Report completed.	Review Committee	End of Semester
5. Write action plan.	Review Committee	End of semester
6. Write Executive summary.	Review Committee	End of Semester
7. Finalize Review Report.	Review Committee	End of semester