Assessing Higher Education Outcomes in Brazil

Brazil is one of the few countries to have developed a full-fledged program to assess outcomes at the higher education level, for both public and private institutions. The National System of Higher Education Assessment (SINAES) was established by federal law in 2004, building upon an earlier program started in 1996, and is administered by the Ministry of Education. The system divides all programs into three groups and operates on a 3-year rotating cycle, each group of programs assessed every 3 years. It may be a good source of information and data on the problems and challenges involved, so that those interested in higher education assessment would certainly profit from taking a closer look at this interesting Brazilian experience.


WHAT THE SYSTEM MEASURES
The National System of Higher Education Assessment has three main components: institutional, program, and student-proficiency assessments.The National Student Performance Exam (ENADE), taken by both first-and last-year students, assesses student proficiency.Institutional and program assessments are based on data collected by the Ministry of Education and on self-evaluations conducted by the institutions.The results from the three components are combined into a final score for each undergraduate program: 40 percent from the National Student Performance Exam score for graduating students; 30 percent from a value-added index, measured using the scores of first-and last-year students; and 30 percent from program and institutional assessments.The exam's average score for the group of last-year students and the final score are reported for each participating program.A reviewing committee visits institutions with final scores below a certain level and produces a report on their findings.

ARE THE RESULTS VALID?
The Ministry of Education has issued a few technical reports explaining the methodology employed by the National System of Higher Education Assessment, but so far, no validity analysis of its results has been developed.Actually, some points have been raised indicating that the system may require some modifications before it could be considered valid for the intended uses.
First, the scoring methodology is uniformly applied to all areas and forms of programs and to all types of institutions.Along with a similar treatment of different programs-like teacher training and engineering-the same principles are also used to assess all institutions.Second, it has been argued that the 30 items used to assess specific area proficiency do not adequately cover the relevant knowledge developed in a degree program.The issue involves content validity, regarding the use of the exam, in the present format, to assess proficiency in any full undergraduate program.A third validity-related concern occurs when groups of students boycott the exam.Even though students' scores are not registered in their school records, there is still resistance from student unions against the system.While students need to take the exam, they may leave it blank, distorting the results and reducing the credibility of the system.Another possible limitation of the national assessment system originates in the three-year cycle used to test the proficiency of students enrolled in a given program.For some programs, in particular those with a small group of graduates, it is likely too long, needing too many iterations to produce results that could be considered meaningful regarding the use of the scores for assessment purposes.

USING (OR MISUSING) THE RESULTS
Although the previous section raises significant issues regarding its validity, the National System of Higher Education Assessment serves the important mission of providing criteria for accrediting degree programs and institutions.This system has had the greatest impact on the rapidly expanding for-profit sector, which is responsible for about 58 percent of the almost 6 million undergraduate students enrolled in higher education institutions in Brazil (based on 2009 data).National media reports about programs on probation, with admission spots cut, or that have had accreditation revoked by the Ministry of Education are becoming more frequent and attract intense public attention.The reports are also used as criteria for distribution of financial aid to students attending private institutions: those institutions with scores below a certain level are denied access funds and benefits from the various federal student-aid programs.As expected, institutions use the published scores and rankings for marketing goals, in particular those in the for-profit sector.

MOVING FORWARD
In a welcome move, the Ministry of Education has recently appointed a permanent specialist reviewing committee to analyze the National System of Higher Education Assessment and make recommendations based on its findings.It is hoped that the committee will help foster a much-needed debate on its aspects and how the results are used.Institutions and specialists should also play a more active role in discussing the whole system.It is expected that some changes will be announced for the 2011 edition of the exam.Alternatives include an increase in the number of items in the specific area section of the exam and a new format for the exam taken by first-year students.Many universities have asked for students' individual grades to be made part of their school records, to make boycotting less attractive, but it is unclear whether such a move is under consideration.
Despite the issues and limitations mentioned above, the National System of Higher Education Assessment has already established itself as a relevant part of the Brazilian higher education scene, playing a central role in accreditation processes and in other federal programs.It may be a good source of information and data on how such a system works and on the problems and challenges involved-so that people interested in higher education assessment could profit from a detailed analysis of the Brazilian experience.