Measuring outcomes paper by Edwin Herr

These ideas were presented by Edwin Herr at the OECD sponsored Toronto conference in October 2003
Edwin L. Herr The Pennsylvania State University

1. What outcomes should policy-makers expect from career guidance services?

In an overly simplistic way, one can argue that policy-makers expect two things of career guidance services. The first is for career guidance services to be instrumental in achieving selected national or subnational goals that relate to goals specified in a particular policy initiative or piece of legislation. From this sociopolitical perspective, the goals attributed to career guidance services can take many forms: e.g. reducing unemployment; assisting persons to move from welfare or the dole to active employment; rehabilitating persons with disabilities or social problems and returning them to the work force; facilitating individual transitions into and induction to the work force, at points of mid-career change, and at the time of retirement; building human capital; helping persons make choices that affect commitment to work values, purpose, and productivity; increasing job satisfaction or worker fit with available job options.

The second expectation of policy-makers is that career guidance services will be cost-effective, returning more in economic and social benefits than they cost. Three years ago, I had the opportunity to review the national policies of 14 nations regarding the provision of career development interventions. In most of these national policies there were statements discussing accountability expectations of career guidance providers, the goals assigned to career guidance, government investment in career guidance services, and presumptions that career guidance services would be cost effective. However, the result seems to be that national policies tend to focus on maximizing the cost effectiveness of career guidance services rather than demonstrating the cost-effectiveness of career services. These, national policies presume benefits to the populations targeted by the legislation or policy but there are few reported efforts to empirically validate these assumptions. On the other hand, one also can argue that there is a large and growing body of research on career guidance practices that frequently parallels but does not intersect with the decisions of policy-makers about what practices or goals to support, in part because these research results have not been systematically or effectively relayed to policy-makers.

To return to the main thrust of this meta element of Theme 5, it is difficult to be definitive about what outcomes policymakers should expect from career guidance services because of the differences in purposes for which policies are written, and the differences in the definitions, language, and meaning of career guidance services within and among nations. For example, in any specific case of policy, is career guidance presumed to be the same as career counseling or career education or are they each different? Do career guidance services subsume career counseling as one of the processes it includes or are career guidance and career counseling separate sets of interventions? Are these terms different from or does career guidance subsume such processes as decision-making courses, work shadowing, work experience, career development assessment, job skills training, computer assisted career guidance, use of the internet for purposes of retrieving career information or self-exploration? Is career guidance to be an independent or discrete intervention or part of a program of interventions (e.g. skill training, financial aid, career guidance)? Even more fundamental to policy outcomes is the meaning of career. Is the term really used to mean occupation, job, the expression of work activity across the lifespan or something else? The answers to each of these questions will have much to do with the specific goals that may be assigned to career guidance by a specific policy.

There are at least five reference points for identifying outcomes that can be used to interpret to policy-makers what they can expect from career guidance services. They include potential outcomes identified (1) in career development theory, (2) the content of career assessment instruments, (3) dependent variables that have been used in research studies of the effects of career guidance services, (4) goal statements specified in descriptions of particular models of career guidance, and (5) what clients say they need from career guidance services. To illustrate the point, examples of goals achieved by various configurations of career guidance services, as affirmed by research studies, include improved school involvement, personal and interpersonal work skills, preparation for careers, career planning skills, career awareness and exploration (Campbell et al, 1983), reduced job stress, increasing job satisfaction, improved person-job fit, reduced length of unemployment, improvement in interpersonal relationships and anger management, improvement in making a career choice, acquisition of skills in decision-making, enhanced general adjustment (Herr & Cramer, 1996), motivational and attitudinal change, learning outcomes (e.g. skills, knowledge, and attitudes), participation in learning, student retention and achievement, job search and re-employment, employment, wider social benefits of guidance and its contribution to the economy (Hughes, Bosley, Bowes & Bysshe, 2002). As suggested, here, the outcomes obtained or expected to be derived from career interventions are diverse, changing, and overlapping. Those identified are much more focused on short-term micro-economic issues than long term macro-economic issues. Some are affective, others are cognitive, still others are behavioral. They vary with regard to whether they assess knowledge of or change in self-characteristics, (e.g. interests, values), occupational or educational options, career certainty, career beliefs, indecision, or the decision-making process. The foci of concern are often attitudes toward choice, independence in choice, self-efficacy beliefs, and personal adjustment. Sometimes the outcomes measure focuses upon degree of anxiety, decision-making style, quality of job-search skills, achievement, persistence, work adjustment, academic performance, or other behavior.

These examples of outcomes, used as dependent variables of career guidance services, and which single or multiple research studies support, are complex, associated unequally across population groups, use a variety of measurement forms, and are the results of different interventions and levels of intensity of treatment. Thus, at the moment, the boundaries of the outcomes are not rigid or clearly catalogued by the quality of evidence supporting each type of outcome. Nevertheless, each type of outcome identified here has been found to be affected by career guidance services or career interventions subsumed by such a term.

A major issue, then, is which sets of outcomes, those dealing with attitudes, knowledge, behavior or subsets of meta dimensions subsumed in each of these categories, or those outcomes most specific to definitions of career, occupations, or work adjustment do practitioners or theorists/researchers educate policy-makers to expect? Or, on the other end of that issue, how does the provider match specific outcomes, available evidence about which career guidance services produce specific outcomes, and the goal statements espoused by the policy-maker?

It seems fair to suggest, that given the need to collaborate with or educate policy-makers about what they should expect as outcomes of career guidance services, there are as yet few models of how to present such a diversity of possible outcomes to policy-makers. Certainly, we in the United States have to date not focused very directly on such models. Clearly, the work of Kileen and his colleagues (1992), Hughes, Bosley, Bowes, and Bysshe (2002), and Mayston (2002), among others in the U.K., are superior to any related forms of conceptualization of which I am aware in the U. S.

For the purposes of discussion about the theme goals here let me mention briefly the brilliant framework of David Mayston (2002) for assessing the benefits of career guidance and its potential use with policy-makers. In very skeletal terms, the framework of Mayston, of the Centre for Performance Evaluation and Resource Management, University of York, is based upon the concept of value added to the value of individual human capital achieved by career guidance. In essence, his emphasis in human capital is on the net present value of changes in future net income the individual will experience as a result of career guidance. Embedded in the concept of value added are important indicators of performance and organizational success, related to the differences in characteristics of different client groups which career providers may work with, and the extent of improvement in the value of human capital attained after “netting off” the prior attainments achieved in their careers. Included in his framework is the computation of social value added by career guidance in reducing social exclusion, and the potential benefits of different levels of quality of career guidance services under varying conditions of risk and uncertainty.

In general, according to Mayston, higher quality career guidance is characterized by more accurate assessments of the chances of success of individual advisees in achieving different possible levels of performance in additional training and education, and in helping to equip individual advisees with the motivation and preparation to have a high probability of success in the goals they choose following exposure to career guidance. Mayston also introduces the concept of perfect guidance as a benchmark for assessing the quality of career guidance. This is defined in terms of the career guidance interview, and any related tests, being able to fully assess the abilities, skills, and attributes of the individual, so that as a result of this assessment the individual can be accurately advised as to whether or not they will succeed in a career move; perfect career guidance would also accurately assess the future state of the economy so as to be able to perfectly predict the future net income the individual advisee will receive if they do make a successful career move.

Mayston also introduces the concept of Type I and Type II errors which individuals will make in their career choices in the absence of such perfect career guidance. A Type I error occurs if the individual rejects a career move even though it would have been beneficial for the individual; implicit in such a choice is less than accurate career information, imperfect assessment of the suitability of a given individual for a particular career move, etc. A Type II error occurs if an individual decides to make a career move for which they do not have the capability to succeed. “In this case, the individual incurs the investment costs associated with the career move and lack of a positive return on the investment. The value of the benefits generated by any given level of quality of career guidance is measured by the reduction in the frequency and costs of Type I and Type II errors” compared to those which they would make in the absence of career guidance. The goal of perfect guidance, as the benchmark for the highest quality of career guidance, is to reduce Type I and Type II errors down to zero, but the likelihood is that career guidance services will be less then perfect leading to an index of quality career guidance which a career guidance provider offers given the proportion which the provider actually achieves related to that which would be obtained by perfect career guidance. Mayston then suggests “when we take into account the costs of providing different levels of quality of career guidance, we can derive an optimal quality of guidance, against which different providers can be judged” (p. 6). Such an approach focuses on reducing the degree of mismatch between individuals and labor market options; it can identify associated information requirements related to the choices to be made; and it could specify metathemes related to awareness, exploration, weighing alternatives, examining one’s self-efficacy and values related to potential choices (e.g. acquiring work skills, etc.)

While I have not done justice to Mayston’s model, I cite it as a way to frame and identify goals of career guidance which are related to employment and re-employment and which would be easy to interpret to policy-makers. If the purpose for career guidance policy were to be work adjustment or reducing job stress or chemical dependency, the model would need to be modified in the elements included.

Another example of a way to bound career guidance outcomes is also found in human capital theory but with a significantly different perspective. In this case we assume that the “worker is an investor” who owns his or her human capital (Davenport, 1999). Human capital here is comprised of how one chooses to commit or express one’s ability (knowledge, skill, talent), behavior (how we perform in contributing to a task), effort (the conscious application of our mental and physical resources to accomplish particular tasks, our work ethic), time (how much time we are willing to invest in a particular job or educational task), and what the worker expects as a return on investment (e.g., intrinsic job fulfillment, opportunity for growth, recognition for accomplishments, financial rewards)? Obviously, each of these elements of human capital possessed by an individual is worthy of analysis and discussion in career planning. More importantly, each of these elements of human capital can be translated into outcomes to be sought in career guidance. In such a paradigm, one can draw a parallel between the worker as an investor and the worker as a career manager. In the latter case, the task is to apply the human capital in those situations where the return on investment is expected to yield valued outcomes. To remain flexible, however, the career manager also must be constantly improving and adding to his or her supply of human capital to make it more attractive to changing organizations and employers.

2. In assessing outcomes for policy purposes, what balance should be struck between different types of outcomes (for example: individual learning outcomes, individual decision-making behavior, economic and labor market outcomes, outcome for education systems; and social and equity outcomes)?

There are a variety of ways one can conceptualize the balances among these outcomes. For example, Watts, Dartrois & Plant (1988) have argued quite persuasively in several documents that “educational and vocational guidance services have a key role in any advanced society both in fostering efficiency in the allocation and use of human resources and in fostering social equity in access to educational and vocational opportunities” (p. 1). In one sense, outcomes of efficiency and social equity are outcomes that are measurable, philosophically succinct and positive, and transcendent. They are anchors of virtually any rationale for implementing career guidance services. One can make a similar case for economic and labor market outcomes. Throughout the history of the evolution of career guidance services, particularly in the twentieth century, has been the realization that for whatever other reasons public policy might support career guidance services, these services have been seen as instruments to achieve economic and labor market outcomes and to do so efficiently and with respect for social equity in the choice of, preparation for, and successful induction, adjustment and advancement in the labor market. The undergirding assumption is that to the degree such outcomes occur, positive economic outcomes will accrue to the individual.

There is also the issue of the interaction among individual learning outcomes, individual decision-making behavior and outcomes for education systems, economic and labor market outcomes, and social and equity outcomes. One can argue that the last three sets of dependent variables – education, economic and labor market, and social and equity outcomes – occur in whatever form in response to individual learning and decision-making behavior. Theorists increasingly attribute individual career behavior and decision-making to learning. Several brief quotes address the point. Donald Super contended that the cement that held all of the elements of his career theory together was learning theory. As an example, he stated that “Interactive experiential learning, self-concept, and occupations-concept formation takes place through the interaction of the individual and the environment” (Super, 1990, p. 204). Krumboltz and his associates have consistently placed learning in a central position vis a v_is decision-making. When speaking about social learning Mitchell and Krumboltz (1984) state “It assumes that the individual personalities and behavioral repertoires that persons possess arise primarily from their unique learning experience rather than from innate developmental or psychic processes. These learning experiences consist of contact with and cognitive analysis of positively and negatively reinforcing events” (p. 235). Further, Krumboltz (1994) states that “People learn their preferences by interacting with their environment in a long and complex series of experiences” (p. 17). Finally, Jepsen (1989), in his model of adolescent career development includes learning related to mastering decision-making processes as well as finding satisfying content: “It involves effective ways to decide in addition to finding actions that lead to pleasing outcomes…” (p. 78).

Within such contexts, then, one can argue that outcomes that include those related to individual learning and decision-making are essential elements of career development and important targets of career guidance services. One can also argue that outcomes of significance to the learning and decision-making processes are those related to educational options, attitudes, interests, values, work skills, and preferred economic and labor market outcomes mediated by efficiency and social equity concerns.

3. What strategies are needed to improve the collection of data that can be used to evaluate outcomes? Who should data be gathered from? What types of data should be gathered? How should it be gathered?

From a strategic standpoint, a fundamental need is more extensive and direct collaboration of theory-building, research, and outcomes assessment by teams of policy-makers, theorist/researchers and practitioners. Policy-makers and career theorists must be in further dialogue with practitioners about the problems they experience in the settings and the populations they serve, what outcomes they are pursuing, why, and how? Theorists/researchers and practitioners need to discuss how outcomes might differ across populations by age, need, and other characteristics. Theorists/researchers need to understand more fully, probably through naturalistic, qualitative studies, the behaviors of practitioners, their problems, and the contexts related to the career concerns of specific populations. Unless practitioners are encouraged to feel that they are stakeholders in obtaining relevant and accurate data to evaluate the outcomes of career guidance services and that knowing about such outcomes will be beneficial to their effectiveness, it may be impossible to get complete and accurate data relevant to the measurement of outcomes. In such cases, theorists/researchers and practitioners need to work together to design evaluative studies which are useful, relevant, and doable.

Data to be collected depends on the outcomes to be sought. Typically, the data would be gathered from the clients who have been exposed to career guidance services. In some instance, however, data may need to be collected from employers, teachers, administrators, or other stakeholders.

The types of data again depend upon the questions being asked and the outcomes to be assessed. Depending upon the specific outcomes, it is possible that standardized career assessments will be the most effective proxies for the knowledge, information or behavior being studied. In addition to the use of formal instruments that assess career attitudes, competencies, knowledge, or behavior, case studies, critical incidents and behavioral observations by employers, teachers and others may be useful. So may focus groups that make use of “flagship” students or employees as persons whose knowledge and skills can be evaluated frequently and in depth. There also is a need for qualitative studies that explore the barriers, obstacles, and contextual constraints that inhibit the effectiveness of practitioners or counselees in achieving the career guidance outcomes at issue.

Data should be gathered in as natural manner as possible but cognizant of sound evaluation/research practices. Evaluation processes should be put in place when new career guidance programs or processes are initiated to identify baseline knowledge, attitudes, information, and behaviors in populations being served so that change in outcomes can be identified. If theorists/researchers and practitioners are cooperating in the evaluation effort, teams of evaluators might be able to gather data without disrupting the on-going career guidance program, at random intervals, and using different forms of evaluation.

In any case, as Hughes, Bosley, Bowes and Bysscke have observed, “there is a need to identify levels of evidence required to inform public policy debate in relation to levels of future investment for “career guidance (p. 5).

Their recommendations for the following elements are quite sound. I have abridged and modified them slightly:

A coordinated strategy to (a) consider relative effectiveness, cost effectiveness, and cost-benefit in a degree of detail which realistically captures the diversity of guidance; and (b) to examine the associations of these kinds, which can be produced and demonstrated with relative ease, and their subsequent educational and career benefits” (p. 5).
Short-term evaluation studies….
A robust research programme….
A systematic review of discrete and integrated interventions to take advantage of the current diversity in the provision of career guidance across nations.
A national research database is required to capture main findings from research in a systematic way and to help [inform policy about evidence-based practices and possible outcomes from career guidance to be achieved in relation to different policy purposes.]
I would agree with these perspectives and, indeed, with those of Mayston that within his frame of reference, a well-designed database could generate performance indicators, identify best practice providers of career guidance, standards and targets for service delivery, and the “detailed appraisal of the case for different levels of financial support for individual career guidance providers, and for the career guidance system as a whole.”

In addition, I have argued elsewhere that “The creation and implementation of long-term and comprehensive policy in career guidance requires a bi-partisan or, ideally, a depoliticised approach at the national level which can identify and support a core of career guidance activities that should be available for all citizens. Such policy support needs to provide for the possibility of sufficient flexibility so that guidance services can accommodate the implications of an occupational structure in rapid transformation, the changing demographic structure (e.g. the rise in immigrant populations, persons of ethnic diversity and women) of the workforce, cross-national mobility in a global economy, and organizational and educational shifts related to the implementation of advanced technology.

A further recommendation is for a life-cycle approach to career guidance policy. In many instances, public policy tends to segment problems artificially by age group or subject matter. As recommended by the research and Policy Committee of the Committee for Economic Development, in the United States, “a life cycle approach to public policy in career guidance would integrate such activities across the life cycle in ways which comprehensively reform, and bring into connection from one life stage to another, policies which are now piecemeal and fragmented. Such an approach to public policy in career guidance would articulate career guidance services intergenerationally, based upon what is known about the salient needs for career maturity and adaptability from one life-stage to another. In essence, public policy would address the unique career guidance needs of children and youth as they explore and anticipate work, develop general employability skills, and engage in career planning; provide programs that facilitate the transition from school to work; facilitate worker adjustment, retraining, and career change; and help older workers to plan for retirement or reduced labor-force involvement as they age” (Herr, 1991, p. 280). There is a knowledge base about each career development life-stage and the transition from one to the other which should inform a life-cycle approach to career guidance and its needs to be responsive both to cultural diversity in workers and to rapid change in the occupational structure. Such a frame of reference could be a useful matrix by which to interpret empirically derived outcomes of career guidance to policy-makers

4. Do individual practitioners, service managers and policy makers have different interests in the outcomes of career guidance? Do they need different types of data? Can all of their needs be accommodated?

Different persons are likely to have different interests in the outcomes of career guidance. To again be overly simplistic, policy-makers are likely to be concerned with summative evaluations and cost benefit outcomes. Practitioners, service managers, theorists/researchers are more likely to be interested in refined formative studies. Summative evaluations ask, did this whole career guidance program, in an aggregate sense, achieve the outcomes expected of it as compared with earlier base-line data about the consumers’ knowledge, information, attitudes, etc. Formative evaluation is more sophisticated in the sense that the question is not simply whether the total program of career guidance services produced the expected outcomes but rather which of the elements of the career guidance services contributed most effectively to the outcomes achieved? Individual career counseling? Group work? Assessment? Topical Workshops? Computer-Assisted Career Guidance System? Work shadowing? etc.

In either formative or summative evaluation studies, there also is the question of cost-benefit analyses. In the summative approach, did the total outcomes achieved (e.g. reduced drop-outs from school or training programs, successful job placement, reduced observations, more effective job search or social skills) represent economic or social benefits that exceeded the total cost of the program. In formative evaluation studies, pertinent questions include what were the costs of practitioner time and other resources related to individual elements of career guidance services? Which of these services was most effective in achieving valued outcomes and which services were least effective? Collecting such data may require self-reports by practitioners about how much time they spend in providing various services; analyses of costs of resources (e.g. assessment materials, directories, computer assisted systems); self-reports by clients about which services were most useful to them as well as follow-up data about increases in salary compared to previous jobs, satisfaction with job, etc.

Individual practitioners, consumers or clients, service managers, policy makers, educators, employers may each view the economic and social benefits of career guidance services differently. From a cost/benefit perspective, benefits may accrue to individuals who are the direct consumers of such services/interventions or, less directly, to educational institutions, employers, governments, or society at large. Economic benefits to clients may be measured in their ability to secure jobs with improved pay, in shortened periods of unemployment, in obtaining greater congruence between personal interests and abilities in a job chosen and in the experience of extended tenure in that job. For educational institutions, benefits may be seen in increased retention of students, thereby preserving the government aid paid per student rather than losing such aid if a student drops out; for employers, it may be reflected in better employee fit, productivity, extended tenure, and flexibility, thereby reducing costs of recruitment and replacement of workers; for government, it may be reflected in reducing the amounts of money required to pay for unemployment benefits or to persons with disabilities thereby increasing the acquisition of taxes from persons helped to return to or to enter the labor market; for society at large, a purposeful, productive, teachable work force may enhance the competitive edge of the nation in competing for market share in a global economy, in being innovative, and in maximizing high skill/high pay jobs in the occupational structure. Each of these outcomes can be measured in economic benefits in direct or in more complex ways framed in terms of short-, medium-, and long-term benefits.

One can argue that the needs of all of the stakeholders in career guidance services can be accommodated if evaluative studies of outcomes are planned and executed to yield such outcomes. One also can note that in policy decisions related to cost-benefit analyses, it is not always the most powerful intervention that is supported. For example, studies of career interventions frequently suggest that individual career counseling is the most powerful for hour of intervention, but it is also more expensive than many other interventions that are group-oriented. In such a circumstance, if individual career counseling and group career counseling were both under consideration for support and the outcomes they each produced favored individual counseling. but group counseling was also found useful and productive of many of the same outcomes as career counseling, it is quite possible that an administrator or policy-maker would decide to support group counseling as a primary treatment rather than individual counseling. Such a decision might hinge on the view that because one can simultaneously provide career counseling to multiple clients (perhaps 6 to 8) in groups much less expensively per hour than is true of individual counseling, group counseling would be the most economical way to reach the largest number of clients. In such cases, the provision of individual counseling would likely be much more restricted to only the neediest of clients rather than to every client.

5. Should evidence on outcomes be linked to the allocation of resources and to priorities for service delivery? If so, how?

Given the competition for limited resources among social services of all types, it seems increasingly difficult to believe that policy-makers in the future are not going to link resource allocations to priorities for service delivery and to evidence on outcomes. Increasingly, in the United States more legislation is holding schools and agencies accountable for documenting the outcomes they achieve and the language of policy is rapidly being reworded by concepts from measurement, testing, assessment and evaluation. These concepts have become standard vocabulary for many practitioners and administrations They include such terms as data-driven, standards, performance indicators, evaluation, continuous quality improvement, total quality management, management by objectives, PERT, PPBS, accountability, benchmarking, strategic initiatives and strategic actions, competency, certification, accreditation, licensure, high stakes testing, results-based, outcome based, evidence-based practice, cost centers, activity-based accounting. Every one of these terms, in whatever setting it is used, is based on some form of quantification, testing, assessment, measurement or evaluation. These terms are increasingly pervasive in policy and in legislation. They appear in varying but increasing ways in vocational education legislation, in the Joint Training Partnership Act, in the Workforce Investment Act and in other legislation. And, such perspectives are likely to intensify in their needs for documentation and linkage to the allocation of resources.

To illustrate the trends in such federal legislation in the United States, one of the most publicized recent legislative initiatives is the 2002 reauthorization of the Elementary and Secondary Education Act (ESEA) with the theme “No Child Left Behind.” This legislation either mandates or has goals that include the following: (1) comprehensive school reform, of which career guidance services might be an element; (2) evidence-based practice and scientifically based research; (3) the provision and existence of “effective” teachers in every classroom in America; (4) professional development and documented accountability; and (5) standards-based assessment and demonstrated outcomes for all students. Schools must administer assessments to students in a variety of subject matter once during grades 3-5, 6-9, 10-12. Schools in which students do not yield satisfactory scores are penalized in some states by loss of funds and other sanctions, including being listed in the press as under-performing schools, schools needing improvement, or successful schools. In some states, schools meeting specific criteria receive bonuses in allocation of resources or other accolades. There also are frequently schools that require exit testing for students who must obtain a particular score in order to graduate from high school or to move from grade to grade. Such legislation increases the need for definitional clarity in what seem like benign terms, “proficiency,” “effectiveness,” etc. We could spend hours talking about the problems and issues associated with such governmental legislation. That is not our purpose here. Thus, I use this example to illustrate that, in the case of the American government, we are seeing rapid changes in the ways funds are allocated through legislation and their linkage to outcomes. Indeed, legislative initiatives in many government agencies have moved from philosophical reasons for government support and general guidelines for recipients of funding to follow, to increasing support for empirical demonstrations that the processes supported are

working. As such, governmental examples of linkage of resources allocation to priorities for service delivery are likely to become much more comprehensive as a strategy across legislative foci.

6. How can the costs of career guidance be assessed in relation to its benefits? How should this data be used to influence policy and resource allocation?

I have already spoken to cost-benefit analysis a bit earlier. In this case, a standard approach is to assume that the costs of career guidance can be assessed by tracking personnel costs (hourly for example) committed to a particular intervention (e.g. individual career counseling, group guidance, assessment, etc.) as well as tracking resource costs related to functions, as well as facility costs, professional training and other pertinent costs. From a formative standpoint the outcomes of these interventions need to be compared in terms of their impact on particular career guidance goals. These outcomes may be assessed by standardized instruments or by other methods. In one sense, you are really asking a step-wise regression question, how much of the variance in the change of career knowledge, attitudes, behaviors is explained by each intervention and which interventions make no impact on such variance. There are obviously many other ways to address cost-benefit analyses. An overarching point is that such analyses are most useful if career guidance interventions are independent of other types of interventions that could confound the analyses.

Depending upon how sophisticated the evaluation program is that links the outcomes of career guidance to cost-benefit effects, one can use randomization of clients to treatment groups, experimental and control groups to evaluate intervention impacts and the costs to achieve them. One can use hierarchical linear modeling (HLM), for example, to simultaneously assess the extent to which several career providers are implementing a common framework of program elements and the benefits that accrue to different groups of clients either on a short-term or longer basis. One could also use various quantitative analyses to rank providers along a continuum of implementation for selected core activities and the outcomes they achieve. Data based on questionnaires, self reports, and structured interviews with selected client or counselor populations could yield important information about practitioners’ time and efforts in using various program elements. Qualitative analyses could more fully describe individualized strategies employed to obtain certain outcomes and identify the critical features of those career guidance strategies that effectively promote enhanced outcomes. These data can then be effectively merged with available cost data to determine cost-benefit ratios. From a research perspective, you can reduce the model of career guidance services provided, to those that are most powerful in their impact on selected career outcomes and not include those services which do not make an impact on selected outcomes. By linking the costs of individual career guidance services to the comparative data on their impact, theoretically you can also find those services that are both the most powerful and the most economical. Such evaluative data relating to costs, impacts, outcomes or benefits can be summarized as is done with other evaluation information and provided to policy makers to both document the impact of a particular program and its costs.

Again, I have given something of an idealized answer here because of the time available to speak. It is important to note that some forms of cost-benefit analysis are much easier to do then others and that the use of randomization of subjects or experimental and control groups in research related to policy formulation and to resource allocations is quite rare.

Further, assessing the impact of career guidance techniques on specific economic indicators (e.g. reduced unemployment, increases in wages, employer ratings of clients, work attitudes and skills) on a short-term basis is easier then assessing social benefits or medium and long-term impact of career guidance services. Social benefits (e.g. reductions in floundering or criminal behavior in the future) tend to be softer, require more assumptions related to the impact of career guidance than do economic factors, and are typically longer term in assessing the presence of such outcomes.

In general, it seems fairly clear that cost-benefit data would be useful in helping policy makers decide on policies related to the importance and the value of career guidance. How this information then relates to resources allocation is less clear except in some return on investment context. For example, policy-makers would find it useful to know that for every dollar they invest in career guidance with a particular population (e.g. unemployed), we are likely to experience a return of $3 in taxes or in improved productivity, etc. That sort of information, which has been used rather extensively in rehabilitation counseling and in addictions counseling is often quite persuasive to policy-makers, but not systematically available across the populations and settings in which career guidance is provided.

References

Campbell et al 1983
Campbell, R., Connell, J. B., Boyle, K. K. & Bhaerman, R. (1983). Enhancing career development. Recommendations for action. Columbus, OH: The National Center for Research in Vocational Education, The Ohio State University.

Davenport 1999
Davenport, T. O. (1999). Human capital. What it is and why people invest in it. San Francisco, CA: Jossey Bass.

Herr et al 1996
Herr, E. L., & Cramer, S. H. (1996). Career guidance and counseling through the lifespan. Systematic approaches. (5th Ed.) New York: Harper Collins.

Herr 1991
Herr, E. L. (1991). Multiple agendas in a changing society: Policy challenges confronting career guidance in the U.S.A. British Journal of Guidance and Counselling, 19, 267-282.

Jepsen 1989
Jepsen, D. A. (1989). Adolescent career decision processes as coping responses for the social environment. In R. Hanson (Ed.). Career development. Preparing for the 21st century. (Chapter 6). Knoxville, TN: The University of Tennessee. Department of Technological and Adult Education.

Hughes et al 2002
Hughes, D., Bosley, S., Bowes, L. and Bysshe, S. (2002). The economic benefits of guidance. Derby, England: Centre for Guidance Studies, University of Derby.

Kileen et al 1992
Kileen, J., White, M., and Watts, A. G. (1992). The economic view of career guidance. London, England: The Policy Studies Institute.

Krumboltz 1994
Krumboltz, J. D. (1994). Improving career development theory from a social learning perspective. In M. L. Savickas & R. W. Lent (Eds.). Convergence in career development theories. Implications for science and practice. (pp. 9-31) Palo Alto, CA: CPP Books.

Mayston 2002
Mayston, D. (2002). Evaluating the benefits of career guidance. Derby, England: Centre for Guidance Services, University of Derby.

Mitchell et al 1984
Mitchell, L. K., and Krumboltz, J. D. (1984). Social learning approach to career decision making: Krumboltz’s theory. In D. Brown & L. Brooks (Eds.). Career choice and development. San Francisco, CA: Jossey-Bass.

Research and Policy Committee 1990
Research and Policy Committee of the Committee for Economic Development (1990). An America that works: The life cycle approach to a competitive work force. New York: Committee for Economic Development.

Watts 1988
Watts, A. G., Dartrois, C. & Plant, D. (1988). Educational and vocational guidance services for the 14-25 age group in the European community. Sittard, Netherlands: Commission of the European Community.