Here we continue our 2017-2018 blog theme focusing on legal education.  Our guest blogger John Bliss recently completed his PhD dissertation research on U.S. legal education and professional identity.  He is currently a Resident Fellow at the Harvard Law School Center on the Legal Profession.  In this post, he returns to a major finding of the Carnegie Report on legal education — that U.S. law schools are not giving their students enough training around professional identity and purpose.  Bliss points out the way that empirical research could help to shed light on this still missing component of U.S. legal education — and reports on two exciting new initiatives to reform legal training.

* * *

The Law School Laboratory: A Call for Developing the “Third Apprenticeship” Through Experimental Research

It has now been a decade since the Carnegie Foundation illuminated a troubling omission in U.S. legal education: a neglected “third apprenticeship” in “professional identity and purpose.”[1] While the standard J.D. curriculum focuses on learning to think like a lawyer (the cognitive apprenticeship) and increasingly offers opportunities to acquire practice skills (the skills apprenticeship), the neglected third apprenticeship laid out in the 2007 Carnegie Report deals with helping students find a sense of personal meaning and public contribution in their roles as lawyers. Legal education has been subjected to a steady stream of faultfinding for over a century, including the recent feeding frenzy of criticism over rising educational debt. But the third apprenticeship detailed in the Carnegie Report merits special attention due to a growing and diverse body of empirical support. In this post, I argue that we have very successfully taken the first two steps in addressing professional identity and purpose—empirical researchers have thoroughly diagnosed the issue and some law schools have innovated in response. I argue that our next step should be to build on the current momentum in both the curricular innovation movement in law schools and the new and expanding networks of legal education researchers to begin to design experimental studies assessing how interventions affect different aspects of professional identity and purpose. Any effort to substantially revise the remarkably enduring U.S. law curriculum and pedagogy needs to present a highly persuasive body of evidence. An ambitious experimental research agenda (developing what I call the “law school laboratory”) would greatly bolster the case for larger reforms. In the next decade, this new wave of research will allow us to begin to systematically fulfill the promise of the third apprenticeship.

What is the problem?

While U.S. legal education strives to produce ethical, thriving, public-regarding professionals, a long line of empirical research has suggested that during law school many students experience a disconnect between the lawyer role and their personal identities and values. This disconnect is evident in their shifts from intrinsic to extrinsic work motivation,[2] from social justice as a vocabulary of motive to zealous advocacy for all clients,[3] from justice-oriented consciousness to game-oriented consciousness,[4] from public-interest to law-firm career aspirations,[5] and from a pursuit of civic impact to an amoral, apolitical, and decontextualized legal epistemology.[6] These effects can be particularly severe among students who are not economically privileged, white, male, and politically moderate.[7] My recent dissertation suggests that elite-school students who start their legal careers in large firms can experience an often distressing degree of “professional role distancing,” particularly if they began law school with a public-interest job preference.[8] This tension with the lawyer role may contribute to students’ widespread mental health declines during and after law school.[9] These struggles likely result from a combination of causal factors including students’ adoption of client-centered neutral partisan norms of professional responsibility, anxiety over grades and classroom dynamics, debt and fear of debt, the choice architecture of the job market, and the transition to becoming more pragmatically concerned with one’s personal financial future.[10] Developing an apprenticeship in professional identity and purpose would not address all of these factors, but there is a growing consensus among scholars of legal education and curricular innovators that developing the third apprenticeship is a vital step in the right direction. If the legal profession is to make good on its longstanding claim that it offers its members meaningful autonomy, intrinsic work satisfaction (the pursuit of a “calling”), and opportunities to contribute to the public interest, neglecting the third apprenticeship in law school reveals a disjuncture with the profession’s fundamental purpose and legitimating narrative.

What innovations are addressing the problem?

Some curricular initiatives have recently emerged to foster professional identity and purpose among law students, often in explicit response to the Carnegie Report. These efforts range from one-day sessions during orientation to first-year and upper-level courses on the profession. At least 30 law schools now provide required professional development curriculum in the first year.[11] Advocates of developing the third apprenticeship have characterized these innovations as a growing “social movement” within law schools.[12] In the most robust first-year offerings, students gain access to empirical perspectives on lawyers’ careers and conduct informational interviews with practitioners in a variety of practice settings.[13] Other approaches have sought to deeply integrate lessons in professional identity and purpose into the core curriculum through simulation learning, practicums, efforts to bridge clinical and doctrinal instruction, problem-solving courses, and attention to legal ethics within the case method.[14]

Next steps?

Building on the substantial record of research diagnosing the neglected third apprenticeship and the innovative efforts to remedy this omission, our next step should be to empirically investigate how different interventions affect specific elements of professional identity and purpose. This can mean assessing the various impacts of existing curricular efforts or developing new intervention projects (perhaps on a smaller scale) that are designed with testable hypotheses. Under any research design, professional identity and purpose is a highly qualitative outcome that needs to be disaggregated into multiple measures. To meet this challenge, we should draw on the tools developed by empirical researchers who have been studying legal education and other educational settings. As a starting place, we can borrow directly from research instruments used in prior studies—including psychological scales,[15] measures of engagement,[16] and questions about career orientations.[17] For hypotheses dealing with students’ later career satisfaction and public interest contributions, we can borrow from the research protocols of large-scale empirical studies of lawyers and other professionals.[18] But we will also need to be innovative in designing measures that are tailored to hypotheses dealing with students’ experiences of professional role distancing, professional purpose, public-interest values, attitudes about lawyers, approaches to job-path decisions, knowledge about careers, and moral judgment.[19] These measures need to be finely tuned to the goals of curricular interventions and sensitive to the interplay of race, class, gender, culture, nationality, and other identities and variables.

What role for RCTs?

In combination with other qualitative and quantitative designs, these assessments should include randomized control trials (RCTs). While researchers in medicine and a wide range of social sciences have long viewed RCTs as the gold standard of empirical evaluation, the fields of law and education have until recently been peculiarly resistant to RCTs.[20] This resistance is now giving way to a burgeoning RCT movement in both fields, as researchers and policymakers seek higher standards of empirical evidence in assessing educational and legal practices.[21]  Of course, educational RCTs raise a number of implementation challenges. In addition to designing suitable outcome measures, these experiments require determining the proper timing of assessment, running the module consistently across sites, limiting contamination as students share materials with their peers in control groups, and navigating a web of contextual factors associated with different teachers, schools, and the timing of the study (if run over multiple years).[22] Furthermore, efforts to pervasively address professional identity throughout a law school’s culture would not be readily amenable to RCT design as all students at a site would be in the treatment group.[23]

While these challenges are substantial, they should not deter us from conducting RCTs in law schools. Investigators can address issues relating to measures, consistency, and contextual factors as they design a detailed research protocol during pilot studies. As recommended in the experimental educational literature, the protocol should include an operations manual covering all study procedures. Measures should focus on the original hypotheses of the project in addition to examining background factors. Further information about contextual factors would be gained by surveying 1L instructors about their pedagogical approaches.[24] In accordance with best practices for RCT reporting, observations that are unrelated to the original hypothesis would be considered as directions for future research.[25] Others researchers would be encouraged to conduct replication studies.

In some respects, legal education offers experimental researchers great advantages over other educational settings. As Daniel Ho and Mark Kelman point out in their discussion of a natural experiment at Stanford Law School, the relative standardization of 1L curriculum and the stratified block randomization in 1L section assignment can enhance internal validity when sampling by section.[26] For example, at a school with adequate randomization processes that divide the incoming 1L class into four sections, investigators can randomly select two sections to receive the treatment and two to serve as comparison groups. To take the example of an intervention focused on the 1L experience, pretest evaluations could be applied to both treatment and control groups during law school orientation. Posttest evaluations could be applied late in the 1L spring semester, in order to assure a high response rate before sections disband for the summer. If hypotheses include career paths, job satisfaction, and public contributions after graduation, later measures could also be applied.

Are there any third-apprenticeship experiments in the pipeline?

At least two ongoing curricular projects dealing with professional identity and purpose are pursuing experimental design. Victor Quintanilla’s 1L Access to Justice Service Learning Curriculum at Indiana Maurer School of Law aims to help students internalize the civic values and commitments of the legal profession.[27] Quintanilla’s forthcoming analysis assesses these learning outcomes and compares results to students not receiving the curriculum. I follow Quintanilla’s lead in my ongoing collaboration with Jon Hanson of Harvard Law School and Howard Gardner of the Harvard Graduate School of Education. We are producing short educational modules that target professional identity and purpose drawing on research-based pedagogical materials from the Good Work Project[28] and exercises dealing with professional identity, values, and legal career paths. These interactive exercises are paired with cautionary tales from the empirical literature about the combination of factors that lead students to experience detachment from the lawyer role (what I have termed “professional identity drift”).[29] This year we have been implementing these modules with first-year students as we begin to develop outcome measures to potentially run the curriculum as an intervention study. This independent-module approach lacks the holistic advantages of more pervasive reforms of the 1L curriculum, but such modules can be particularly amenable to experimental research as they involve lower costs and can target specific hypotheses and mechanisms. Depending on how the study is designed, one might hypothesize that participants in the module would experience significant measurable reductions in professional role distancing, uninformed decision-making about legal career paths, disconnect between public-interest values and professional role, disengagement with their legal education, or mental health issues relative to students in a control group.

What is gained with an RCT design?

RCTs have a powerful capacity to distinguish the observed impact of an intervention from other causal factors. Educational settings do not make a perfect laboratory, but educational researchers have established a strong record of best practices for RCT design. Armed with rigorous experimental methods, we can begin to parse how different interventions work. This does not necessarily mean shifting to entirely evidence-based practices. But including RCTs in our toolkit of pedagogical self-evaluation and reform is consistent with recommendations from education scholars and the recent ABA call for assessment of learning outcomes in order to “shift the emphasis from what is being taught to what is being learned by the students.”[30] Given the advantage of near-random section assignments at many law schools, the opportunity to run legal education RCTs is particularly compelling. As with experimental work in other contexts, access can be the greatest hurdle, particularly where institutions fear that a benefit would be offered to members of the treatment group that is denied to members of the control group. But this objection commits the very causal fallacy that RCTs are designed to address—we don’t know whether the intervention is beneficial until we compare it to a counterfactual.

Hope for reform?

If rigorous evaluation of interventions shows significant results in promoting professional identity and purpose, this would help us move forward with larger reforms on a more empirically-informed basis. I propose an ongoing and iterative process between research and teaching experimentation, drawing on tools from educational research and other fields. This is arguably an opportune time for this proposal. Amid the ongoing enrollment crisis (and other intersecting crises of legal education),[31] law schools appear to be particularly receptive to innovations. At the same time, there is great momentum in legal education scholarship, including the rise of new collaborative networks of researchers,[32] symposia on legal education research,[33] workshops and conferences on fostering professional identity in law school,[34] and vibrant discussion about the future of legal education in prominent legal scholarship blogs.[35] Running RCTs and other empirical assessments of curricular innovations is not only in the interest of those seeking to improve legal education but also social scientists seeking to understand professional socialization processes. By treating students with curricular innovations and observing what changes, we can deduce more about what they were experiencing in the first place. In other words, I suggest deferring to Dr. House:

Dr. House: We treat it. If she gets better, we know that we’re right.

Dr. Cameron: And if we’re wrong?

Dr. House: We learn something else.[36]

 

John Bliss is a Resident Fellow at the Harvard Law School Center on the Legal Profession.

 

Notes:

[1] William M. Sullivan et al., Educating Lawyers: Preparation for the Profession of Law 1252 (2007).

[2] See Kennon M. Sheldon & Lawrence S. Krieger, Does Legal Education Have Undermining Effects on Law Students? Evaluating Changes in Motivation, Values, and Well‐Being, 22.2 Behav. Sci. & the Law 261 (2004).

[3] See Debra Schleef, Managing Elites: Professional Socialization in Law and Business Schools (2006).

[4] See Robert Granfield, Making Elite Lawyers: Visions of Law at Harvard and Beyond 36-50 (1992).

[5] Students’ “drift” away from public-interest career goals has been the subject of a robust literature over the past several decades. See, e.g., Robert V. Stover, Making It and Breaking It: The Fate of Public Interest Commitment During Law School (1989); John Bliss, From Idealists to Hired Guns? An Empirical Analysis of “Public Interest Drift” in Law School, 51 U.C. Davis L. Rev. (forthcoming 2018) (summarizing the literature on public interest drift and presenting qualitative findings from a longitudinal qualitative study of students’ shifts in career orientations).

[6] See Elizabeth Mertz, The Language of Law School: Learning to “Think Like a Lawyer” 226 (2007).

[7] See Carrie Yang Costello, Professional Identity Crisis: Race, Class, Gender and Success at Professional Schools (2005); Yung-Yi Diana Pan, Incidental Racialization: Performative Assimilation in Law School (2017).

[8] In longitudinal identity mapping exercises, students who had recently decided to work for large firms tended to visually locate the lawyer role on the periphery of their identities, and sometimes completely outside the identity space, explaining that they no longer viewed the lawyer role in the same politically and personally self-expressive terms they anticipated when they began law school. See John Bliss, Divided Selves: Professional Role Distancing Among Law Students and New Lawyers in a Period of Market Crisis, 42 Law & Soc. Inquiry 855 (2017).

[9] See Sheldon & Krieger, supra note 2.

[10] See Neil W. Hamilton, Professional Formation with Emerging Adult Students in the 21-29 Age Group: Engaging Students to Take Ownership of Their Own Professional Development Toward Both Excellence and Meaningful Employment, J. Prof. Law 125 (2015).

[11] See Neil W. Hamilton, The Next Steps of a Formation-of-Student-Professional-Identity Social Movement: Building Bridges among the Three Key Stakeholders—Faculty and Staff, Students, and Legal Employers and Clients, 13 Univ. of St. Thomas L.J. 10 (2017).

[12] Id.

[13] See Ann Southworth, Bryant Garth and Catherine Fisk, Some Realism about Realism in Teaching about the Legal Profession, in Stewart Macaulay et al. eds., The New Legal Realism: Translating Law-and-society for Today’s Legal Practice 74 (2016) (discussing the required first-year course on the legal profession offered at UC Irvine School of Law, which aims to help students “search for a good fit between their aptitudes and values and opportunities in the profession”).

[14] See David I.C. Thomson, Teaching Formation of Professional Identity, 27 Regent Univ. L. Rev. 303 (2014) (summarizing these curricular initiatives and introducing the “Guided Sequence for Formation of Professional Identity,” which provides a framework for teaching professional identity in both the doctrinal and clinical curriculum); Neil W. Hamilton, Verna E. Monson, and Jerome M. Organ, Empirical Evidence That Legal Education Can Foster Student Professionalism/Professional Formation to Become an Effective Lawyer, 10 Univ. St. Thomas L.J. 11 (2012) (describing the pervasive approach taken by the University of St. Thomas School of Law, which infuses lessons in professional identity throughout the law school culture, including career services, structured mentorship, and integration of faith and professional role); Sullivan, supra note 1, at 12 (suggesting that the apprenticeship in professional identity and purpose should be integrated with the cognitive and skills apprenticeships).

[15] See, e.g., Sheldon & Krieger, supra note 2 (drawing on established measures in the empirical tradition of self-determination theory to examine law students’ subjective well-being, motivation, and values); Jennifer Robbennolt’s ongoing study at the University of Illinois College of Law employs a wide range of psychological scales including measures of optimism, grit, uncertainty tolerance, work passion, job satisfaction, and emotional regulation.

[16] See, e.g., Carole Silver, Amy Garver, and Lindsay Watkins, Unpacking the Apprenticeship of Professional Identity and Purpose: Insights from the Law School Survey of Student Engagement, 17 Legal Writing 373 (2011) (discussing the beta questions added to the Law School Survey of Student Engagement in 2008 and 2009 about professional identity and purpose).

[17] See, e.g., Stover, supra note 5 (presenting an extensive survey of students’ career orientations).

[18] See Ronit Dinovitzer, Bryant G. Garth, Richard Sander, Joyce Sterling, and Gita Z. Wilder, After the JD: The First Results from a National Study of Legal Careers (2004).

[19] See Hamilton & Monson, supra note 11 (drawing on the Defining Issues Test and an essay on ethical identity development to show that students at St. Thomas School of Law experience an increase in moral judgment capacity during their legal education. The authors suggest that this finding is particular to St. Thomas by referencing data from other schools in the Law School Survey of Student Engagement).

[20] See James D. Greiner & Andrea Matthews, Randomized Control Trials in the United States Legal Profession, 12 Annual Rev. L. Soc. Sci. 295 (2016); Carole J. Torgerson, Educational Research and Randomised Trials, 36.11 Medical Education 1002 (2002).

[21] In the legal profession, the RCT movement is most visible in Jim Greiner’s Access to Justice Lab at Harvard Law School, which lists a dozen ongoing and recently completed RCTs, many of which serve as the basis of multiple papers (see http://a2jlab.org); in education, the recent turn to experimental research has been spurred by the founding in 2002 of the What Works Clearinghouse (WWC) within the Institute of Education Sciences (the research and assessment arm of the Department of Education), which has from its inception heavily prioritized RCTs. See Matthew C. Makel, & Jonathan A. Plucker, Facts Are More Important Than Novelty: Replication in the Education Sciences, 43.6 Educational Researcher 304, 304 (2014).

[22] See Gail M. Sullivan, Getting Off the “Gold Standard”: Randomized Controlled Trials and Education Research, 3.3 J. Grad. Med. Educ. 285 (2011).

[23] See Hamilton & Monson, supra note 11, at 48-50 (assessing the impact of the pervasive professional identity initiatives at St. Thomas School of Law through a pretest, posttest study design and comparisons to data from national studies and past research).

[24] See id.

[25] See http://www.consort-statement.org/extensions/checklists.

[26] See Daniel E. Ho & Mark G. Kelman, Does Class Size Affect the Gender Gap? A Natural Experiment in Law, 43.2 J. Legal Stud. 291 (2014) (showing that law students who were randomly assigned to smaller class sizes tended to exhibit a smaller gender gap in academic performance than those assigned to larger classes; also showing that later reforms in the grading system and pedagogical approaches had a much greater tendency to reduce and even eliminate the gender gap).

[27] See Victor D. Quintanilla, Human-Centered Civil Justice Design, 121 Penn St. L. Rev. 745, 805 (2017).

[28] See Howard E. Gardner, Mihaly Csikszentmihalyi, and William Damon, Good Work: When Excellence and Ethics Meet (2008).

[29] See Bliss, supra note 5.

[30] See ABA Section of Legal Education and Admissions to the Bar Managing Director’s Guidance Memo, Guidance to Standard 315 (2015).

[31] See Eli Wald, The Contextual Problem of Law Schools, 32 Notre Dame J. L. Ethics & Pub. Pol’y (forthcoming 2018).

[32] See, e.g., the establishment in 2017 of the AALS section, “Empirical Study of Legal Education and the Profession” and a new CRN with the Law and Society Association (pending approval in 2018), “Legal Education,” that I have submitted along with co-organizers David Sandomierski and Swethaa Ballakrishnen.

[33] Including dedicated events held in 2017 at the American Bar Foundation and the Center for Empirical Research on the Legal Profession at UC Irvine School of Law.

[34] Including events held at the Institute for the Advancement of the American Legal System at the University of Denver Sturm College of Law and at the Holloran Center at the University of St. Thomas School of Law.

[35] Including posts on Prawfsblawg, Brian Leiter’s Law School Reports, The Faculty Lounge, Law Deans on Legal Education, and TaxProfBlog.

[36] House: Pilot (Fox television broadcast November 16, 2004).