Opinion ID: 2294246
Heading Depth: 1
Heading Rank: 22

Heading: The PJP Process

Text: The Professional Judgment Panel (PJP) approach is one of four accepted methodologies utilized to create a school funding formula. It brings educators and other individuals with knowledge of the education system together to identify resources necessary to educate students in a hypothetical district(s) to a specified standard. The process is now known as the costing out process. By way of the same, the level of resources needed for students to perform to specified standards, in New Jersey the CCCS, is identified. In or about 2002 the DOE hired APA to conduct adequacy studies and to run the PJP process. There are four generally accepted methodologies to determine adequacy budgets: the successful school district approach; the PJP approach; the evidence based model approach (EBM); and the cost function approach. The DOE chose the PJP process which is the most commonly accepted methodology to determine adequacy budgets. The process was described and summarized in the RCE. D-2. The PJP process was relied upon by the DOE to begin the process of the creation of a new funding formula in New Jersey. APA is a, if not the, leading national consultant in the creation of school funding formulas utilizing the PJP process. APA has been retained by multiple states to utilize the process to develop adequacy budgets. The process is founded on the concept the more eyes the better; that is, the more experienced educators provide their expertise to an accepted construct, the more reliable the developed adequacy budget will be. APA identified the type of panelist required, generally experienced educators, and the DOE thereafter selected the panelists. All data provided to the various panelists was provided by the DOE. In New Jersey, three separate panels were convened. The lists of invited panelists were identified; D-2, appendix 6, 7 and 8 representing invited panelists for rounds 1 through 3, respectively. The PJP process begins with the development of a hypothetical district(s) that reflects the demographics of the school districts in the State; panels are then assembled to determine resources needed in that hypothetical district(s) to meet a specified standard; and the identified resources are costed out. In New Jersey the panelists were not provided with information concerning the then existing funding system, any of the concerns expressed in the various Abbott decisions or the deficiencies found therein, the supplemental program standard determined to be constitutionally required in Abbott V or, for that matter, any specific information as to the Abbott districts. The first panel consisted of seven employees from the DOE. [8] The composition of this panel was challenged by the plaintiffs. Justin Ryan Silverstein (Silverstein), then an associate and now a partner with APA, testified he was satisfied the composition of this panel did not affect its validity. The seven panelists comprising the first panel were given background information (D-2, appendix 4-1), various instructions (D-2, appendix 4-3), the CCCS (D-2, appendix 4-6 to 10), the assessment criteria (D-2, appendix 4-11), graduation requirements and length of school day and year (D-2, appendix 4-12). The panelists were provided a blank slate as to the resources to be deemed necessary. The panelists were told to limit their recommendations to what is necessary to meet the CCCS, not to construct a dream school. D-2 at 2. An APA representative was present and acted as a facilitator after the panel was convened and provided with the above information. The panelists met on January 21st through January 23rd, 2003 representing the longest period in APA's experience for the conduct of the first panel. The first panel was advised the CCCS and its assessment criteria were the standards to be utilized. The panelists were then to create a hypothetical district(s). A hypothetical district(s) was purposefully used with the understanding there would be no attempt to replicate any particular district. Rather, the hypothetical district(s) was to allow educators to review and determine needs premised upon what would be recognizable to the panelists. The panel was to conduct its work premised upon a weighted student formula. That is, the panelists used a base of 1.0 for elementary students without particular needs, and then additional characteristics were given additional weights. The four major characteristics were the size of the district, at-risk, LEP, and special education students. The weights were used to increase the base cost to provide for the resources and additional services necessary for students at higher grade levels and for students with special needs to achieve the CCCS. Six hypothetical districts were created. Concededly, none of the hypothetical districts were premised upon information directly concerning the Abbott districts. Two districts were K-8 and were described as small and very small, and four were K-12 districts described as small, moderate, large and very large. At-risk students were defined as the percentage of low income students who were receiving free lunch, LEP students were the percentage of students in each district with limited English proficiency, and there were three levels of special education students: mild, moderate and severe. Data was assembled by the DOE and provided to the panelists with the understanding geographic location was not to be considered. The panelists then identified resources, which the panelists constructed to include four principal categories: instructional staff, support staff, administrative staff, and other costs. Within the staff categories were various sub-categories of particularized personnel. Other costs included, but without limitation, supplies, equipment, technological resources, etc. Determinations were made as to each category and its sub-categories by way of resources needed for each. See D-2, appendix 9-23 for a complete listing of the various resources addressed. The first panel completed its work. The second panel consisted of forty invited panelists. [9] The panelists composition was purposefully diversified to include educators as well as other individuals including a representative of a teachers' union, an advocacy group (interestingly, the director of plaintiffs' counsel, The Education Law Center), a labor leader and various educational associations. Silverstein testified this was the first such panel that had as its members a representative of an advocacy group and a teachers' union. The panelists were divided into five individual panels with each panel provided a hypothetical school district (K-8 small and very small; four K-12 small through very large). The panelists met from February 20th through February 21st, 2003 with Silverstein noting two days is the average length of time for the second panel. Each panelist received the results of the first panel by way of Excel spreadsheets and were instructed they could make modifications as appropriate without limitations. The second group of panelists were also provided with the same materials as had been provided to the first panelists, but also received the results of the first panel. An APA representative was present for each panel and Silverstein testified the five individual panels understood their respective tasks, worked through the process, and produced what was deemed necessary. The third panel, consisting of eight panelists (D-2, appendix 8), met on March 11th and March 12th, 2003 and reviewed the results obtained by the first and second panels. They were also provided with the same information as had been received by those panels. The panelists were five school superintendents, a professor from Kean University, College of Education, a school business administrator, and a school board member. The third panelists were told to create the final resources needed to adequately fund a system that could meet the CCCS. This panel's work was more focused but, again, the panelists were told they could make changes as deemed appropriate and necessary. The third panel completed its work in March 2003. See D-2, appendix 9-1 to -43 for final determination of resources needed. After the panel completed its work the costing out process was conducted by the DOE with APA's assistance. For reasons not made clear, the APA's study was put on hold and its report, the RCE, was released in December 2006. D-2. Although the PJP process generally only takes months to complete, the reason for the delay in the issuance of the RCE remains obscure. Despite the delay, Silverstein testified, unequivocally, the study remained reliable. The APA thereafter reviewed the costing out process completed by the DOE and was satisfied the same was valid, reliable and credible. Dean David Monk (Monk or Dean Monk), the Dean of the College of Education and a Professor of Educational Administration at Pennsylvania State University, was offered as an expert on behalf of the State, in part, to review the validity of the PJP process. His curriculum vitae evidenced his considerable academic achievements. D-91. Dean Monk is a recognized expert in educational finance. His testimony was thoughtful, temperate, non-partisan and moderate in approach. Of all the experts who testified concerning the PJP process, Monk's testimony was the most considered, evenhanded and well structured. Dean Monk has taught, researched and published concerning the subject of adequacy studies. He was qualified as an expert in education policy, finance and adequacy study methodology. Monk was retained by New Jersey as a consultant in the summer of 2007 to advise the State on the development and implementation of a new school funding formula and to critique the PJP process, its conclusions, as well as the other studies commissioned by the defendant. He was charged specifically with reviewing the methodology utilized including the PJP process. The Dean opined the results of the PJP process and the utilization of a weighted student formula would provide an appropriate formula or an acceptable result if the process utilized was correct and the funding adequate. A Weighted Student Formula is an appropriate and reasonable method to determine school funding. Experts in the field of adequacy have not only recognized a Weighted Student Formula as an appropriate method for school funding, but the growing consensus is that it is a preferred way to develop a school funding formula. D-123 ¶ 13. The process, when properly conducted, is constructed to provide an equitable result for all students, both within and beyond the Abbott districts. The Dean defined appropriate as the considered judgment of the resources needed to meet established standards, or in New Jersey's case, the CCCS. The process yields the best estimate of the resources needed to achieve the CCCS and is applied in an even handed manner. One of the benefits of the PJP process is it can be tailored to fit the needs of an individual State. The process itself, according to the Dean, brings together educators with varying perspectives who can share his/her acquired wisdom and expertise as it relates to the New Jersey educational process and the system to be constructed. The PJP process has been used to develop formulas for school fundings in many states in this country. Dean Monk commented the PJP process is one of the four generally accepted methodologies for the preparation of adequacy studies. He noted no one method is perfect, each has its own strengths and weaknesses, but the PJP process was the most often utilized and the most popular for adequacy studies. He went so far as to reference the PJP process as the preferred approach and has found the same to be reliable generally and specifically as it pertained to New Jersey. Dean Monk testified it is important to have a teacher's perspective on the panel(s), but a teacher's focus is often upon the class and, therefore, should only be one voice among many needed and afforded the opportunity to participate in the process. Administrators generally bring a broader perspective to the panel. One of the particular benefits of the PJP process is there is no over-reliance on any one voice, such as that of a teacher. In reviewing the RCE and APA's conduct in running the PJP process in New Jersey, the Dean noted APA's intervention was very capable, able and the firm and Augenblick, in particular, have a well-regarded reputation for running PJP panels. APA are among the top experts in [the development of adequacy funding formulas through the PJP process]. D-123 ¶ 19. Augenblick, as well as APA, has a wealth of experience in many states and is considered a leader in the field as it concerns the PJP process. The Dean testified the PJP process was part of the State's good faith effort to construct a constitutionally permissible formula and the process itself had been conducted in good faith. He further opined, the use of the PJP process was reasonable and provided a systemic approach to connect the inputs and outputs of the educational funding system. D-123 ¶ 26. The Dean was not concerned about the delay between the completion of the PJP process and the issuance of the RCE as it was his thought if the inflationary impact was considered, as it was in the instant matter, then the delay is not of moment. The same is, in part, attributable to the lack of fundamental change in resources/input information and the modest and minor changes made to the CCCS during the period of delay. As the resources/input data remains relatively stable over a three to five year period, the delay occasioned in the instant matter was not considered to be one of significance. Dean Monk reviewed and considered the reports of Odden, D-105, and Picus, D-74. He described both as leaders in their field who are well regarded nationally. Monk found their studies, utilizing the EBM, to be reassuring as they had both come to similar conclusions as that contained in the PJP process using different approaches. [10] On the whole, the court found Monk to be highly persuasive and a compelling witness as his opinions were reserved, yet firm. The principal attack on the PJP process itself on behalf of the plaintiffs was authored by Dr. Margaret E. Goertz (Goertz). Although Goertz' qualification as an educator cannot be gainsaid, P-1, her critique of the PJP process was less than persuasive. Goertz has been involved with the Abbott litigation for in excess of twenty years. Goertz testified in the remand hearing conducted before then A.L.J. Steven L. Lefelt in 1987, and thereafter testified before the Honorable Paul G. Levy, J.S.C. and the Honorable Michael Patrick King, P.J.A.D. in conjunction with subsequent remand hearings. One is compelled to wonder whether she has developed a vested interest in the issues presented thereby precluding a dispassionate review. Goertz' consideration of the PJP process was premised upon her review of other participants in the matter, principally Odden and Picus. She was compelled to admit both of those experts found the input/resources set forth by the PJP process were adequate to ensure a thorough and efficient education. Goertz' familiarity with the process as an adequacy methodology was premised primarily upon her reading the various reports and reviewing the methodology utilized. She acknowledged she has not conducted a PJP process, has not been a participant in the PJP process, and recognized Dr. Augenblick as more expert on the PJP process, how it is operated, how it is run, than she. She further acknowledged the PJP process is one of four accepted methodologies in attempting to construct an adequacy budget, but also testified she was unable to offer an opinion whether the PJP process is an appropriate method to develop an adequacy formula. Goertz, 11 T 55:23-56:6. [11] Goertz found five flaws in the PJP process. [12] The purported flaws were as follows: 1. The first panel was comprised of State employees only; 2. There were few teachers on the various panels; 3. The panelists were not provided with information concerning the Abbott litigation and the Court's imposed remedies; 4. The age of the report; and 5. The use of a model district. None of the purported flaws are found to be persuasive. Although it was conceded the first panel was comprised solely of DOE employees, [13] Goertz makes no mention that panels 2 and 3 were afforded unbridled authority to modify the recommendations of panel 1 as deemed appropriate. The criticism that there were few teachers on the panel and an under-representation of the Abbott districts, was an insight into what appears to be Goertz' vested interest. Goertz was compelled to admit on panel 3, the final panel to determine needed resources, three of the eight panelists were representatives of the Abbott districts and 37.5 percent hardly represents a small representation. Concededly, the panelists were not provided information concerning the Abbott decisions or its imposed remedies, but the same ignores the PJP methodology which allows participants to identify needed resources to teach all children, not only those in Abbott districts. Goertz noted, as had other witnesses, the PJP process concluded in 2003 yet the report was not authored until 2006. Goertz was unable to substantiate the assertion significant changes had been made to the CCCS in the interim, which assertion was directly contravened by Commissioner Davy, although she did note subsequent to 2006 the assessment criteria had become more stringent. Lastly, Goertz criticized the use of a model district asserting the same was mismatched to any of the Abbott districts. The same again ignores, though, the methodology of the PJP process which always utilizes a model district with the recognition a model district is utilized with the expectation it will not mirror or replicate any particular district. It is not possible to reconcile Goertz' recognition of the PJP process as an accepted methodology while not accepting the utilization of a model district. Goertz' secondary objection concerning the use of a model district as prohibited by Abbott IV is apparently contravened by the remand order. Were the Court to have found utilization of a model district prohibited, as urged by Goertz, then logically there would be no need for a remand hearing as the utilization of a model district was clearly before the Court when it rendered its November 2008 decision. While certainly understanding Goertz' position one cannot separate the product from the process, the same should not serve to invalidate a methodology which Goertz accepted. Recognizing there is no study on any correlation between funding and educational outcome, Goertz was not prepared to opine what amount of funding was necessary to ensure a thorough and efficient education while she was compelled to acknowledge New Jersey is one of the highest spending states in the country on educating its youth. Dr. Clive Belfield (Belfield) was also called as an expert on behalf of the plaintiffs. Although the primary focus of Belfield's testimony concerned supplemental programs as addressed in Abbott V and X, including high quality preschools under SFRA, he did testify concerning the PJP process. It is noteworthy, though, there is no direct mention or criticism of the PJP process in the summary of his opinions (P-18), or in his augmented certification to this court dated January 29th, 2009 (P-19). Belfield acknowledged the PJP process is one of the four accepted methodologies with no one method being clearly better than the other three. His personal preference is for a blended approach utilizing the successful school district approach with the EBM. Belfield believes it is preferable to review what programs are effective, then estimate the costs to replicate, fund accordingly and have the districts be obligated to implement successfully the program with the State being responsible for monitoring performance. Belfield did, though, opine during the course of his testimony the PJP process in New Jersey was poorly implemented. He first suggested the fault was not with the panelists, but rather, his conclusion of poor implementation was premised upon the flawed conclusions reached. He thereafter suggested the panelists must have been asked the wrong questions, although no specification was provided. Upon further inquiry, Belfield testified the right questions concerning input/resources/staffing were not posed. Belfield was particularly concerned the PJP process provided funding for at-risk populations capped at sixty percent, particularly when 24 of the 31 Abbott districts have an at-risk population exceeding that percentage and three have an at-risk population exceeding eighty percent. Belfield believed there was no literature to support capping expenses for the at-risk population at sixty percent. Belfield concluded there must have been insufficient data provided concerning at-risk students and populations in the various Abbott districts. Belfield's concern in this area is worthy of consideration. That said, his opinion was premised upon the outcome derived, rather than the process itself. The plaintiff also called Dr. Bruce Baker (Baker), a prolific writer and an energetic educator. Baker authored the most statistically comprehensive attack on the PJP process. He has been a consultant on various adequacy studies and has written widely on the subject. He was qualified as an expert in school finance, educational costing methods and studies. His primary critique of the PJP process was the process itself was structurally flawed and the assumptions utilized were arbitrary. From an analytic framework he criticized the PJP process in the following respects: 1. The hypothetical districts utilized did not represent adequately the Abbott districts; 2. The panelists were not provided with the Abbott mandated supplemental programs; 3. The six models utilized were prepared by the DOE rather than by the panelists which was inconsistent with prior PJP processes conducted by APA; 4. The second round panelists were over-represented by the DFG districts G, H and I and under-represented by representatives of the A, B districts including the Abbott districts; and 5. Without additional validation, the PJP process may not establish the requisite causative link between resources and outcome. As with the other experts, Baker testified there are four accepted methodologies for conducting an adequacy study. He distilled the four methods to two: 1. Input oriented methods (the PJP and the EBM which processes estimate costs to achieve a desired outcome by identifying necessary resources); and 2. Outcome oriented methods (the cost function and successful school district approach which look to the relationship between outcomes and resources necessary to obtain those outcomes). Baker criticizes the input methods as they are not measured against a known outcome and, therefore, the method is more hypothetically based than factually based. Baker understandably asserts the more comprehensive and relevant the data provided, the more reliable the outcome obtained. Baker believes the outcome methods are preferable as they are less hypothetical in that they utilize a known outcome and, with the correct data provided properly analyzed, the more certain the result obtained. Baker was retained by the plaintiff to review: 1. whether the funding as set forth in the SFRA is constitutionally adequate; 2. the PJP cost analysis; 3. the design of the SFRA; 4. the relationship between the PJP and the SFRA; and 5. whether the PJP process in New Jersey provides the requisite link between costs/resources necessary to achieve a desired outcome in all districts, and particularly the Abbott districts. For purposes of this section items 2, 4, and 5 are relevant. Baker acknowledged if the PJP process is conducted properly by informed panelists who discuss the full range of issues an acceptable result may be obtained. He acknowledged Augenblick has conducted many PJPs, had been retained by many states to perform adequacy studies using the PJP process, and Picus and Odden were recognized experts in the field of adequacy studies. Baker posits, though, the results obtained by the PJP process should be tested empirically utilizing an outcome based methodology to ensure greater reliability. He opined the result must be validated, that there must be a reality check. Baker criticized the information provided to the various panelists. He noted the panelists were not provided with the Court's prior Abbott rulings, a list of required supplemental programs, or the staffing resources necessary to comply with the Court's mandate for the supplemental programs. He acknowledged this is not a criticism of the PJP process itself, but rather, was a failure to comply with Court mandates. As the Court has indicated the parity remedy for Abbott districts is not the sine qua non of a successful funding formula, this court is not prepared to accede to the assertion the PJP process cannot establish the necessary funding without the panelists being provided with the Court's prior mandates. It further presupposes the retention of the various supplemental programs is required, notwithstanding the purported constitutionality of SFRA. Baker criticized the panelists who comprised the various panels. He testified having reviewed all of Augenblick's prior PJP panels, the first panel has generally addressed school level resources only, whereas the second panel then addressed district level resources needed, as well as considering special or additional needs. Baker noted the New Jersey PJP process was a modified prototype in the following respects: 1. The DOE provided the data and chose the panelists; 2. The first panelists were comprised of DOE personnel only; and 3. The second panelists comprised an over-representation of G, H and I districts and an under-representation of A and B and Abbott districts at the district level. Baker commented, as set forth on P-54 at 9, figure 3, that 19 of the 35 invited panelists were from the G, H, and I districts, and 12 of 15 of the upper level administrators were from those same districts. Baker asserted as educators chosen to be panelists bring their own experiences to the process, over-representation by the wealthier districts may not bring to the discussion the requisite knowledge of Abbott districts, or may not adequately represent those districts' interests. It is noted Baker suggested no attack on the work conducted by the third panel. This represents a serious deficiency in Baker's critique of panels one and two as the third panel was given the unbridled right to make changes and modifications it deemed necessary for the resources needed to meet the CCCS. This is even more significant as the Abbott districts were represented by three of the eight panelists in the third panel, or 37-½ percent. Baker admitted there was no information available to him as to what the various panels discussed, how consensus was achieved, or even what happened during the panelists' work, nor did he have any information the panelists did not have sufficient experience to do what they were charged to do. Baker next explored the prototype school districts used with his basic criticism the models do not look like the Abbott districts. Baker opined there was insufficient representation of the Abbott school districts in the models used as the models failed to account for a high percentage population of students and families in poverty as those districts were outside the hypothetical range of the six models. See P-54 at 11, figure 5. Further, the percentage of large Abbott districts by way of enrollment were also outside the prototypes utilized. P-54 at 12, figure 6. Baker therefore opined the six hypothetical districts did not represent adequately actual districts in New Jersey, particularly the Abbott districts and, as such, the per-pupil cost determined by the prototype as applied to the Abbott districts may be unnecessarily skewed. Baker further criticized the process as lacking consideration of additional costs when the percentage of poverty students exceeds sixty percent, which is many of the larger Abbott districts. Baker opined the assumption costs even off after a certain poverty level is obtained is not supported adequately by any study referenced. As such, Baker, as did Belfield, questions whether the PJP process identified sufficient resources for districts with high concentration of poverty which, of course, are the Abbott districts. Baker, 17 T 99:21-100:6; Belfield, 15 T 60: 25-61:6. Dr. Baker is a magician with numbers and to that deference is due. What must though be examined is whether his statistical analysis leads to a meaningful critique as contrasted to a mere statistical review of SFRA. Figures often beguile me, particularly when I have the arranging of them myself in which case the remark attributed to Disraeli would often apply with justice and force: There are three kinds of lies: lies, damned lies and statistics. Mark Twain, Autobiography of Mark Twain 149 (Harper 1959). The PJP process is accepted for what it wasone step in an intricate, involved methodology by which the DOE and thereafter the legislature and the Governor, attempted to construct and enact a comprehensive funding formula in an attempt to ensure a thorough and efficient education for all students in New Jersey. It appears clear each of the four methodologies has its strengths and weaknesses and no one methodology can ensure the development of a failsafe funding formula to meet any specified standard. That said, it also appears clear APA implemented a fair process leading to an informed review of the necessary funding required to attempt to ensure a thorough and efficient education as required by the CCCS. After due consideration of the multiple critiques of the PJP process, and acknowledging no one methodology can predict with unerring accuracy the monies needed to meet the standards provided (here, the CCCS), the court is satisfied the PJP process established fairly and equitably the first step in constructing a constitutionally mandated equitable funding formula for all districts, include the Abbott districts.