6 Standard Setting
The standard-setting process for the Dynamic Learning Maps® (DLM®) Alternate Assessment System in English language arts (ELA) and mathematics derived cut points for describing student achievement relative to four performance levels based on results from the 2014–2015 DLM alternate assessments. For a description of the process, including the development of policy performance level descriptors, the 4-day standard setting meeting, follow-up evaluation of impact data and cut points, and specification of grade- and content-specific performance level descriptors, see Chapter 6 of the 2014–2015 Technical Manual—Year-End Model (Dynamic Learning Maps Consortium, 2016).
6.1 Standard Adjustment for Blueprint Revisions
For the spring 2020 assessment, the testing blueprint was revised for both ELA and mathematics. State education agency members agreed to reduce the total number of Essential Elements (EEs) on the blueprint to increase the number of items that assess each standard. For a complete description of the blueprint revisions, see Chapter 3 of the 2019–2020 Technical Manual Update—Year-End Model (Dynamic Learning Maps Consortium, 2020). As a result of the blueprint revisions, at standards adjustment process was planned for May 2020. The adjustment process consisted of two phases. In the first phase, DLM staff would recommend preliminary cut points based on the proportion of EEs reduced from each subject and grade’s blueprint, the expected distribution of total linkage levels mastered under the new blueprint, and the expected changes to consortium-level impact data. In the second phase, state education agency members would use impact data from the spring 2020 assessment to review the preliminary cut points and optionally adjust the cut points by \(\pm1\). However, because of the COVID-19 pandemic, all states canceled their spring assessments, and therefore performance levels were not reported. Thus, the standards adjustment was rescheduled for the spring 2021 assessment administration.
As described in Chapter 1 of this manual, instruction during the 2020–2021 academic year and the spring 2021 assessment were also affected by the COVID-19 pandemic, with effects varied by state and district. Although some states and districts administered the assessment as intended, the variability in participation and unknown effect of the pandemic on instruction throughout the year meant that impact data from the spring 2021 assessment would likely be unreliable and not representative of data collected under normal conditions. Thus, in consultation with the Technical Advisory Committee (TAC), DLM staff made the decision to proceed with only the first phase of the standards adjustment process in 2021. That is, the cut points used for the spring 2021 assessment were set administratively, based on analyses conducted in anticipation of the originally scheduled adjustment process in 2019–2020. The preliminary cut points were shared with the TAC in February 2020, and the TAC expressed support for the preliminary cut points and the process used to arrive at the cut points. Phase two of the adjustment process will be conducted as part of the spring 2022 assessment administration.
6.2 Administrative Adjustment
The standards adjustment process had two overarching goals:
- Maintain high expectations for students
- Minimize impact in performance level shifts
Some amount of variation in performance across years was expected due to the change in assessment design that coincided with the changes to the assessment blueprint (i.e., more items per EE and linkage level and a change to testlets measuring a single EE).
To arrive at the preliminary cut points, the original cut points were first reduced by the proportion of blueprint reduction. That is, if there was a 20% reduction in the total number of EEs for a subject and grade, each cut point for that grade was similarly reduced by 20%. The spring 2019 assessment data were then used to estimate the distribution of total linkage levels mastered under the revised blueprint. That is, using the spring 2019 assessment data, a new total linkage levels mastered was calculated using only the EEs that were retained in the blueprint revision. In two grades (sixth and eighth grade ELA), a new writing EE was added to the blueprint. For these students, the mastery for the new EE was calculated as the average of the mastery on the other writing EEs. This new distribution of expected linkage levels mastered and the initial proportionally reduced cut points were then used to calculate the expected impact data under the new blueprint. The initial cut points were then adjusted by \(\pm1\) to minimize the differences from the performance level distributions that were observed during the spring 2019 assessment (i.e., prior to the blueprint revisions). This resulted in the preliminary cut points shown in Table 6.1 that were supported by the TAC. These preliminary cut points were used to score the 2021 assessments. However, due to the confounding factors of assessment administration changes and COVID-19, the cut points were not fully evaluated. State partners will have the option to adjust the cut points in Table 6.1 by \(\pm1\) based on an impact data review in spring 2022.
| Grade | Emerging/Approaching | Approaching/Target | Target/Advanced |
|---|---|---|---|
| English Language Arts | |||
| 3 | 23 | 31 | 44 |
| 4 | 25 | 37 | 49 |
| 5 | 21 | 31 | 45 |
| 6 | 18 | 31 | 44 |
| 7 | 20 | 38 | 53 |
| 8 | 18 | 37 | 55 |
| 9 | 15 | 39 | 55 |
| 10 | 13 | 38 | 58 |
| 11 | 15 | 39 | 58 |
| 12 | 15 | 39 | 58 |
| Mathematics | |||
| 3 | 9 | 15 | 27 |
| 4 | 10 | 15 | 28 |
| 5 | 8 | 17 | 26 |
| 6 | 8 | 18 | 24 |
| 7 | 9 | 18 | 27 |
| 8 | 9 | 24 | 31 |
| 9 | 9 | 18 | 29 |
| 10 | 8 | 19 | 31 |
| 11 | 6 | 12 | 24 |
| 12 | 6 | 12 | 24 |