Glossary

accommodation. A change in how a test is presented, in how it is administered, or in how the test taker is allowed to respond. This term generally refers to changes that do not substantially alter what the test measures. The proper use of accommodations does not substantially change academic level or performance criteria. Appropriate accommodations are made to provide equal opportunity to demonstrate knowledge. The most frequently used accommodations in NAEP are large-print booklets, extended time in regular test sessions, reading questions aloud in regular sessions, small groups, one-on-one sessions, scribes or use of computers to record answers, bilingual booklets (mathematics assessment only), and bilingual dictionaries (not for the reading assessment). In NAEP, accommodations may be provided to certain students with disabilities (SD) and/or English learners (EL), as specified in the student's Individualized Education Program (IEP).

achievement-level percentages. The percentage of students within the total population, or in a particular student group, who meet or exceed expectations of what students should know and be able to do. Specifically, it is the weighted percentage of students with NAEP composite scores that are equal to, or exceed, the achievement-level cut scores specified by the National Assessment Governing Board.

administration form. Form used by field staff to prepare booklets, record attendance status, and assessment participation status.

administration schedule (AS). A form prepared for each session to be held in a cooperating school. The administration schedule serves as a student roster to be used to carry out the session that lists the students that are to be included in the assessment in that session.

advanced math. Used for the High School Transcript Study, advanced mathematics includes courses, other than calculus, that are generally taken after algebra II (e.g. AP statistics and precalculus).

advanced science. Used for the High School Transcript Study, advanced science are science courses that contain advanced content (like AP Biology, IB Chemistry, AP Physics, etc.) or are considered second-year courses (Chemistry 2, Advanced Biology, etc.). Students may take advanced science courses (like second-year chemistry) instead of physics.

algorithmic scoring. Synonymous with machine scoring, algorithmic scoring is the application of a score rubric to incorrect, partially correct, or completely correct responses to test questions that have a definable and finite number of possible response patterns.

almanac. A comprehensive collection of tables of NAEP results.

Alpha sample. The Alpha sample is the name given to the fourth and eighth grade public school and student samples for the operational studies in a given NAEP year.

American Community Survey (ACS). The American Community Survey (ACS) is an ongoing survey conducted by the U.S. Census Bureau. It uses a series of monthly samples to produce annually updated estimates for the same small areas (census tracts and block groups) formerly surveyed via the decennial census long-form sample.

assessment administrator (AA). A trained proctor who administers the assessments in a particular session. Also called an exercise administrator.

assessment coordinator (AC). A trained contractor who administers the assessment and oversees all NAEP activities in a school.

assessment session. A group of students reporting for the administration of an assessment. Most schools conduct only one session, but some large schools conduct as many as 10 or more. Also referred to as a "session."

Authorized Return Service. A service offered by United Parcel Service, which provides for the easy return of UPS-compatible packages. Preprinted return labels are provided to the shipper by UPS; the shipper includes the label with an outbound shipment or distributes separately to its customers.

average weighted percent correct. For multiple choice and dichotomous constructed-response items, weighted percent correct is the same as percent correct, i.e., percentage proportion of examinees who received a correct score on the item. For polytomous items, weighted percent correct is the sum of percentage proportion of examinees in each score category weighted by the magnitude of each score category and standardized with a maximum credit of 1. For example, if there are 3 scoring categories (0, 1, and 2) for an item and percentage distribution for the item across three score categories is 20%, 40%, and 40%, respectively, then the weighted percent correct will be: 20(percent)* 0 (point)/2 (maximum score) + 40 (percent)* 1 (point)/2 (maximum score) + 40* (percent)*2 (point) /2 (maximum score) = 60 (percent). Average weighted percent correct refers to an average of weighted percent correct across items.

background questionnaires. The instruments used to collect information about student demographics and contextual information about student educational experiences.

Beta sample. The Beta sample is the name given to fourth and eighth grade public school and student samples for any studies other than the operational studies in a given NAEP year.

bias. In statistics, the difference between the expected value of an estimator and the population parameter being estimated. If the average value of the estimator across all possible samples (the estimator's expected value) equals the parameter being estimated, the estimator is said to be unbiased; otherwise, the estimator is biased.

BIB (Balanced Incomplete Block) booklet design. A complex variant of matrix sampling in which items are administered so that each pair of items is dispensed to a nationally representative sample of respondents in a specific pattern.

biserial correlation coefficient. The correlation between a dichotomous variable and a continuous variable obtained by hypothesizing the existence of a continuous "latent" variable underlying the dichotomous variable.

block. A group of assessment items created by dividing the item pool for an age or grade into subsets. Blocks are used in the implementation of the BIB spiral sample design.

booklet. An assessment instrument created by combining blocks of assessment items. Prior to 2017, most NAEP assessments utilized a paper-based format, and were administered to students via printed test booklets. Selected items have been trans-adapted from the paper-based assessments (PBA) for administration in a digitally based assessment (DBA) environment via digital test forms.

booklet distribution map, booklet map, bookmap, bundle map. A plan for assembling booklets into bundles for distribution to students.

bridge study. A study in which two randomly equivalent samples of students are selected. One sample is assessed using the existing design and the other using the modified design. The primary purpose of a bridge study is to maintain a constant scale for trend reporting.

bundle. A package of booklets delivered to a testing site for administration to students.

Bureau of Indian Affairs. Bureau of Indian Affairs provides education services to approximately 48,000 students of American Indian ethnicity.

Bureau of Indian Education (BIE). An office within the Bureau of Indian Affairs with responsibility for providing quality educational opportunities for American Indians. The Bureau operates elementary and secondary schools for American Indians, funded by the federal government.

calibrate. To estimate the parameters of a set of items using responses of a sample of examinees.

Carnegie unit. The number of credits a student received for a course taken every day, one period per day, for a full school year; a factor used to standardize all credits indicated on transcripts across the study.

causal relationship. A relationship between two variables in which changes in the value of one variable cause changes in the value of the other variable.

Census division. A grouping of states within a census geographic region, established by the Census Bureau for the presentation of census data. The nine divisions are intended to represent relatively homogeneous areas that are subdivisions of the four census geographic regions, and are as follows: 1. New England - Maine, New Hampshire, Vermont, Massachusetts, Rhode Island, Connecticut. 2. Mid Atlantic States - New York, New Jersey, Pennsylvania. 3. East North Central - Ohio, Indiana, Illinois, Michigan, Wisconsin. 4. West North Central - Missouri, Iowa, Minnesota, North Dakota, South Dakota, Nebraska, Kansas 5. South Atlantic - Delaware, Maryland, District of Columbia, Virginia, West Virginia, North Carolina, South Carolina, Georgia, Florida. 6. East South Central - Kentucky, Alabama, Tennessee, Mississippi. 7. West South Central - Louisiana, Arkansas, Texas, Oklahoma. 8. Mountain - New Mexico, Arizona, Colorado, Utah, Nevada, Montana, Idaho. 9. Pacific - California, Oregon, Washington, Alaska, Hawaii.

certainty. An entity included in a sample with certainty has a selection probability of one.

certainty PSU. A primary sampling unit (PSU) that is automatically included in the sample. Its selection probability is one.

charter school. A public charter school is a publicly funded school that, in accordance with an enabling state statute, has been granted a charter exempting it from selected state or local rules and regulations. A charter school may be newly created, or it may previously have been a public or private school; it is typically governed by a group or organization (e.g., a group of educators, a corporation, or a university) under a contract or charter with the state. In return for funding and autonomy, the charter school must meet accountability standards. A school's charter is reviewed (typically every 3 to 5 years) and can be revoked if guidelines on curriculum and management are not followed or the standards are not met.

city. City includes territories

inside an urbanized area and inside a principal city with population of 250,000 or more,
inside an urbanized area and inside a principal city with population less than 250,000 and greater than or equal to 100,000, and
inside an urbanized area and inside a principal city with population less than 100,000.

For more information about the school location variable, visit https://nces.ed.gov/surveys/ruraled/definitions.asp.

classical test statistics. Counts, percentages, measures of item difficulty, and measures of item discrimination that are not based on Item Response Theory (IRT).

classical test theory. A set of measurement concepts that postulates that a test score can be decomposed into a true score and an error component; that the error component is random, has a mean of zero and is uncorrelated with true scores; and that observed scores are linearly related to true scores and error components.

Classification of Secondary School Courses (CSSC). A coding system employed for the purpose of standardizing High School Transcript Study (HSTS) transcripts. The CSSC, a modification of the Classification of Instructional Program (CIP) is used for classifying college courses and contains 2,268 course codes. (For more information see http://nces.ed.gov/surveys/hst/courses.asp).

cluster item. A pseudo-item formed by combining the responses to two or more actual items presented in an assessment. For instance, a cluster item composed of several multiple-choice items might be the number of those items to which the student responded correctly.

cluster sampling. The selection of sets (clusters) of units rather than individual units. In cluster sampling, survey population members are divided into unique, nonoverlapping groups prior to sampling. Clusters are often naturally occurring groups such as schools, or geographic units such as city blocks. Once clusters are randomly selected in the sample, all sampling units in each cluster are included in the sample. Clustered sampling usually decreases the precision of the statistics as compared to stratified sampling. On the other hand, clustering usually results in reducing survey costs. Specifically, transportation and training costs are substantially lessened, as all sampling units are surveyed in one location.

clustering. The process of forming sampling units as groups of other units.

codebook. A formatted printout of NAEP data for a particular sample of respondents.

coefficient of variation (CV). The ratio of the standard deviation of an estimate to the value of the estimate.

common block. A group of background items included at the beginning of every assessment booklet.

common calibration linking. (Also referred to as concurrent calibration linking.) Linking Item Response Theory (IRT) scales by calibrating responses to items on the scales together using items that are common to the scales to provide a connection between the scales.

common population linking. Linking scales by matching the distributions of scores on two different scales for a single group or for randomly equivalent groups of examinees.

complex sample design. A sample design that incorporates stratification, multistage sampling, and/or varying probabilities of selection. NAEP utilizes all of these. This term is contrasted with the term simple random sample design.

composite scale. An overall subject-area scale based on the weighted average of the scales that are used to summarize performance on the primary dimensions of the curricular framework for the subject-area assessment. For example, the mathematics composite scale is a weighted average of five content-area scales: number sense, properties, and operations; measurement; geometry and spatial sense; data analysis, statistics, and probability; and algebra and functions. These five scales correspond to the five content-area dimensions of the NAEP mathematics framework.

conditional correlation coefficient. An estimate of the correlation between subscale scores taking into account the population model of NAEP. See also marginal correlation.

conditional probability. Probability of an event happening, given the occurrence of another event.

conditioning. The process of imputation used in NAEP that allows plausible values to be drawn at random from a conditional distribution of a NAEP respondent, given his or her response to cognitive exercises and to a specific subset of background variables (conditioning variables).

conditioning variables. Demographic and other background variables characterizing a respondent. These variables are used to construct plausible values.

confirmatory factor analysis. A method that provides an explanation of the relationships among variables in terms of a smaller number of unobserved variables called factors, in which assumptions about the relationships among the variables are tested.

consistent estimator. An estimator that when calculated with data from the whole population will equal the value of what is being estimated.

construct. An abstract image, idea, or theory, especially a complex one, formed from a number of simpler observable elements. Constructs represent ideas constructed by scientists to help summarize a group of related phenomena or objects.

constructed-response item. An assessment item that requires students to provide a written response of their own, as opposed to a selected-response item which typically provides response options to choose from. NAEP paper-based assessments (PBAs) and digitally based assessments (DBAs) both include constructed-response items, although in DBAs the responses are typed using a keyboard rather than written in a printed test booklet as they are in PBAs.

contrasts. Variables that define specific groups; most often these variables equal 1 when the group is one to which the student belongs and 0 otherwise.

core academic courses. A course type defined for the High School Transcript Study, core courses are English, mathematics, science, and social studies.

Core Based Statistical Area (CBSA). The 2020 Census standards provide that each CBSA must contain at least one urban area of 10,000 or more population. Each metropolitan statistical area must have at least one urbanized area of 50,000 or more inhabitants. Each micropolitan statistical area must have at least one urban cluster of at least 10,000 but less than 50,000 population.

correlation. A measure of the relation between two or more variables. Correlation coefficients can range from -1.00 to +1.00. The value of -1.00 represents a perfect negative correlation while a value of +1.00 represents a perfect positive correlation. A value of 0.00 represents a lack of correlation.

course types. The High School Transcript Study reports credits earned for three types of courses: core academic, other academic and, other courses.

credits earned. In the High School Transcript Study, course credits earned are converted to standardized Carnegie units, in which a single Carnegie unit is equal to 120 hours of classroom instruction over the course of a year. One Carnegie credit is often described as what a student earns for completing a one year course that meets 40 minutes per school day (assuming 180 school days in a school year). However, it should be noted that some courses may meet for one semester for 0.5 credits, while others may meet for 60 minutes each day for a year for 1.5 credits.

CSV (comma-separated values). A file format used as a portable representation of a database. Each line is one entry or record; the fields in the record are separated by commas. This format is often used to import data into spreadsheet software.

Current Population Survey (CPS). The Current Population Survey (CPS) has been conducted by the Bureau of the Census for more than 50 years. The CPS is the primary source of information on the labor force characteristics of the U.S. population.

curriculum level. For the purposes of the High School Transcript Study, three curriculum levels were defined: standard, midlevel, and rigorous. The curriculum levels are based on the number of credits and the types of courses graduates take.
This is a modified version of curriculum levels used by Laura Horn and Lawrence K. Kojaku (U.S. Department of Education. National Center for Education Statistics. High School Academic Curriculum and the Persistence Path Through College, NCES 2001-163. Project Officer: C. Dennis Carroll. Washington, DC: 2001). This modification was made to ensure that HSTS data for earlier years are consistent with 2005.

cut score. The minimum score required for performance at each NAEP achievement level. NAEP cut scores are determined through a standard-setting process that convenes a cross-section of educators and interested citizens from across the nation. The group determines what students should know and be able to do relative to a body of content reflected in the framework. The National Assessment Governing Board then adopts a set of cut scores on the scale that defines the lower boundaries of NAEP Basic, NAEP Proficient, and NAEP Advanced.

degrees of freedom (df) [of a variance estimator]. The number of independent pieces of information used to generate a variance estimate.

Delta sample. Sample of students enrolled in private schools in grades 4, 8, and 12.

Department of Defense Dependents Schools (DoDDS). (see also DoDEA). One of two distinct educational systems operated by the Department of Defense Education Activity (DoDEA). DoDDS provides comprehensive educational programs on military installations overseas.

Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS). (see also DoDEA). One of two distinct educational systems operated by the Department of Defense Education Activity (DoDEA). DDESS provides comprehensive educational programs on military installations located in seven states and Puerto Rico.

Department of Defense Education Activity (DoDEA). A civilian agency of the U.S. Department of Defense. It is divided into two separate but parallel systems: the Department of Defense Dependents Schools (DoDDS) overseas, and the Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) in the United States.

derived variables. Student group data that were obtained through interpretation, classification, or calculation procedures rather than from assessment responses.

design effects. The ratio of the variance for the sample design to the variance for a simple random sample of the same size.

dichotomous item. In NAEP, a common multiple-choice item or an item that requires a constructed response from the student, with the response being subsequently scored in one of two categories, being either correct or incorrect.

differential item functioning (DIF). An item exhibits differential item functioning if the probability of doing well on the item depends on group membership, even after controlling for overall performance.

disadvantaged minority. Used to define minority school status, historically disadvantaged minorities include Black students and Hispanic students.

discriminant validity. A type of construct validity where it is shown that assessment scores have a low correlation with other scores that should not be related to the assessment of interest; in contrast, convergent validity shows that assessment scores have a high correlation with other scores that should be related to the assessment of interest.

distractor. An incorrect response choice included in a multiple-choice item.

district size. A variable used in sampling schools which classifies schools into two groups: large districts and small districts. Large districts contain at least 20 percent of the jurisdiction's eligible grade enrollment. Small districts contain less than 20 percent of the jurisdiction's eligible grade enrollment.

economically disadvantaged (ED). NAEP collects and reports economically disadvantaged status as defined by the state for federal accountability reporting. This varies by state and includes economic disadvantaged indicators such as eligibility for state or federal assistance programs. Using National School Lunch Program (NSLP) data as the sole proxy for economic status has not happened since before 2015.

Educational Testing Service (ETS). The item development, instruments, database, and data analysis contractor for NAEP.

effect size. A way to compare the scores for two groups or for the same group under two different conditions that takes into account how variable the scores are; the standardized mean difference is the most common example.

English learners (EL). A term used to describe students who are in the process of acquiring English language skills and knowledge. Some schools refer to these students using the term limited-English-proficient (LEP). "Limited English Proficient" is also the terminology used in NAEP technical documentation prior to the 2005 NAEP assessment, while the term "English language learners" (ELL) was used from 2005 through 2018. Note that for Puerto Rico, "EL" identifies Spanish learners.

estimation. Process by which sample data are used to indicate the value of an unknown quantity in a population.

ETS. See Educational Testing Service.

excluded student questionnaire. An instrument completed for every student who was selected to participate but ultimately excluded from the assessment.

excluded students. Sampled students determined by the school to be unable to participate because they are English learners (EL) or have a disability.

exercise administrator (EA). Westat field staff member hired and trained to administer the assessment and to assist his or her NAEP supervisor with other NAEP activities in the school. As of 2003, this term is no longer used in Westat administrative materials. Also called an assessment administrator.

expected a posteriori (EAP) scores. Used as proxies for the latent construct (e.g., persistence in learning, academic self-discipline, enjoyment of complex problems) computed at the student level. The EAP scores are computed by multiplying the likelihood function by the weights from the prior distribution, and the resulting posterior distribution is averaged over the given range of theta.

expected value. The average of the sample estimates given by an estimator across all possible samples. If the estimator is unbiased, then its expected value will equal the population value being estimated.

explicit stratification. Explicit stratification consists of building separate sampling frames, according to the set of explicit stratification variables under consideration; used for categorical variables. Contrast this with implicit stratification.

factor analysis. A procedure that provides an explanation of the relationships among variables in terms of a smaller number of unobserved variables called factors.

Falcon System. A computer software package for data entry that allows the user to define the data entry screens and output record layout in one step.

field director. Westat home office staff member who coordinates and oversees all aspects of NAEP field procedures.

field manager. Westat field staff member hired to coordinate all NAEP field activities with the State Departments of Education and the Westat home office staff.

field supervisor. Westat field staff member hired to manage assessment teams, to select the samples of students to be assessed, and to send NAEP materials to the participating schools.

field test. Items in NAEP mathematics and reading assessments at grades 4 and 8 go through two phases of pretesting: pilot testing and field testing. A field test is the second phase of pretesting and is given one year prior to the operational NAEP assessment. After the field test, the development for the assessment instruments for the following year is finalized. The instruments are then administered to a nationally representative sample of students, and the analytical steps for estimating the distribution parameters of items by population and reporting groups are conducted. NOTE: Previously, the term "field test" referred to the first phase of item pretesting in all NAEP subject-area assessments. However, beginning with the 2003 assessments, the term applies only to reading and mathematics. The phase of pretesting formerly referred to as a field test, beginning in 2003 and for all future assessments, will be referred to as the "pilot test." All items in NAEP assessments are pilot tested, but only reading and mathematics are field tested.

focal group. The group of students of interest in an analysis of differential item functioning; often this is a group of students considered to be in the minority.

focused BIB spiraling. A variation of BIB spiraling in which items are administered so that each pair of items within a subject area is dispensed to a nationally representative sample of respondents.

focused booklet design. A booklet design in which each booklet contains items pertaining to a single subject area.

foils. The correct and incorrect response choices included in a multiple-choice item.

frequency distribution (FD). Summarizes the distribution of data points, showing how frequently different values appear.

gamma sample. A nationally representative sample of students enrolled in twelfth grade in public schools.

gender. NAEP results are reported separately for males and females, based on students' self-reported gender.

grade enrollment. The number of students within an assessed grade.

grade point average (GPA). GPA is used in the High School Transcript Study. Points are assigned to each letter grade as follows: A=4 points; B=3 points; C=2 points; D= 1 point; F= 0 points. The points are weighted by the number of Carnegie credits earned, so that a course with 120 hours of instruction counts twice as much as one with 60 hours. The average of the points earned for all the courses taken is the grade point average. Courses in which a graduate did not receive a grade, such as pass/fail and audited courses, do not factor into the GPA calculation. No additional grade points are assigned for Advanced Placement, International Baccalaureate, and other honors classes. This process does not standardize for differences in grading practices among schools and teachers.

grid. This selected-response item type allows students to evaluate mathematical statements or expressions with respect to certain properties. The answer is entered by selecting cells in a table in which rows typically correspond to the statements and columns to the properties checked.

group effect. The difference between the mean for a specific group and the mean for the nation.

hierarchical stratification. A form of stratification that uses multiple levels based on varying unit characteristics (e.g., Census region followed by school type). The nesting of lower-level strata within the higher-level strata may vary between higher-level strata in hierarchical stratification. For example, in one level of the higher-level stratum, there may be no lower-level strata and in another of the higher-level stratum, there may be numerous lower-level strata.

high density schools. Defined by the Office of Indian Education as those schools in which at least 25 percent of the students are American Indian or Alaska Native.

High School and Beyond. A longitudinal study following cohorts of 1980 high school students from which the 1982 High School Transcript Study (HSTS) sample was drawn. Samples for subsequent studies were drawn from the corresponding NAEP samples.

High School Transcript Study (HSTS). A periodic study developed by NCES to provide the Department of Education and other educational policymakers with information about current course offerings and students' course-taking patterns in the nation's secondary schools.

hits. The number of times a school was selected for NAEP. Large schools may be selected, or hit, more than once. For these schools, additional students are selected within the school. For example, if a school hit once had up to 30 students sampled, a school hit twice may have had up to 60 students sampled.

hotdeck. An imputation method that replaces missing values in a data set with values from other records in the data set.

hybrid MB1. A subsample of the short-form mathematics sample to which the first and third blocks of the main market-basket form MB1 and the second block of the second main market-basket form MB2 were administered.

hybrid MB2. A subsample of the short-form mathematics sample to which the first and third blocks of the second main market-basket form MB2 and the second block of the main market-basket form MB1 were administered.

image-based scoring system. Constructed-response items are scanned, and all student responses to those items are captured and stored on a server until scorers have been trained. All responses on the server for each particular constructed-response item are sent to those specific trained scorers. Once that item is completed, the next constructed-response item is trained and scored.

implicit stratification. A method of achieving the benefits of stratification often used in conjunction with systematic sampling. The sampling frame is sorted with respect to one or more stratification variables but is not explicitly separated into distinct strata. Contrast this with explicit stratification. See also sort variable.

imputation. Prediction of a missing value based on some procedure, using a mathematical model in combination with available information. See also plausible values.

imputed race/ethnicity. The race or ethnicity of an assessed student as derived from his or her responses to particular common background items. A major NAEP reporting group for assessments prior to 2001.

imputed values. Values generated through imputation. In NAEP, the imputed values are called plausible values. See also imputation.

independent samples. Two samples are independent if the realization of the first sample does not affect how the second sample is drawn.

Individualized Education Program (IEP). A written statement for each individual with a disability that is developed, reviewed, and revised in accordance with Title 42 U.S.C. Section 1414(d).

ineligible. Ineligible students are students who are not eligible to be sampled for the NAEP assessment. This category includes students found to be in a grade other than the one being assessed, or who are listed as enrolled but not physically attending the school, as with homeschooled students. Also included in the category of ineligible students are foreign exchange students and those who attend classes off campus.

intraclass correlation. A ratio of the variance of interest over the sum of the variance of interest plus error. In NAEP, it is used to describe the accuracy of raters scoring student constructed item responses.

item. The basic scorable part of an assessment; a test question.

item bias. An item is biased if the probability of the student doing well on the item depends not only on

what the examinee knows and can do and
the characteristics of the item as reflected in the item parameters,

but also on

a characteristic of the item that is unrelated to the construct being measured.

Item bias is shown through a statistical technique that updates the entries in a multiway frequency table, so that the resulting quantities in the cells conform to a new set of marginal figures, while preserving the higher-order associations present in the original table. See also bias.

item map. Item maps illustrate the knowledge and skills demonstrated by students performing at different scale scores on a given assessment. The item map provides concrete examples of what students at various achievement levels likely know and can do in a subject.

item response function (IRF). An equation or the plot of an equation that indicates the probability of an item response for different levels of the overall performance.

Item Response Theory (IRT). Test analysis procedures that assume a mathematical model for the probability that an examinee will respond correctly to a specific test question, given the examinee's overall performance and characteristics of the questions on the test.

iterative proportional fitting. A statistical technique that updates the entries in a multiway frequency table, so that the resulting quantities in the cells conform to a new set of marginal figures, while preserving the higher-order associations present in the original table.

jackknife. A replication method that estimates standard errors of percentages and other statistics. It is particularly suited to complex sample designs. In the jackknife, sample units are grouped into pairs (replicate groups). Portions of the sample (replicates) are formed by repeatedly omitting one half of the units in one of the replicate groups and calculating the desired statistic (replicate estimate). The number of replicate estimates is equal to the number of replicate groups. The variability among the replicate estimates is used to estimate the overall sampling variability.

jurisdiction. Any government-defined geographic area sampled in the NAEP assessment (e.g., a state, the District of Columbia, a United States territory, a Trends in Urban Districts Assessment (TUDA) district, the Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS), a subdivision within a state or county).

key teacher. The teacher or school staff member who is most knowledgeable about a student with a disability (SD) or English learner (EL).

Keyfitz process. A process used to control the overlap between two samples which share the same sampling frame (Sample 1 and Sample 2). To minimize overlap, Sample 1 is drawn first, and an adjustment is made to the measures of size of sampling units such that the conditional probability of selection is small or zero for selection for Sample 2 given that the unit was sampled in Sample 1. To maximize overlap, Sample 1 is drawn first, and an adjustment is made to the measures of size of sampling units such that the conditional probability of selection is large or one for selection for Sample 2 given that the unit was sampled in Sample 1. The unconditional probability of selection for each unit in Sample 2 (regardless of Sample 1) is as originally specified.

Knowledge and Skills Appropriate (KaSA). KaSA mathematics refers to special sections of mathematics questions administered in Puerto Rico and the nation that more precisely and reliably measured the full range of mathematical abilities.

large city. Territory inside an urbanized area and inside a principal city with population of 250,000 or more. NAEP uses large city as a comparison group for the Trends in Urban Districts Assessment (TUDA). In order to make comparisons between the TUDAs and large cities, the NAEP large city jurisdiction also includes those portions of the participating TUDA districts which fall outside of the city limits. Large city is not synonymous with the term inner city.

LASSO (Least Absolute Shrinkage and Selection Operator). LASSO is a type of regression that helps simplify models by reducing the number of predictors used. It adds a penalty term to the ordinary least square regression, encouraging some coefficients to shrink to zero, effectively limiting the number of predictors in the model. LASSO works especially well when there are multiple highly correlated predictors and is useful for automatically selecting the most important predictors.

limited English proficient (LEP). A term used to describe students who are in the process of acquiring English language skills and knowledge. Some schools refer to these students using the term English learners, or EL.

linking form. A group of items that are administered in order to put scores from different tests on the same scales. For example, a subsample of the short-form mathematics student sample was tested using a linking form created by combining one block from the main market-basket form with two main NAEP blocks for the purpose of putting the market-basket form on the same scale as the main NAEP mathematics assessment.

Local Assessment Administrator. Prior to the advent of NAEP administration of all assessment components, the state component was administered by state personnel. Local Assessment Administrators were school or district staff members appointed to attend training and administer the assessment.

locale. NAEP results are reported for four mutually exclusive categories of school location: city, suburb, town, and rural. The categories are based on standard definitions established by the Federal Office of Management and Budget using population and geographic information from the U.S. Census Bureau. Schools are assigned to these categories in the NCES Common Core of Data based on their physical address. The classification system was revised for 2007 and 2009; therefore, trend comparisons to previous years are not available. The new locale codes are based on an address's proximity to an urbanized area (a densely settled core with densely settled surrounding areas). This is a change from the original system based on metropolitan statistical areas. To distinguish the two systems, the new system is referred to as "urban-centric locale codes." The urban-centric locale code system classifies territory into four major types: city, suburban, town, and rural. Each type has three subcategories. For city and suburb, these are gradations of size: large, midsize, and small. Towns and rural areas are further distinguished by their distance from an urbanized area. They can be characterized as fringe, distant, or remote. To see the full description of the urban-centric and the metro-centric locale codes, visit the CCD website.

logistic regression model. A regression model for binary (dichotomous) outcomes. The data are assumed to follow binomial distributions with probabilities that depend on the independent variables.

longitudinal. A sample survey that follows the experiences and outcomes over time of a representative sample of respondents (i.e. a cohort) who are defined based on a shared experience (e.g. shared birth year or grade in school).

long-term trend. NAEP assessments that are designed to give information on the changes in the basic achievement of America's youth. They are administered nationally and report student performance at ages 9, 13, and 17 in mathematics and reading. Measuring trends of student achievement or change over time requires the precise replication of past procedures. Therefore, the long-term trend instrument does not evolve based on changes in curricula or in educational practices.

low density schools. Defined by the Office of Indian Education as those schools in which fewer than 25 percent of the students are American Indian or Alaska Native.

machine-readable catalog. Computer-processing control information, Item Response Theory (IRT) parameters, foil codes, and labels in a computer-readable format.

marginal correlation coefficient. An estimate of the Pearson product-moment correlations between subscale scores. See also conditional correlation.

marginal maximum likelihood methods. Statistical estimation methods where the estimate of a parameter is selected to be the estimate that gives the greatest probability to the outcome that was actually observed. These are often iterative methods.

market basket. A collection of test questions representative of some larger content domain; an easily understood index to summarize performance on the items. NAEP conducted a market-basket special study during the 2000 assessment.

matching criterion. The score that is used for determining which students in a study of differential item functioning are similar to one another.

matrix sampling. Sampling plan in which different samples of respondents take different samples of items.

maximum likelihood factor analysis. Factor analysis in which the parameters of the factor model are estimated using maximum-likelihood methods; maximizing the likelihood function is a common statistical approach to estimating parameters.

MB1. The national main market-basket form. A subsample of the short-form mathematics sample where students were tested on new items from the 1999 mathematics field tests. The test questions were grouped into three blocks.

MB2. The second national main market-basket form. A subsample of the short-form mathematics sample where students were tested on secure items from the 1996 main NAEP mathematics assessment (also used again in 2000 main mathematics assessment). The test questions were grouped into three blocks.

mean square error. A quantity indicating the degree to which survey estimates differ from the population values. Mean square error is the variance plus the squared bias.

measurement error. An error component or variability due to imprecision in the measurement of individual proficiencies. Because the NAEP design gives each student a small proportion of the pool of assessment items, the assessment cannot provide reliable information about individual performance.

metropolitan statistical area (MSA). A geographical MSA is an area with a large population nucleus and its adjacent communities that have a high degree of social and economic integration with that nucleus. MSAs are defined in terms of entire counties, except in six New England states where they are defined in terms of cities and towns. An MSA has a city of at least 50,000 population or an urbanized area of at least 50,000 with a total metropolitan population of at least 100,000 (or 75,000 in New England).

midlevel curriculum. One of the curriculum levels defined for the High School Transcript Study—at least four credits of English; three each of social studies, mathematics (which includes geometry and Algebra I or II), and science (which includes two subjects among biology, chemistry and physics); and one credit of foreign language.

minority school status. A measure of the level of historically disadvantaged minority student groups being served by schools participating in the High School Transcript Study. Low minority schools have less than 5 percent disadvantaged minority students. Medium minority schools have 5 to 50 percent disadvantaged minority students. High minority schools have over 50 percent disadvantaged minority students.

multiple matrix sampling. Sampling plan in which different samples of respondents take different samples of items.

multiple-choice item. An item that consists of one or more introductory sentences followed by a list of response options that include the correct answer and several incorrect alternatives.

multi-stage sample design. Indicates more than one stage of sampling. The following is an example of three-stage sampling: (1) sample of counties (primary sampling units or PSUs), (2) sample of schools within each sample county, and (3) sample of students within each sample school.

multi-stage testing (MST). an adaptive test in which students are given an online assessment which adapts to their ability level.

NAEP. The National Assessment of Educational Progress (NAEP), also known as "the Nation's Report Card," is the only nationally representative and continuing assessment of what America's students know and can do in various subject areas. Since 1969, assessments have been conducted periodically in mathematics, reading, science, writing, U.S. history, geography, civics, the arts, and other subjects.

NAEP achievement levels. Performance standards set by the National Assessment Governing Board that provide a context for interpreting student performance on NAEP, based on recommendations from panels of educators and members of the public. The levels, NAEP Basic, NAEP Proficient, and NAEP Advanced, measure what students should know and be able to do at each grade assessed. See each NAEP subject for a detailed description of what students should know and be able to do at each level at grade 4, 8, or 12.

NAEP Advanced. This level signifies superior performance beyond NAEP Proficient.

NAEP Basic. This level denotes partial mastery of prerequisite knowledge and skills that are fundamental for performance at the NAEP Proficient level.

NAEP Proficient. This level represents solid academic performance for each NAEP assessment. Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real world situations, and analytical skills appropriate to the subject matter.

NAEP region. See the entry for region for more information. For regions used for the National Indian Education Study (NIES), see the entry for region (NIES).

NAEP scales. The scales common across age or grade levels and assessment years used to report NAEP results.

NAEP State Coordinator (NSC). Staff member of participating state Department of Education who works with field staff to coordinate all NAEP activities in the state. Full-time federally funded coordinators are also responsible for coordinating NAEP activities in their state including promoting understanding of NAEP and coordinating assessment administrations.

NAEP TUDA Coordinator (NTC). Staff member of participating Trends in Urban Districts Assessment (TUDA) who works with field staff to coordinate all NAEP activities in the district. District coordinators usually work with a team of other district staff to coordinate NAEP activities in their district including promoting understanding of NAEP and coordinating assessment administrations.

NAIS. See National Association of Independent Schools.

National Assessment Governing Board. Independent organization whose members are appointed by the U.S. Secretary of Education. The Governing Board provides overall policy direction to the NAEP program. It is an independent, bipartisan group whose members include governors, state legislators, local and state school officials, educators, business representatives, and members of the general public.

National Association of Independent Schools (NAIS). A membership organization that represents nearly 1,200 U.S. independent schools, including day, boarding, and day/boarding schools; elementary and secondary schools; and boys', girls', and coeducational schools.

National Indian Educational Study (NIES). The national Indian Educational Study is a two-part study designed to describe the condition of education for American Indian/Alaska Native students in the United States.

national linking sample. Prior to 2002, separate state samples and national samples were drawn for the NAEP state and national assessments. NAEP state scales were linked to the national scales using a common population linking procedure in which the mean and standard deviation of the aggregate state sample was matched to the mean and standard deviation of a national linking sample. This national linking sample (NLS) contained the subset of national data that was representative of the students from jurisdictions participating in the state assessments. Beginning in 2002, a combined state and national sample has been drawn for NAEP state and national assessments, and the establishment of a national linking sample is therefore no longer necessary.

National Public. Students from public schools only. Includes charter schools; excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. To be used when comparing national data to those of states, urban districts or regions.

National School Lunch Program (NSLP). A federally assisted meal program that provides low-cost or free lunches to eligible students. It is sometimes referred to as the free/reduced-price lunch program. Free lunches are offered to those students whose family incomes are at or below 130 percent of the poverty level; reduced-price lunches are offered to those students whose family incomes are between 130 percent and 185 percent of the poverty level.

NCES Private School Survey (PSS). A survey of private school information collected by the National Center for Education Statistics (NCES). Enrollment grade span and other data for individual private schools was aggregated into data for use in sampling schools and in preliminary session allocation.

new enrollees. New enrollees are students that enrolled in a school after the original list of students was created and the original sample was subsequently drawn.

new school. A new school is a school selected from the new school sampling frame, created to update the NAEP school frame to account for newly constructed or newly eligible schools not on the original NAEP school frame.

noncertainty. Selected with a probability less than one.

noncertainty PSU. A primary sampling unit (PSU) that is selected with a probability less than one.

nonresponse. The failure to obtain responses or measurements for all sample elements.

nonresponse adjustment class. A set of units (e.g., schools or students) that are grouped together for the purpose of calculating nonresponse adjustments. The units are homogeneous with respect to certain unit characteristics, such as school size, location, public/private, student's age, sex, and student disability status.

nonresponse class level. A particular value of a unit characteristic that defines a nonresponse adjustment class.

nonsampling error. A general term applying to all sources of error, with the exception of sampling error. Includes errors from defects in the sampling frame, response or measurement errors, and mistakes in processing the data.

nonsectarian. Having no religious orientation or affiliation.

not-reached item. An item to which the student did not respond because the time limit was up for the section of the assessment on which s/he was working. After the first "not reached" item, the student will have no responses to any further questions on that section of the assessment.

objective. A desirable education goal accepted by scholars in the field, educators, and concerned laypersons and established through a consensus approach.

observed race/ethnicity. Race or ethnicity of an assessed student as perceived by the exercise administrator (EA).

off track students. Students who attend year-round schools, but are not in school at the time of the assessment. (In year-round schools, a certain percentage of students are "off" at any given time, on vacation, etc.)

off-task response. A response that is unrelated to the question being posed; differs from an incorrect response to the question or an omitted response.

omitted response. A missing response prior to the last observed response; this is considered an intentional behavior. This term is contrasted with the term not reached.

on the fly. Automatically generated in real time.

online bundle assignment and distribution system. A software program that uses the information from the quality-control scanning of the bundle barcodes, the supervisor list, and the distance from the distribution point to assign specific bundles to specific sessions/schools and supervisors for bulk distribution.

other academic courses. A course type defined for the High School Transcript Study, courses are fine arts, foreign languages, and computer-related studies.

other courses. A course type defined for the High School Transcript Study, other courses includes courses such as, vocational education, general skills, personal health and physical education, religion, military science and special education, etc.

oversampling. Deliberately sampling a portion of the population at a higher rate than the remainder of the population.

p+. The proportion of examinees who received a correct score on the item.

parental education. A NAEP reporting group defined by the highest level of education of the mother and father of an assessed student as derived from the student's response to two background questionnaire items.

Pearson. The materials and scoring contractor for NAEP.

Pearson product-moment correlation. A statistical index that quantifies the degree of relationship between the two scores.

percent correct. The percent of a target population that would answer a particular exercise correctly.

percentage of exact agreement. A quantitative index of the degree of decision consistency. It is used to describe the accuracy of raters scoring constructed-response items.

percentile. A score location below which a specified percentage of the population falls. For example, in 1998, the tenth percentile of fourth-grade reading scores was 167. This means that in 1998, ten percent of fourth-graders had NAEP reading scores below 167, while 90 percent scored at or above 167.

performance levels. Reported as the percentages of students attaining specific levels of performance corresponding to five points on the NAEP long-term trend reading and mathematics scales (150, 200, 250, 300, and 350). The specific descriptions for each level reflect the types of questions students performing at that level were more likely to answer correctly than students at lower levels. The five performance levels in each subject are applicable for 9-, 13-, and 17-year olds, however, the likelihood of attaining higher performance levels is related to a student’s age.

Phase Review System (PRS). A quality control method that requires the user to document the Business Plan, the Financial Plan, the Operations Plan, and the Risks of the program. The documents generated are reviewed by management at specific times in the program's lifecycle. It is program team focused.

pilot test. A pretest of items to obtain information regarding clarity, difficulty levels, timing, feasibility, and special administrative situations. The pilot test is performed before revising and selecting the items to be used in the assessment, or in the case of math and reading at grades 4 and 8, before selecting items to be used in the field test.