Item response theory procedures for shortening tests

Ottavia M. Epifania & Friends

The 2-Parameter Logistic Model

Item Response Function:

\[P(x_{pi} = 1|\theta_p, b_i, a_i) = \frac{\exp[a_i(\theta_p - b_i)])}{1 + \exp[a_i(\theta_p - b_i)]}\]

Item Information Function:

\[IIF_i = a_i^2P(\theta, b_i, a_i)[1-P(\theta, b_i, a_i)]\]


Test Information Function:

\[TIF = \sum_{i = 1}^{||B||} IIF_i\]

Intuitively

Some premises

Key concept

TIF target (\(\mathbf{TIF}^*\)) describing the desired characteristics of a test

The aim of the algorithms

Minimize the distance between \(\mathbf{TIF}^*\) and that of the short test forms (STFs)

Their differences

The method for selecting and including the items in \(Q \subset B\) from the item bank

Mean TIF

TIF is considered as mean TIF \(\rightarrow\) as the number of items increases, the TIF increases

Bruto (tu quoque?!)

The bad

\(\forall Q \in\mathcal{Q} = 2^B \setminus \{\emptyset, B\}\),

  1. \(\mathbf{TIF}^{Q} = \frac{\sum_{i \in Q} IIF_i}{||Q||}\)
  2. \(\overline{\Delta}_{\mathbf{TIF}^{Q}} = \mathit{mean}(|\mathbf{TIF}^* - \mathbf{TIF}^{Q}|)\)

\(Q_{bruto} = \arg \min_{Q \in \mathcal{Q}} \overline{\Delta}_{\mathbf{TIF}^{Q}}\)

Item Locating Algorithm – ILA

The ugly

\(B\): Item bank

\(Q^k \subset B\): Set of item selected for inclusion in the STF up to iteration \(k\) (\(Q^0 = \emptyset\))

\(\mathbf{TIF}^*\): TIF target

\(TIF^k = \frac{\sum_{i\in Q^k} IIF_i}{||Q^k||}\), where \(||Q^k||\) denotes the cardinality of \(Q^k\), \(\mathbf{TIF}^0 = (0, 0, \ldots, 0)\)

Frank1

The good

At \(k =0\), \(\mathbf{TIF}^0 = (0, 0, \ldots, 0)\), \(Q^0 = \emptyset\), iterate

  1. \(A^k = B \setminus Q^k\)

  2. \(\forall i \in A^k\), \(\mathbf{PIF}_{i}^k = \frac{\mathbf{TIF}^k + \mathbf{IIF}_{i}}{||Q^k||+1}\)

  3. \(D = \arg \min_{i \in A^k} |\mathbf{TIF}^* - \mathbf{PIF}_i^k|\)

Termination criterion: \(|\mathbf{TIF}^* - \mathbf{PIF}_D^{k}| \geq |\mathbf{TIF}^* - \mathbf{TIF}^{k-1}|\):

  • If false, \(k = k + 1,\) \(Q^{k+1} = Q^k \cup \{D\}\), restart from 1

  • If true, stops, \(Q_{Frank} = Q^k\)

Simulation time

100 data frames

  1. Generate an item bank \(B\) of \(6\) items:

    • Difficulty parameters: \(\mathcal{U}(-3, 3)\)

    • Discrimination parameters: \(\mathcal{U}(.90, 2.0)\)

  2. Random item selections of lengths \(l\) from \(B\) (\(M_l = 3.34 \pm 1.13\)) + modification parameters \(\mathcal{U}(-0.20, 0.20)\) \(\rightarrow\) \(\mathbf{TIF}^*\)

  3. Considering \(\mathbf{TIF}^*\) at Step 2 and item parameters at Step 1:

    • Bruto \(\rightarrow\) Systematically tests

    • ILA \(\rightarrow\) Forwardly searches considering a single \(\theta\)

    • Frank \(\rightarrow\) Forwardly searches considering the whole latent trait

Results

When \(||Q_{ILA}|| = ||Q_{Frank}|| = ||Q_{Bruto}||\)

But \(Q_{Ila} \neq Q_{frank} \neq Q_{bruto}\)

In the end


Mathematically, we are at loss

Psychologically? I don’t know

Do they work? Hopefully

Acknowfndabfjknc

Pasquale Anselmi, Egidio Robusto, Livio Finos, Gianmarco Altoè


This happened too

Roses are red, violets are blue,

my computer broke down on these simulations,

and I broke down too

Psicostat meeting