5.4 Comparison of Several MV Means (wk5)

5.4.1 Paired Comparison

Recall:

for univariate, let $X_i - Y_i = D_i \sim N(\delta, \sigma_d^2)$, $i=1, \cdots, n$

then for $H_0 : \delta = 0$, test stat $t = \tfrac{\bar D}{\tfrac{S_d}{\sqrt{n}}} \overset {H_0}{\sim} t_{n-1}$.

Assume independent rvec $\pmb D_1 , \cdots, \pmb D_n \sim N_p (\pmb \delta , \Sigma_{\pmb d})$.

then test stat $T^2 = n(\bar {\pmb D} - \pmb \delta)' S^{-1}_{\pmb d} (\bar {\pmb D} - \pmb \delta) \sim (n-1)\tfrac{p}{n-p} F_{p, n-p}$.

Hypothesis Testing:

$ H_0 : = $

T^2 = n({D} )’ S^{-1}{d} ({D} ) F{p, n-p}

reject $H_0$ if $T^2 > \tfrac{(n-1)p}{n-p} F_{p, n-p} (\alpha)$.

$100(1-) % $ CR for $\pmb \delta$:

({D} - )’ S^{-1}{d} ({D} - ) F{p, n-p} ()

$100(1-) % $ simultaneous CI for individual $\delta_i$:
Bonferroni’s $100(1-) % $ simultaneous CI for individual $\delta_i$:

$ \[\begin{alignat*}{3} \bar d_i \pm &\sqrt{\tfrac{(n-1)p}{n-p} F_{p, n-p} (\alpha)} &\sqrt{\tfrac{S^2_{d_i}}{n}} \tag{2} \\ \bar d_i \pm &t_{n-1} \left( \tfrac {\alpha} {2p} \right) &\sqrt{\tfrac{S^2_{d_i}}{n}}\tag{3} \end{alignat*}\] $

–

====

5.4.1.0.1 Different Approach

let $\pmb X = \left[ x_{11}, \cdots, x_{1p}, x_{21}, \cdots, x_{2p} \right]_{1 \times 2p}' \sim N_{2p}(\pmb \mu, \Sigma)$.

then $\pmb D = C \pmb X$, where $C = ( \[\begin{matrix} 1 & & \pmb 0 & \vdots & -1 & & \pmb 0 \\ & \ddots & & \vdots & & \ddots & \\ \pmb 0 & & 1 & \vdots & \pmb 0 & & -1 \end{matrix}\]

)_{p 2p} $.

at here,

$ \[\begin{align*} E(\pmb D) &= E(C \pmb X) = C \pmb \mu \\ &= \pmb \delta\\ \\ Cov(\pmb D) &= Cov(C \pmb X) = C \Sigma C' \\ &= \Sigma_d\\ \\ \pmb D &= C \pmb X \sim N_p (C \pmb \mu, C \Sigma C') \end{align*}\] $

therefore, given $H_0 : C \pmb \mu = \pmb 0$,

test stat $T^2 = n (C \bar {\pmb X})' (CSC')^{-1} (C \bar {\pmb X}) \overset {H_0}{\sim} \tfrac{(n-1)p}{n-p} F_{p, n-p}$

graph, check normality:

5.4.2 Comparing Mean Vectors from Two Populations

Recall: univariate, $ t = {sqrt{S_p^2 ( + )}} t_{n_1 + n_2 - 2}$

for MV, assume below, where $(\pmb X_{11}, \cdots, \pmb X_{1n_1})$ and $(\pmb X_{21}, \cdots, \pmb X_{2n_2})$ are independent.

$ X_{11}, , X_{1n_1} N_p (_1 , _1 )

$ $

X_{21}, , X_{2n_2} N_p (_2 , _2 ) $

at here,

$ H_0 : _1 - _2 = $

5.4.2.0.1 case 1: $ _1 = _2 = $

이하 대부분은 벡터에 관한 이야기이다.

$\bar X_i$ estimates $\mu_i$, $i=1,2$.

$S_p$ estimates $\Sigma$, where $S_p = {(n_1-1) + (n_2-1)} $.

the test stats Hotelling’s $ T^2 = ( {X_1 } - {X_2} ) ’ S_p^{-1} ( {X_1 } - {X_2} ) $

where $ {p [ (n_1 - 1) + (n_2 - 1) ]} ; T^2 = {p [ (n_1 + n_2 - 2) ]} ; T^2 F_{p, n_1 + n_2 -p - 1}$. (p.285 for pf)

CR for $\mu_1 - \mu_2$ will be

Pr = 1-

where $ c^2= {n_1 + n_2 - p - 1} ; T^2 F_{p, n_1 + n_2 -p - 1} ()$. * 이때 constant가 역수가 되었음을 눈치. * The equality will define the boundary of a region. * The region is an ellipsoid centered at $(\bar X_1 - \bar X_2)$.

5.4.2.0.1.1 Example) Testing $H_0 : \mu_1 - \mu_2 = 0$ at $\alpha=0.05$ is equivalent to see whether falls within the confidence region

Axes of the confidence region * let $\lambda_1 , \cdots, \lambda_p$ are ev of $S_p$. * let $e_1 , \cdots, e_p$ are evc of $S_p$.
- then $e_i$’s are the direction of CI
- $ $are the half-length of the CR Link

let $ c^2= {n_1 + n_2 - p - 1} ; T^2 F_{p, n_1 + n_2 -p - 1} ()$.

$100(1-\alpha)%$ simultaneous CI for $a'(\mu_1 - \mu_2)$, $\forall a$:
$

a’ ( X_1 - X_2 ) c

5.4.2.0.1.2 Example) simultaneous CI for $(\mu_{1i} - \mu_{2i}), i=1, \cdots, p$.

let $a' = \left[0, \cdots, 0, 1, 0, \cdots, 0 \right]$. 이때 $a'$가 하나만 1이고 나머지 0이면, 어떤 특별한 한 axis로 proj하라는 의미. link

let $\mu_1 - \mu_2 = \left[ \mu_{1i} - \mu_{2i} \right]_{i=1,\cdots,p}$.

$ a’(X_1 - X_2) = X_{1i} - X_{2i}$, $a' \left( \dfrac {1}{n_1} + \dfrac {1}{n_2} \right) S_p a = \left( \dfrac {1}{n_1} + \dfrac {1}{n_2} \right) S_{p \; ii}$ * $S_{p \; ii}$ : p번째 변수의 표본 cov. 이는 단변량에서 나왔던 공통 cov, 즉 샘플 se와 표기법이 동일해지며 유사하다. (ch1) link

the Bonferroni’s $100(1-)% $ simultaneous CI for $(\mu_{1i} - \mu_{2i})$ is $ (X_1 - X_2) t_{n_2 + n_2 -2, ()} $.

5.4.2.0.2 case 2: $ _1 = _2 $

assume $n_1 - p , \; n_2 - p$ are large.

for $H_0 : \mu_1 - \mu_2 = 0$, test stat becomes $T^2 = (\bar X_1 - \bar X_2 )' \left[ \dfrac{1}{n_1} S_1 + \dfrac {1}{n_2} S_2 \right]^{-1} (\bar X_1 - \bar X_2 ) \overset{H_0}{\sim} \chi_p^2$.


$
E(\bar X_1 - \bar X_2 ) = \mu_1 - \mu_2
$

$
Cov(\bar X_1 - \bar X_2 ) = Cov(\bar X_1) + Cov(\bar X_2 ) - 2 Cov(\bar X_1, \bar X_2 ) = \dfrac{1}{n_1} \Sigma_1 + \dfrac {1}{n_2} \Sigma_2 - 0
$


$
\bar X_1 - \bar X_2 \overset{\cdot}{\sim} N_p \left( \mu_1 - \mu_2, \dfrac{1}{n_1} \Sigma_1 + \dfrac {1}{n_2} \Sigma_2  \right)
\tag{∵ CLT}
$

$\\[3ex]
$

$
\text{under } H_0, 
$

$
S_1 \overset{p}{\to} \Sigma_1, S_2 \overset{p}{\to} \Sigma_2 \tag{∵ WLLN}
$

$
(\bar X_1 - \bar X_2 )' \left[ \dfrac{1}{n_1} S_1 + \dfrac {1}{n_2} S_2\right]^-1 (\bar X_1 - \bar X_2 ) \overset{app}{\sim} \chi_p^2 \tag{∵ Slutsky's thm}
$

why Cov become 0???

i.e. reject $H_0$ if $T^2 > \chi_p^2 (\alpha)$.

CI becomes

Pr = 1-

차이는~~

Remark: if $n_1 = n_2 = 2$,

$ \[\begin{align*} \dfrac{1}{n_1} S_1 + \dfrac{1}{n_2} S_2 &= \dfrac{1}{n} (S_1 + S_2) \\ &= \dfrac{1}{n} \left[ \dfrac{1}{n-1} \sum_{n=1}^n (\pmb X_{1i} - \bar {\pmb X_1})(\pmb X_{1i} - \bar {\pmb X_1})' + \dfrac{1}{n-1} \sum_{n=1}^n (\pmb X_{2i} - \bar {\pmb X_2})(\pmb X_{2i} - \bar {\pmb X_2})' \right] \\ &= \dfrac{1}{n} \dfrac{1}{n-1} S_p \ast 2(n-1) = \dfrac{2}{n} S_p \end{align*}\] $

i.e. case 1 and case 2 are the same procedure when the sample sizes are the same for large sample sizes.

$100(1-\alpha)%$ simultaneous CI for $\pmb a'(\pmb \mu_1 - \pmb \mu_2)$, $\forall \pmb a$:

a’ ( {X_1} - {X_2} )

5.4.2.0.3 Other Statistics for Testing two Mean Vectors

let $W=(n_1-1)S_1 + (n_2-1)S_2$: within SS, $B=n_1 (\bar {\pmb X_1} - \bar {\pmb X})(\bar {\pmb X_1} - \bar {\pmb X})' + n_2 (\bar {\pmb X_2} - \bar {\pmb X})(\bar {\pmb X_2} - \bar {\pmb X})'$
Wilk’s Lambda:
- when two-sample procedure, Hotelling’s $T^2$

$ ^= $

Lawley-Hotelling’s Trace:

$ tr(BW^{-1}) $

Pillai Trace:

$ tr $

Roy’s Largest Root:
- maximum ev of $B(B+W)^{-1}$.

5.4.2.0.4 Testing Equality of Covariance Matrices

$ H_0 : _1 = _2 $

let $S_p = \dfrac{1}{n_1 + n_2 - 2} \left[ (n_1 - 1) S_1 + (n_2 - 1) S_2 \right]$.

$ \[\begin{align*} M &= (n_1 + n_2 - 2) \ln \vert S_p \vert - (n_1 - 1) \ln \vert S_1 \vert - (n_2 - 1) \ln \vert S_2 \vert \tag{test stat} \\ C^{-1} &= 1 - \dfrac{2p^2 + 3p -1}{6(p+1)} \left( \dfrac {n_1 + n_2 - 2}{(n_1-1)(n_2 - 1)} - \dfrac {1}{n_1 + n_2 - 2} \tag{Scale Factor} \\ MC^{-1} &\sim \chi_v^2, \; \; \; \; \; v=\dfrac{p(p+1)}{2} \end{align*}\] $

reject $H_0$ if $MC^{-1} > \chi_v^2(\alpha)$

5.4.3 Profile Analysis (for $g=2$)

Recall:

$H_0: \pmb \mu_1 = \pmb \mu_2$, when $\Sigma_1 = \Sigma_2 = \Sigma$

$ \[\begin{align*} T^2 &= (\bar {\pmb X_1} - \bar {\pmb X_2})' \left[ \left( \tfrac{1}{n_1} + \tfrac{1}{n_2} \right) S_p \right]^{-1} (\bar {\pmb X_1} - \bar {\pmb X_2}) \\ &\overset {H_0} {\sim} \tfrac {(n_1 + n_2 -2)p} {n_1 + n_2-p-1} F_{p, \; \; n_1 + n_2 -p -1} \end{align*}\] $

let’s $H_0: C \pmb \mu_1 = C \pmb \mu_2$, when $\Sigma_1 = \Sigma_2 = \Sigma$, where $C_{q \times p}$, $q \le p$ and $rank(C)=q$.

$ \[\begin{align*} T^2 &= (\bar {\pmb X_1} - \bar {\pmb X_2})' C' \left[ \left( \tfrac{1}{n_1} + \tfrac{1}{n_2} \right) CS_p C'\right]^{-1} C(\bar {\pmb X_1} - \bar {\pmb X_2}) \\ &\overset {H_0} {\sim} \tfrac {(n_1 + n_2 -2)q} {n_1 + n_2-q-1} F_{p, \; \; n_1 + n_2 -p -1} \end{align*}\] $

Profiles are constructed for each group.

Consider two groups. Questions:

Are the profiles parallel?

$ \[\begin{alignat*}{3} &&H_0 : \mu_{11}-\mu{12} = \mu_{21}-\mu{22}, \mu_{12}-\mu{13} = \mu_{22}-\mu{23}, \mu_{13}-\mu{14} = \mu_{23}-\mu{24}, \cdots, \mu_{1,p-1}-\mu{1,p} = \mu_{2,p-1}-\mu{2,p} \\ &\iff & H_0 : \mu_{11}-\mu{21} = \mu_{12}-\mu{22} = \cdots = \mu_{1p}-\mu{2p}} \\ &\iff C_{(p-1) \times p} &H_0: C \pmb \mu_1 = C \pmb \mu_2 \end{alignat*}\] $

This is equivalent to test the equal mean vector of the transformed data $C \pmb X_1$ and $C \pmb X_2$.

Populations 1: $C \pmb X_{11}, \cdots, C \pmb X_{1n_1} \sim N_{p-1} (C \pmb \mu_1 , C \Sigma C')$ Populations 2: $C \pmb X_{21}, \cdots, C \pmb X_{2n_2} \sim N_{p-1} (C \pmb \mu_2 , C \Sigma C')$

reject $H_0: C \pmb \mu_1 = C \pmb \mu_2$ (i.e. paralle profiles), if $

T^2 = ({X_1} - {X_2})‘C’ ^{-1} C({X_1} - {X_2}) > d^2 = (n_1 + n_2 - 2) F_{p-1,n_1+n_2-p} ()

5.4.3.0.1 2. Coincident Profiles

Assuming that the profiles are parallel, are the profiles coincident?

$ \[\begin{align*} &H_0 : \mu_{1i} = \mu_{2i}, i=1, \cdots, p \\ \iff & H_0 : \pmb 1 ' \pmb \mu_1 = \pmb 1 ' \pmb \mu_2 \end{align*}\] $

is the case where $C$ is replaced by $\pmb 1 '$.

reject $H_0$ if

$ \[\begin{alignat*}{2} T^2 &= \pmb 1 ' (\bar {\pmb X_1} - \bar {\pmb X_2}) \left[ \left(\dfrac{1}{n_1} + \dfrac{1}{n_2} \right) \pmb 1 ' S_p \pmb 1 \right]^{-1} (\bar {\pmb X_1} - \bar {\pmb X_2}) && \\ &= \left( \dfrac{\pmb 1 ' (\bar {\pmb X_1} - \bar {\pmb X_2})}{\sqrt{\left(\dfrac{1}{n_1} + \dfrac{1}{n_2} \right) \pmb 1 ' S_p \pmb 1}} \right)^2 &&> F_{1, n_1 + n_2 -2} (\alpha) (n_1 + n_2 - 2) \dfrac{p-1}{n_1 + n_2 - p } F_{p-1,n_1+n_2-p} (\alpha) \end{alignat*}\] $

5.4.3.0.2 3. Flat Profiles

3.Assuming that the profiles are coincident, are the profiles level?

$ H_0 : {11} = {12} = } = {1p} = {21} = {22} = } = {2p} $

by 1 and 2, we can collapse two groups into one.

$ X_{11}, , X_{1n_1}, X_{21}, , X_{2n_2} N_p (, ) $

this is one population problem

$ C_{(p-1) p}, H_0: C = 0 $

reject $H_0$, iff

$ T^2 = (n_1+n_2) {X}‘C’ [CSC’]^{-1} C {X} > d^2 = (n_1 + n_2 - 1) F_{p-1,n_1+n_2-p+1} () $

이는 1번에서의 그것과는 $F$분포의 df가 변화했다는 점에 주목. - $\bar {\pmb X} = \tfrac{1}{n_1 + n_2} \left( \sum_{j=1}^{n_1} \pmb X_{1j}+ \sum_{j=1}^{n_2} \pmb X_{2j} right)$. - $S = n_1 + n_2$ sample covariance matrix, using data.

5.4.4 Comparing Several Multivariate Population Means

Recall:

In univariate, two-sample t-test is extended to Analysis of Variance(ANOVA).

$ H_0:_1 = =_g $

$ F^= {SSE/df_2} F_{df_1 , df_2} $ - where - SSR: sum of squared regression, - SSE: sum of squared error, - SST: sum of squared total - $df_1 = g-1, df_2 = N-g, N=\sum_{i=1}^g n_i$.

Assume $g$ population or treatment groups, and each groups are independent. 각 population은 같은 Cov를 갖고 같은 숫자의 패러미터를 갖되 총 observation 숫자랑 각각의 population mean은 다름.

Population 1~g: $\pmb X_{i1}, \cdots, \pmb X_{in_i} \sim N_p(\pmb \mu_i , \Sigma)$.

Model

$ X_{ij} = {i} + {ij}, ; ; ; ; ; i=1, , g, ; ; j = 1, , n_i

$ H_0: _1 = _g $

$ X_{ij} = \[\begin{bmatrix} X_{ij1} \\ X_{ij2} \\ \vdots \\X_{ijp} \end{bmatrix}\] {p } , {ij} = \[\begin{bmatrix} \mu_{i1} \\ \mu_{i2} \\ \vdots \\ \mu_{ip} \end{bmatrix}\] {p }, {ij} = \[\begin{bmatrix} \epsilon_{ij1} \\ \epsilon_{ij2} \\ \vdots \\ \epsilon_{ijp} \end{bmatrix}\]

_{p } $

Assumptions
1. The random samples from different populations are independent.
2. All populations have a common covariance matrix $\Sigma$.
3. Each population is Multivariate Normal. This assumption can be relaxed by C.L.T., when the sample sizes $n_1 , \cdots, n_g$ are large.

5.4.4.0.1 One-Way MANOVA

The quantities SSR, SSE and SST become matrices in MANOVA.

$ \[\begin{align*} B &= \sum_{i=1}^g n_i (\pmb X_i - \pmb X) (\pmb X_i - \pmb X)' \tag{SSR} \\ W &= \sum_{i=1}^g \sum_{j=1}^{n_i} (\pmb X_{ij} - \pmb X_i) (\pmb X_{ij} - \pmb X_i)' \\ &= (n_1 -1)S_1 + \cdots + (n_g -1)S_g \tag{SSE} \end{align*}\] $

Note:

$ \[\begin{alignat*}{3} (\pmb X_{ij} - \bar {\pmb X}) &= (\bar {\pmb X_i} - \bar {\pmb X}) + (\pmb X_{ij} - \bar {\pmb X_i})&& \\ (\pmb X_{ij} - \bar {\pmb X}) (\pmb X_{ij} - \bar {\pmb X}) ' &= (\bar {\pmb X_i} - \bar {\pmb X}) (\bar {\pmb X_i} - \bar {\pmb X}) ' + &&(\bar {\pmb X_i} - \bar {\pmb X}) (\pmb X_{ij} - \bar {\pmb X_i})' + (\pmb X_{ij} - \bar {\pmb X_i}) (\bar {\pmb X_i} - \bar {\pmb X}) ' + (\pmb X_{ij} - \bar {\pmb X_i})(\pmb X_{ij} - \bar {\pmb X_i})' \\ \sum_{i=1}^g \sum_{j=1}^{n_i} (\pmb X_{ij} - \bar {\pmb X}) (\pmb X_{ij} - \bar {\pmb X}) ' &= \sum_{i=1}^g n_i (\bar {\pmb X_i} - \bar {\pmb X}) (\bar {\pmb X_i} - \bar {\pmb X}) ' &&+ \sum_{i=1}^g \sum_{j=1}^{n_i} (\pmb X_{ij} - \bar {\pmb X_i})(\pmb X_{ij} - \bar {\pmb X_i})' \\ T &= B &&+ W \end{alignat*}\] $

B: Between Sum of Squares
W: Within Sum of Squares

Any test statistic will be a function of B and W. Popular test statistics use eigenvalues of $BW^{-1}$.

let $\lambda_1, \cdots, \lambda_r$ be ev of $BW^{-1}$, where $r=$ ## of non-zero ev’s.

Wilk’s Lambda (LRT)

$ = = = _{i=1}^r (1+_1)^{-1} $

Pillai’s Trace

$ \[\begin{align*} V &= tr[B(B+W)^{-1}] = tr[B(B(I+B^{-1}W))^{-1}] = tr[B(I+B^{-1}W)^{-1}B^{-1}] \\ &=tr[B^{-1}B(I+B^{-1}W)^{-1}] = tr[(I+B^{-1}W)^{-1}] = tr[I+(B^{-1}W)^{-1}]\\ &=\sum_{i=1}^r \left( \dfrac{\lambda_i}{1+\lambda_i}\right) \end{align*}\] $

Lawley-Hotelling’s Trace

$ T = tr(BW^{-1}) = _{i=1}^r _i $

Roy’s Largest Root

$ U = _{i=1,,r} { _i } $

Sampling Distribution of Wilk’s Lambda

$ \[\begin{alignat*}{2} p=1, g \ge 2: &\left(\dfrac{\sum_{i=1}^g n_i - g}{g-1}\right) \left(\dfrac{1-\Lambda^\ast}{\Lambda^\ast}\right) &&\overset{H_0}{\sim} F_{g, \sum_{i=1}^g n_i - g} \\ p=2, g \ge 2: &\left(\dfrac{\sum_{i=1}^g n_i - g-1}{g-1}\right) \left(\dfrac{1-\sqrt{\Lambda^\ast}}{\sqrt{\Lambda^\ast}}\right) &&\overset{H_0}{\sim} F_{2(g-1), 2(\sum_{i=1}^g n_i - g-1)} \\ p\ge1, g = 2: &\left(\dfrac{n_1 + n_2 - p -1}{p}\right) \left(\dfrac{1-\Lambda^\ast}{\Lambda^\ast}\right) &&\overset{H_0}{\sim} F_{p, n_1 + n_2 - p -1} \\ p \ge 1, g \ge 3: &\left(\dfrac{\sum_{i=1}^3 n_i - p-2}{p}\right) \left(\dfrac{1-\sqrt{\Lambda^\ast}}{\sqrt{\Lambda^\ast}}\right) &&\overset{H_0}{\sim} F_{2p, 2(\sum_{i=1}^g n_i - p-2)} \\ \text{large sample sizes}: &- \left( \sum_{i=1}^g n_i -1 -\dfrac{p+q}{2}\right) \ln \Lambda^\ast &&\overset{H_0}{\sim} \chi^2_{p(g-1)} \tag{Why?} \end{alignat*}\] $

Self-Study