The Neyman-Pearson Lemma:

The Neyman-Pearson Lemma:

                   Was introduced by Jerzy Neyman and Egon Sharpe Pearson in a paper in 1933. It shows that the likelihood ratio test is the most powerful test, among all possible statistical tests.

Likelihood ratio tests are useful to test a composite null hypothesis against a composite alternative hypothesis.

Neyman-Pearson Lemma.

                    It is used for testing a statistical hypothesis to test whether they performed test is the most powerful test about the population parameter with the consideration of the supposed probability distribution.

It allows seeing whether the rejection region which has been selected is the best one or not. It helps to assess the statistical power of the hypothesis test. The statistical power of the hypothesis test states that the null hypothesis has been correctly rejected in favor of the alternative hypothesis.

A test with the highest power of all the tests for the same level of significance is called the most powerful test. Suppose if the results of the observations are used to test the null hypothesis as against the simple alternative hypothesis, the error arises from the rejection of null hypothesis being verified, as per a statistical test formulated to test a null hypothesis against the alternative hypothesis if the null hypothesis is actually true.

The most powerful tests are constructed by Neyman-Pearson Lemma. As per this, the most powerful test is the likelihood-ratio. A test proposed for testing the simple null hypothesis against a simple alternative hypothesis which offers the least probability of error among all the tests is the most powerful test. As the statistical test power is obtained by subtracting the probability of a type 2 error by one, the most powerful test is formulated in terms of probabilities of errors of type 1 and type 2 errors.

Notation:

   If  is a random sample of size n from a distribution with probability density (or Mass) function f(x; /theta) then the joint probability density  (or Mass) function of  is denoted by the likelihood function L(Ɵ). That is, the joint p.d.f or p.m.f. is:

L (Ɵ) = L (Ɵ;) = f (; Ɵ)

Note that for the sake of ease, we drop the reference to the sample  in using L(Ɵ) as the notation for the likelihood function. We’ll want to keep in mind though that the likelihood L(Ɵ) still depends on the sample data.

Example:01

       Suppose  is a random sample from an exponential distribution with parameter Ɵ. Is the hypothesis H: Ɵ = 3 a simple or a composite hypothesis ?

                 Answer:

The p.d.f of an exponential random variable is:

                        f (x) =

For x≥0. Under the hypothesis H: Ɵ = 3, the p.d.f. of an exponential random variable is:

                               f (x) =

For x≥0. Because we can uniquely specify the p.d.f. under the hypothesis H: Ɵ = 3, the hypothesis is a simple hypothesis.

Example:02

        Suppose  is a random sample from an exponential distribution with parameter Ɵ, is the hypothesis H: Ɵ˃2 a simple or a composite hypothesis?

             Answer:

Again, the p.d.f of an exponential random variable is:

                         f (x) =  for x≥0. Under the hypothesis H: Ɵ˃2, the p.d.f of an exponential random variable could be:

                             f (x) =  for x≥0 or, the p.d.f. could be:

                  f (x) =

for x≥0. The p.d.f. could, in fact, be any of an infinite number of possible exponential probability density functions. Because the p.d.f is not uniquely specified under the hypothesis H: Ɵ˃2, the hypothesis is a composite hypothesis.

Neyman-Pearson Lemma:

           Suppose  is a random sample from a probability distribution with parameter Ɵ. Then, if C Is a critical region of size α and k is a constant such that:

                     ≤ k inside the critical region C.

And:

                                            ≥ k outside the critical region C.

Then C is the best, that is, most powerful, critical region for testing the simple null hypothesis Ho: Ɵ =  against the simple alternative hypothesis: Ɵ =  .

Neyman-Pearson Lemma.

Theorem:

The likelihood ratio test for a simple null hypothesis:  versus a simple alternative hypothesis:  is a most powerful test.

Definition:

A uniformly most powerful test in testing a simple null hypothesis:  versus a composite alternative hypothesis:  or in testing a composite null hypothesis:  versus a composite alternative:, is a size  test such that this test has the largest possible test power  ) for every simple alternative hypothesis: :  , among  all such tests of size .

Power Function of the test.

                               δ (.):  (i.e. probability of rejection the null when truth is ) (so, if  is the null, then we call it type I error, if  is alternative then it’s power  of test).
Intuition: The power function says, based on this test, what is the probability that the test will “reject” if the truth is at .
If: θ =  , then  is the probability of a Type I error
If :θ =  , then    is the probability of rejecting correctly

Power function, Size of Test, and Power of Test:
1. define the null and alternative hypotheses and form a test. Test is: Reject the null if T(X) > c and do not reject the null if T (X) < c, Based on this test, we calculate a power function.
2. To find size of the test, see the largest value of the power function over the range of θ ∈Θ0
3. To find power of the test for a particular alternative, find  (i.e. what is the probability of rejecting the null correctly when the truth is∈Θ K.

NOTATION:

The Neyman-Pearson Lemma test is quite limited it can be used only for testing a simple null versus a simple alternative. So it does not get used in practice very often. But it is important from a conceptual point of view.

The Likelihood Ratio Test (LRT).

This test is simple reject  if () ≤ C

Where,

                              () =

Where,  maximizes L(Ɵ) subject to Ɵ ε.

The Neyman-Pearson Lemma:

 asserts that, in general a best critical region can be found by finding the n-dimensional points in the sample space for which the likelihood ratio is smaller than some constant.

Consider a hypothesis test between two point hypotheses: . The uniformly most powerful (UMP) test has a rejection region defined by:

                    W =

Where,  denotes the likelihood of the sample x and K is a constant determined by the size α such that:

                      Pr ( = α

Example:

         Find the most powerful test for testing.

 Has a U (0, 1) distribution

                      Vs

: X has an exp (1) distribution.

f (x/) = (x)              f (x/) = (x)

L (x/) =

For any k  iff x or x  = c. suppose we want the test to have specified size α.

                   α = (X

                       =(X) = C             (0).

Take c = α. Test becomes

          Ɵ(x) =

Power is

     (X =  = 1-.

For α = 0.01, 0.05, and 0.10, the respective powers are 0.378, 0.417 and 0.463.

The likelihood ratio test (LRT).

                          Is any test that has a rejection region of the form: {x:  ≤ c} where c is any number satisfying 0 ≤ c ≤ 1. The rationale behind LRTs may best be understood in the situation in which f(x/Ɵ) is the probability distribution of a discrete random variable. In this case, the numerator of ⋋(x) is the maximum probability of the observed sample over all possible parameters.

The ratio of these two maxima is small if there is a parameter point in  for which the observed sample is much more likely than for any parameter in  in this situation, the LRT criterion says  should be rejected and  accepted as true.

Consider testing:: Ɵ ≤ versus  where  is a value specified. The likelihood function is an increasing function of Ɵ on -.

The likelihood ratio test statistics is

              ⋋(x) =