Thursday, January 05, 2012

The first PISA results for India: The end of the beginning

by Lant Pritchett.

     Now this is not the end. It is not even the beginning
of the end. But it is, perhaps, the end of the beginning.

Winston Churchill, November 1942

The PISA 2009+ results are the end of the beginning. For the last decade there has been a debate. Some argued the levels of learning inside Indian elementary schools (primary and upper primary) are a national scandal and a threat to the future of India's society, polity, and economy. Others appeared to believe that the main, if not only, problem with Indian schools was that not enough children attend them and that with more money and more of the same, all would be well. The last five years saw a relentless accumulation of evidence about the crisis of learning. The establishment has tried to deny, deflect, and dismiss the evidence on learning. Eventually the Government of India agreed to participate in the PISA (Programme for International Student Assessment) - but only for two states, Tamil Nadu and Himachal Pradesh - and both sides agreed PISA was the litmus test. The PISA 2009+ results, which are both official and are beyond gain-saying are unspeakably bad. They confirm the worst of what anyone has been saying about the levels of learning in India elementary education.

  • In reading of the 74 regions participating in PISA 2009 or 2009+ these two states beat out only Kyrgyzstan.
  • In mathematics of the 74 regions participating the two states finished again, second and third to last, again beating only Kyrgyzstan.
  • In science the results were even worse, Himachal Pradesh came in dead last, behind Kyrgyzstan, while Tamil Nadu inched ahead to finish 72nd of 74.

But just coming in last (if we can dismiss as a relevant comparator for India a tiny Central Asian state) does not convey the enormity of how bad these results were, as not only was India last, it was far, far, behind its aspirations, both at the bottom and at the top levels of performance.

PISA expresses the levels of performance in two ways, an overall index number and the fraction of students achieving various "levels" of achievement. The PISA index numbers for each subject are scaled so that the typical OECD student is at 500 and the standard deviation across OECD students is 100. The testing of thousands of students allows the results to present not only the average but also the worst (5th percentile) and best (95th percentile) students do in each country/region. PISA also classifies student performance into "levels" that represent different degrees of mastery of the material.

Table 1 compares India's performance to three groups of countries. The economic superstars have successfully completed the transition from poor to rich economies in just two generations - Singapore, Hong Kong, Korea (China's only results are just for the city of Shanghai, which are the highest scores of any region tested, but this is too a typical to really be comparable) and India aspires to their sustained success economically. The current super powers are represented by the USA and the OECD average reflects India's aspirations as a superpower. The rising powers are represented by the BRIC countries of Russia and Brazil which reflect the rise of the emerging markets.

Compared to the economic superstars India is almost unfathomably far behind. The TN/HP average 15 year old is over 200 points behind. If a typical grade gain is 40 points a year Indian eighth graders are at the level of Korea third graders in their mathematics mastery. In fact the average TN/HP child is 40 to 50 points behind the worst students in the economic superstars. Equally worrisome is that the best performers in TN/HP - the top 5 percent who India will need in science and technology to complete globally - were almost 100 points behind the average child in Singapore and 83 points behind the average Korean - and a staggering 250 points behind the best in the best.

As the current superpowers are behind the East Asian economic superstars in learning performance the distance to India is not quite as far, but still the average TN/HP child is right at the level of the worst OECD or American students (only 1.5 or 7.5 points ahead). Indians often deride America's schools but the average child placed in an American school would be among the weakest students. Indians might have believed, with President Obama, that American schools were under threat from India but the best TN/HP students are 24 points behind the average American 15 year old.

Even among other "developing" nations that make up the BRICs India lags - from Russia by almost as much as the USA and only for Brazil, which like the rest of Latin America is infamous for lagging education performance does India even come close - and then not even that close.

To put these results in perspective, in the USA there has been huge and continuous concern that has caused seismic shifts in the discourse about education driven, in part, by the fact that the USA is lagging the economic superstars like Korea. But the average US 15 year old is 59 points behind Koreans. TN/HP students are 41.5 points behind Brazil, and twice as far behind Russia (123.5 points) as the US is Korea, and almost four times further behind Singapore (217.5 vs 59) that the US is behind Korea. Yet so far this disastrous performance has yet to occasion a ripple in the education establishment.

Table 1: Comparing Indian (Tamil Nadu and Himachal Pradesh) students mastery of mathematics to economic superstars, current superpowers, and rising superpowers
Country/Region  5th   mean   95th  HP+TN average to comparator average HP+TN average to comparator 5th percentile HP+TN best (95th) to comparator's average HP+TN best (95th) to comparator's 95th
Points TN/HP is behind (-)/ahead(+)

Economic Superstars
Singapore 383 562 725 -217.5 -38.5 -99 -262
Hong Kong 390 555 703 -210.5 -45.5 -92 -240
Korea 397 546 689 -201.5 -52.5 -83 -226

Current Superpower
OECD avg. 343 496 643 -151.5 1.5 -33 -180
USA 337 487 637 -142.5 7.5 -24 -174

Rising Superpowers
Russia 329 468 609 -123.5 15.5 -5 -146
Brazil 261 386 531 -41.5 83.5 77 -68

Indian States
Tamil Nadu 241 351 468
Himachal Pradesh 223 338 458
Average of TN and HP 232 344.5 463
Source: PISA 2009 Plus Results, Table B.3.1 for first three columns and author's calculations.

I have emphasised Mathematics because many believed math was an Indian strong suit. The results for reading and science are similarly bad. Table 2 shows science results in a different format, which shows the proportion of children in various categories of performance. There are three points:

  1. "Below level 1" doesn't even have a description as it implies that so little proficiency is demonstrated it is impossible to distinguish from not knowing anything at all. In the USA, even with its socio-economic and racial inequalities and language inequalities and its failing inner city schools, only 4.2 percent are in this category. In HP 57.9 percent of 15 year olds in school cannot be distinguished from not having learned any science at all and in TN 43.6 percent all in this category - ten times as many as the USA.
  2. PISA considers "level 2" as the minimum level that provides the science competencies that will enable them to participate actively in life situations related to science and technology. Since more than 80 percent of students in both HP and TN are level 1 or below this most students in these states have reached age 15 ill-equipped for the century they will face.
  3. While a thin elite that competes for the few highly selective technical institutes are globally competitive, this is a tiny fraction of the population. The estimate of the fraction of TN or HP students at level 6 in science proficiency was zero. Their estimate of the fraction at level 5: also zero. Of course this does not mean there are not such students in these states, of course there are, just that from the samples available in the study the best estimate was so small as to be indistinguishable from zero.

Table 2: Comparison of science proficiency in Tamil Nadu and Himachal Pradesh to India's aspirations
Country/Region Below level 1 Level 1 1 Level 5 5 Level 6 6
Singapore 2.8 8.7 15.3 4.6
Hong Kong 1.4 5.2 14.2 2
Korea 1.1 5.2 10.5 1.1
OECD avg. 5 13 7.4 1.2
USA 4.2 13.9 7.9 1.3
Russia 5.5 16.5 3.9 0.4
Brazil 19.7 34.5 0.6 0
Tamil Nadu 43.6 40.9 0a 0a
Himachal Pradesh 57.9 30.9 0a 0a
Source: PISA 2009 Plus Results. Description of levels Table 3.2, percentages Table B.3.4.

1) At Level 1, students have such a limited scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific explanations that are obvious and follow explicitly from given evidence.

5) At Level 5, students can identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to situations. They can construct explanations based on evidence and arguments based on their critical analysis.

6) At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they demonstrate willingness to use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.

a) In Table B.3.4 these are reported as blank but the estimated percentages in below 1 to level 4 sum to exactly 100 percent. Obviously this not imply that there are exactly zero students in all of these two states meeting these levels but that with the sample sizes assess students of 1616 in HP and 3210 in TN there was insufficient information to create a non-zero estimate.

These results on PISA 2009+, while tragic for what they imply for Indian youth and perhaps shocking to newcomers to this subject, come as no surprise to those who have been working on basic education in India:

  • Das and Zajonc (2008) used results from Orissa and Rajasthan to create indices on mathematics performance similar to those of TIMSS (Trends in Mathematics and Science Study) and found these states near the bottom of the global rankings.
  • Educational Initiatives carried out an 18 state study using sophisticated testing instruments and found levels of performance on TIMSS comparable items that were stunningly lower. For instance on the open ended question "Write a fraction larger than 2/7" less than 30 percent of Indian students in standard 8 could answer correctly compared to more than 70 percent internationally.
  • The APRest study led by Karthik Muralidharan and Venkatesh Sundararaman in rural AP asked the same questions of students in grades 2 to 5 and found very slow rates of learning progress.
  • The results year after year from the ASER [2010 2009] study supported by Pratham find that significant fractions of students in Standard 8 cannot master even Standard 2 curricular basics. In rural areas nationwide a third of children in grade 8 could not do a simple division problem and almost 20 percent could not read a level 2 text. The 2011 results, due out in a few weeks will show continued stagnation or even retrogress in learning.
  • Numerous studies by MIT's JPAL, World Bank, NCAER/University of Maryland and other researchers found levels of performance that were shockingly low compared to curricular expectations.

These PISA 2009+ results are the end of the beginning. The debate is over. No one can still deny there is a deep crisis in the ability of the existing education system to produce child learning. India's education system is undermining India's legitimate aspirations to be at the global forefront as a prosperous economy, as a global great power, as an emulated polity, and as a fair and just society. As the beginning ends, the question now is: what is to be done?

30 comments:

  1. Hi Ajay,

    I am not at all surprised with this report. If anybody has been part of a recruitment function one knows the pathetic quality of education being churned out.

    Its one thing to make policies and quite another to implement. Im told that each school/college is supposed to have a minimum number of qualified teachers. If somebody does a genuine diligence they will find that a number of these institutions pay 'teachers' just so that they come and sign the attendance registers. While the classes are taken by other 'teachers'. So guess what would be the quality of education!

    We just have to move out of the degree mindset - as all we are doing is creating a bunch of worthless yet 'qualified' young people - whos lives have been ruined, by the system, as they have been told they are engineers etc whereas a lot of them would not even be good data entry operators.

    If we wait for the next report we can be quite sure it will only be a worse result!

    Tarun Hukku

    ReplyDelete
  2. Personally I do not believe that the quality of education in India is as poor as the article seems to make us believe (although the quantity/percent educated may be poor). So my suspicion went to the procedure in which this test was conducted, and whether it was a biased questionnaire. For example, not all children can read American English / English in an American Context with equal ease. Little on the front of "risk factors to interpretation of results" seems to be available in the linked results document.

    However I found the following in the linked article: "Himachal Pradesh-India and Tamil Nadu-India did not meet PISA standards for student sampling. Due to
    irregularities in the student sample numbers, it was established after the testing that these economies sampled from student lists that were often incomplete: not all 15-year-olds within the school were listed. It was not possible to determine whether any bias existed in the obtained sample. Caution should be exercised when using the data from Himachal Pradesh-India or Tamil Nadu-India and when interpreting the reported analyses."

    ReplyDelete
    Replies
    1. hmmm .. very relevant to have this info before we correctly assess and remedy the situation

      Delete
    2. Looks like the two states tried to withhold the weak students so they won't show up on test date to worsen their overall score!

      Delete
  3. Whoa this is abysmal indeed! One question I couldn't find an answer to in the PISA 2009+ report was about the language of the questions. Were all the questions in English?

    ReplyDelete
  4. Kumar, what would you say about the ASER reports then ?

    The questions were framed by Indians, tests were given by Indians and the organization (Pratham) is also Indian.

    Tarun has correctly pointed out a big problem. Teacher quality and motivation is a huge issue.

    ReplyDelete
  5. 'THE LEANING TOWER - PISA'
    A GOOD ARTICLE.LIKED THE COMMENT. THERE IS A CONCERN HERE & WE NEED TO ACT SOON.
    I AM INTO EDUCATION IN INDIA . WHEN WE TOUR SCHOOLS ESP IN TAMILNADU,WE DO FIND THE QUALITY BAD.
    THE REASON IS THAT WHEN A PERSON DOES NOT GET ANY OTHER JOB HE BECOMES A TEACHER.THERE IS NO MOTIVATION TO MAKE THEM TEACH BETTER.TEACHERS SEEM TO TEACH WELL WHEN STUDENTS GO TO THEIR HOMES FOR 'TUITIONS'. PARENTS HAVE NO TIME FOR THEIR KIDS. PRIVATE SCHOOLS BECOMING FEE ORIENTED HAVE MUSHROOMED & GOOD TEACHERS GRADUATE TO BECOME ADMINISTRATORS TO FILL IN THE COFFERS OF THE SCHOOL. RTA CAN DO PRECIOUS LITTLE .WE NEED GREAT EDUCATIONISTS & POLITICAL WILL TO CHANGE ALL THIS. Prasad

    ReplyDelete
  6. This result is obviously true, more so in Higher Education. Upon reading this report, the Government's miraculous solution will be.. wait for it.. here it comes.. "Increased Reservation"... the thunderous applause you're hearing is coming from Sibal and Arjun..

    ReplyDelete
  7. I am not sure about HP, but I studied in Tamil Nadu for a couple of years and I know the educational system there is pathetic (as evidenced by my poor grammar). The emphasis is totally on rote learning. You mug-up a fact, vomit it in the exam, and then forget about it. There was no role for applications, logic and out of the box thinking. Even in mathematics, students used to mug-up problems/sums from the textbook and we used to have marks cut for spelling mistakes. The emphasis was on trying to score full marks rather than trying to understand the concept. SO it is not surprising that Tamil Nadu education fares so badly.

    ReplyDelete
    Replies
    1. I completed my higher education in Hyderabad, AP. The education system there was no different than TN. They put tremendous pressure on children and make them mug up the books and score 99% at the cost of their future and our country's future. Almost 20 years later, it has gotten worse. Now, it starts as early as 6th grade. They attend school 6 days a week for 12 hours a day. What would the tender minds do when they are under so much stress all the time? They'll lose interest in learning and would refuse to absorb knowledge as their brains are tired all the time.

      Delete
  8. the report spells out in a systematic form which so many of us are aware of.

    i am a part of my school alumni association doing a wee bit to be helpful for the current students.

    a lot can be done, but we feel handciapped that unless the contents (this is a gujarat board school in surat) in the form of syllabus and text books are not changed, what ever to be done will be extra and students complain that between school and tuition timings they hardly have time an inclination for self study.

    ReplyDelete
  9. Why is there this "deafening slilence" in the mainstream media on this. Is it incredulity or this does not fit into what they want to feed their readers / viewers which is about individual achievements by Indians and how great and smart, we are.

    ReplyDelete
  10. The same picture emerges when one examines International Math Olympiad results. India's right tail of the distribution (for IQ/Math ability/whatever proxy for these you want to use) produces few of the genius-level people you need to win gold medals at the IMO.

    With malnourishment being endemic, you can be sure that a large chunk of Indians are cretins.

    Anyone who has read the book IQ and the Wealth of Nations will not be surprised with these results, though many dismiss it as pseudo-scientific. Note, however, that TIMSS correlates with SAT scores and RAPM IQ scores at >0.8 levels.

    GMAT average scores for Indians are not very great either (Times of India covered this a while ago, but were gloating about how the average Indian did better on the GMAT than the average American), and we're sampling at the very extreme ends of the Indian IQ distribution here too.

    On USA: Here's a better analysis of how USA does based on U of Chicago grad student Tino Sanandaji's analysis:

    http://super-economy.blogspot.com/2010/12/amazing-truth-about-pisa-scores-usa.html

    Anyway, I'm not sure if much of what I've written is politically correct. Any talk of IQ seems beyond the pale of civilized discourse, and I'm not claiming that IQ has high heritability (http://en.wikipedia.org/wiki/Heritability) in much of the 3rd-world, so I hope you let this comment pass.

    ReplyDelete
  11. I have been teaching 8th and 9th standard students, not whole class, but toppers for NTSE exams for last 5 years. I have observed few things.
    1. Teachers in schools are not of good quality. They hardly teach Maths conceptually, but prefer solving problems sets and pointing out students that exam questions will appear from these sets.
    2. There is huge information gap between parents from higher middle class and parents below this class. Parents with good information set are less dependent on schools and 'tuition'and they encourage their children to do many other things than typically be class topper and get a campus job at the end of education.
    One thing I think generally is education system is too much loaded. Except international schools, in other schools, class size in much more than a teacher can handle. This is true even in tuition.
    The other strange thing is anybody can open a degree college allotting degree in science or arts in any big village. But nobody opens a technical education institute which will provide employment. SO we have degree colleges which oversupplies degree holders who wait to get office job and flood every exam for government job.

    ReplyDelete
  12. Thanks for the insightful post. I have a question regarding the sample set - was the PISA survey in India conducted for government schools alone, or a mix of government and private schools?

    ReplyDelete
  13. More than half of the country is functionally illiterate. So, the question of quality of education in Maths and Science is not even relevant in many parts of the country.

    Did they test in Kerala? I am sure the scores would differ from state to state and vary based on rural, urban settings, economic development etc. I guess we need comprehensive information about this.

    Thanks to Kumar for pointing out the anomalies in this study. It definitely took away the sting.Also, thanks to Ajay for bringing this up.

    ReplyDelete
  14. Whatever may be the sample, the point is that we all end up acquiring degrees without substance. This indeed is a cause for concern.

    ReplyDelete
  15. Firstly, we need to have a comparative study done within India itself using PISA type test (region-wise and urban-rural comparisons) to think of required changes. The second point I wish to make is that in China, only Shangai was considered. It is like taking only Delhi, Mumbai, Chennai, Kolkata or Bangalore which will no doubt throw up a different result altogether (may even be near the top of the table).

    ReplyDelete
  16. 75% of the Indian population gets affirmative action, and by definition, this segment has lower IQ.

    But even beyond IQ, the rote-learning curriculum of the State Boards, as compared to CBSE, makes it impossible for students to try to solve PISA type questions, which require thinking

    ReplyDelete
  17. Calculating Average Indian IQ from PISA

    TN raw math PISA score = 351
    TN implied IQ = 100 - 1.5 x 15 = 78

    HP raw math score = 338
    HP implied IQ = 100 - 1.62 x 15 = 76

    Indian Avg IQ based on raw PISA = 77

    --

    Next step is to remove the bias caused by the PISA sample having
    75% bilingual kids

    ( Tibetan kids facing Hindi PISA exam and Telegu kids facing Tamil PISA exam )

    TN mono-lingual = 378
    Implied IQ = 500 - 1.22 x 15 = 82

    HP mono-lingual = 401
    Implied IQ = 500 - 15 = 85

    --

    Next there is a 40 point difference between scores for 'Village' and scores for 'Large city'
    In HP and TN, The village category is over-represented by a factor of 4
    Even worse, in HP, City and Large City are entirely removed from the survey sample

    So adding an urban correction of 20 ( half the village-large city difference )


    TN semi-urbanised mono-lingual = 378 + 20 = 398
    Implied IQ = 85

    HP semi-urbanised mono-lingual = 401 + 20 = 421
    Implied IQ = 100 - 0.79 x 15 = 88

    Current Indian IQ = 86

    --

    Next we look to the future as malnutrition is removed

    The only Indian kids who go to govt school is for the mid-day meal,

    If they are not starving they go to private school

    Private schools score 45 more than govt schools and thats the future as poverty reduces

    --

    HP - future - semi-urbanised- mono-lingual = 401 + 20 + 45 = 466

    Implied IQ = 95


    TN - future - semi-urbanised - mono-lingual = 378 + 20 + 45 = 443

    Implied IQ = 91

    Future Indian IQ = 93

    Given the huge bias in sampling towards over-representing the lower end IQ,
    by the poverty pimp NGOs, I am certain that none of the CBSE or Cambridge schools
    that serve the top 15% are included in the survey

    And they have an entirely different IQ profile and cause an IQ bulge at the top end

    ReplyDelete
    Replies
    1. 1. Your estimation that 75% of HP tested were Tibetans (bilinguals) is too generous either way. Tibetans in China tested close to OECD average. By your estimation, 75% of Americans would also be bilingual and disadvantaged.
      2. Village vs city. This is the same well-accepted statistical methodology for every country. Why should India be treated differently. You method will give every village the same IQ as city dwellers. That is just not true.
      3. You forget that only 15% of Indian school children reach high school to be tested by PISA. PISA results represents only the top 15% of India's population. Where as 90% of Chinese school children reach high school to be tested.
      4. Based on my calculation, the top 2% of India's population has the same IQ as the bottom 3% of China's.

      Delete
    2. I am a teacher of physics at the top end of top end schools ... I really mean that! I teach my students to re-read their results from experiments for sense. I would ask you to do the same ... "top 2% of India's population has the same IQ as the bottom 3% of China's".
      This simply cannot be true, can it?

      Delete
  18. I am not a big fan you say of our educational system.If i get a chance, i take a dig at how pathetic and dissillusioned, "HOW WE ARE SUPREME BEINGS IN THE WORLD OF EDUCATION". I do not know how this test is. But i think they have to test the best of every country. I think they should not go at random. They should go for may be the first ranker in CBSE or ICSE. I think if this exam is not an reflection of entire India. There is not reason, we cannot make in the worst case to the TOP 10. We may not be intellectuals , may be our Educational system does not make us creative. But with the amount of information that is bombarded to us, we are clever in our own ways, of digesting "hard topics " (of course not in the right way IMHO), of keeping us on our toes.

    ReplyDelete
    Replies
    1. You are not right in thing so. Please consider, how good we are we only 1% of our population will outperform average people in other countries? Here we are talking about Avg. IQ in different countries. Not Super Performers of these countries.

      Delete
  19. Shameless NCERT is in full denial mode questioning the methodology of PISA testing.

    ReplyDelete
  20. .Off with Kapil Sibal's head!Denying the truth is the hallmark of Indian politicians.

    ReplyDelete
  21. It's about average iq here again sadly. We tend to look at students who have gone to the united states to study and settled down there, and point out how (insert figure)% of scientists in america are indians and (insert figure) % of doctors are indians and (insert figure) percentage of nasa scientists are indians and all that bullshit. The fact is such indians make up a miniscule fraction of our country's population. The average indian has a very low iq, as is evident by the lack of rule of law here, the complete lack of innovation, the fact that they vote on the basis of silly things like caste and power without realizing how they are being duped by their leaders and so on. Alot of people tend to claim that India was a powerful civilization historically and richer than the west and so on . But that is incorrect too. While our rulers were despots who passed down power from father to son, Greece had already developed democracy 3000 years ago and Rome had a republican system of government and indoor plumbing for its citizens 2000 years ago ! So I doubt that we were ever as great as we say we are

    ReplyDelete
  22. On the comparison to the 'Rising Superpowers,I could see China is missing.Is the author implying something or trying to hide something?

    ReplyDelete

Please note: Comments are moderated; I will delete comments that misbehave. The rules are as follows. Only civilised conversation is permitted on this blog. Criticising me is perfectly okay; uncivilised language is not. I delete any comment which is spam, has personal attacks against anyone, or uses foul language.