October 15, 2009

Showing But Not Telling: Why Charts Don’t Always Present The Complete Picture

by Freddy J. Nager, Founder of Atomic Tango LLC + Guy Who Loves To Question The Answers

Sometimes you look at a chart and it seems that everything is clear as day. Then you look closer and that “day” turns out to be a foggy afternoon at San Francisco Airport with everything completely grounded.

For example, check out this chart of school staffing, spending and test scores in the United States:

spending per pupil

I received this from a Republican (yes, I really have Republican friends) who argued that, “While spending has increased 150% per pupil and staffing has increased over 70% since 1970, achievement (in the form of student test scores) has either stagnated or declined… This is why I contend that the nation does not need a federal Department of Education. In terms of increasing student achievement, it has been a failure.”

Looks pretty convincing, right? But for me this chart raised more questions than it did answers:

1. Do the test results reflect changes in family income? Lots of middle and upper class students have left the public school system in the past 40 years. The American middle class in general has shrunk. In addition, more than 40 years ago, many working class children didn’t even go to high school — they went straight into farming or factory work at age 16. Now the working class constitutes a majority of many public schools. Level test scores don’t look so bad in that light.

2. Did all schools enjoy equal increases in spending? How did schools with no increases in spending or staffing perform? How about schools that endured budget and staffing cuts? If the latter two cases showed test score declines, then increased spending was effective.

3. What was the money spent on? If a school already has nice facilities (like in the suburbs), any increase in spending can go to teaching and learning tools. For a school that started with inadequate facilities, increased spending often goes to just keeping pace with repairs, crowding and security. Then there are the costs of those learning tools: Personal computers didn’t even exist in 1970, now they’re present in most schools. Meanwhile, the price of textbooks has far outpaced inflation.

4. Do the expenses include school construction? The past 40 years has seen a surge in new school construction, since 100-year-old buildings tend to fall apart. The problem is that land values and construction costs have skyrocketed since 1970, particularly in major metropolitan areas. There’s not exactly a lot of cheap empty space in major cities to just start building. If a 2-bedroom condo in L.A. costs $1 million, what does constructing a brand new school from scratch cost?

5. Is immigration considered here? The percentage of public school students for whom English is a second language has increased substantially since 1970. Poor immigrants usually don’t go to private schools. Indeed, considering the number of languages spoken in today’s urban public schools, it’s amazing that reading-test scores have even been flat. ESL also costs extra. Would reading-test scores have plummeted had expenditures and staffing not increased?

6. Weren’t teacher salaries extremely low to begin with? Although salaries have increased, have they kept pace with salary increases in other industries and with the cost of living, particularly in major cities? What do public schools have to spend to attract good teachers in the 21st century? Some regions of the country (Santa Barbara, California, for example) have had to subsidize housing for public employees. Is that included in the expenses?

7. Did the researchers always measure inner city and poor rural students as comprehensively as they do now? The U.S. Census didn’t.

8. Have the tests — and how they’re graded changed — over the course of 40 years? Since I work in education, I’m guessing, oh yeah.

9. Could there have been a deleterious effect on student ability to learn thanks to increased television, video games, parental drug abuse, poor childhood nutrition and other social factors? With both parents now working in most households, particularly in the working class, public schools have been asked to do a lot more child raising, from discipline to providing proper meals to simply transporting the kids to campus. You must also factor in the number of public school students raised in single parent households, which has also increased.

That’s just a quick brainstorm of mitigating factors. There could be many more. The point is that simple charts can be impressive to look at, but they don’t necessarily prove anything — particularly since a 2-dimensional graphic can only show a few elements. Pictures can be helpful, but they’re not always worth 1,000 words.

Tags : ,

Freddy is the Founder & Creative Strategist of Atomic Tango. He also teaches at the University of Southern California (go Trojans!), shoots pool somewhat adequately, and herds cats. Freddy received his BA from Harvard and his MBA from USC.

5 Responses

  1. Oh my god. Is this guy serious?

    Teacher salaries are a joke. Teachers in general are overworked, unappreciated, and underpaid.

    You can thank Republicans for slashing school funding, which is driving up tuitions and costs, lowering teacher salaries, cutting school programs, and causing public colleges to start moving to the private sector. That’s all we need. More privatization.

    I hate it when people find one piece of partisan information and use that to justify their beliefs. That’s like the argument Republicans use about health care…”Oh, well, one person in Canada was denied a heart surgery and died! Down with health care reform! Down with socialism!” Meanwhile thousands of people die every year because of our defunct health care system.

    I could go on and on, but I’ll stop. Either way, your buddy should look into the facts a little further instead of deciding that one graph justifies destroying our educational system.

  2. Here’s something totally anecdotal (umm . . . or maybe not):

    I’m very close to two kids who have recently had educational “incidents”. One is in a reputable public school, the other in an Upper-East-Side-of-Manhattan private school.

    When the public school kid ran into trouble, the system’s solution was to drop him into a less-challenging class, even though he clearly had the ability to handle what was on his plate. When the private school student had trouble, they assigned her extra help.

    That’s about money, folks. And guess which child got into a FABULOUS college?

    Yes, the statistics can be made to look like spending more money didn’t yield results, but that’s the NATURE of statistics; they said whatever you want them to say.

    What a buffoon.

    Jeff Yablon

  3. The elephant in the room not taken into consideration in the explanation of this chart is the increasing and staggering amount of money poured into Special Education during those years.

    I used to do research for an educational publisher, and they asked us to investigate what the cost of books contributed to this chart (vs buildings, teacher salaries, etc.) Over time, all those other things were flat (in real dollars); almost all of the increase in educational spending over the period could be attributed to increasing Special Education requirements.

    A township near me had at one point, in a single class, TWO children who required full-time aides to accompany them to each class. Both children were almost completely incapacitated and nonverbal. Can you imagine what the salaries (+ benefits) for these two aides could purchase for the “normal” kids in the school if it were not set aside for Special Ed? On the other hand, do we want to go back to a time when the deaf, blind, autistic, or otherwise disabled were rejected from public schools, and those with other learning problems (like dyslexia) were ignored? Doesn’t providing these children with a solid education make economic sense, helping at least some of them to become functional, tax-paying adults?

    It’s never quite as black and white as the some people would like it to be.

  4. Are these comments for real? Are these people that dedicated to one side of the issue to completely hide from some very serious issues to consider? While there are many factors that must be considered, these stats are amazing and, as a tax payer, very troubling…

    Just as if you had a man that says he is thirsy and you keep giving him cup after cup of water, and he still says he’s thirsty and so you give him more and more…yet still he is thirsty. There comes a point where you must acknowledge that possibly there is something wrong and more water won’t solve the situation.

    Perhaps we are at that point? I am not the one to make that call, but it is very concerning. My personal oppinion is that it is a cultural thing and those that want an education, that are driven by there parents to get one will get one and those that aren’t won’t…But that is not a problem that more money will solve.

    But, why is this a Republican / Democrat issue? Is it really possible from the information given that this Republican could have a valid concern? And if the spending is absolutely required to keep the scores from falling off of the chart maybe we need to spend more just to keep our heads above water. However, if this is the case, I think it is a sad commentary on where the state of this great nation has come to. Anyway…

  5. TB – The increasing cost of education is largely due to Special Ed spending, and Special Ed students don’t really fit your simplistic “it is a cultural thing and those that want an education, that are driven by there parents to get one will get one and those that aren’t won’t” model.

    A deaf child and his parents might desperately want an education, but be unable to function in a regular classroom without costly assistance.

    And Democrat is a noun, not an adjective. So one would say “It’s a Republican/Democratic issue”.

Leave a Reply

Your email address will not be published. Required fields are marked *