A Simple Procedure to Develop Local Norms


Kevin C.H. Parker
Psychology Department
University of Waterloo

Abstract

Outlines a simple procedure for development of Local norms. Local norms are useful to cope with populations not accommodated by existing norms to identify differences of a local population from other groups, to identify local errors to test administration and to identify problems in the test. Procedures for dealing with ethical issues, data collection, and data analysis are outlined.

A Simple Procedure to Develop Local Norms

Tests come in many different forms, but those tests which have been standardised offer something special because they have norms. Norms are tables or charts that describe the performance of a large group of individuals who were given the test. This means that whenever the test is given to someone new, that person's performance can be compared to the norm group's performance. Without the norm group to refer to, performance can only be understood based on the experience of the person evaluating the test. with the norm group, the experience of many other tests and testers augments each test given.

The Diagnostic Inventory for Screening Children (OISC) to a norm based standardise test that allows a tester to compare a child's performance against the performance of the 572 children in the normative group. (Mainland and Amdur, 1982). Many settings will find the norms to be adequate for all of their needs, Out some settings will have special needs not met by these norms. For example, where an needs children (e.g. gifted, premature children) there may agency is dealing with special developmentally delayed, or be measurable and important differences among the children. These children, however, were by design uncommon in the group used to develop norms, so the norms don't contain enough information to help differentiate one special child from another. In this kind of development of local norms can be very helpful.

Local norms are simply norms developed from a locally obtained group of individuals. Thus, an agency that has been using the DISC for some time could systematically assemble the DISC scores of all of its children and produce norms that incorporate issues are experiences that are locally relevant.

In the following pages, I should introduce a procedure that will allow an agency to develop its own set of Local norms. In order to make it as useable as possible, I assume only that you can do and understand basic mathematical operations including squaring and taking square roots. A basic understanding of statistics and of psychological measurement would be helpful, but not essential. Brown (1983) and Wetkowitz, Ewen, and Cohen (1976) are two of many readable undergraduate textbooks in this area that you might consider using as references to expand your understanding.

The Finished Product

It seems worthwhile to begin with an outline of what the final product of Local norms for the DISC wilt Look like. Any set of local norms will have to be broken down by age and by DISC scale. The simplest set of Local norms will be a set of eight tables (one for each of the eight DISC scales modelled on the Tables in the OISC manual (Tables -1 to A-8). Each table will consist of one or more columns of numbers. Each column will represent a group identified by age and any other relevant variable (e.g. gender). Each row will represent a given score on the DISC. Each number in the columns of the table will represent the percentage of children who got the raw score for that row or less. Thus the numbers will go from 0 to 100 as the row score

Most people preparing Local norms will want to add some "wrinkles" of their our that are relevant to themselves. Thus the norms might add some other distinction to age so that there are multiple sets of age tables. For example, there might be one set of norms for boys are one for girls, or one set tot children with hearing problems and another for children with vision problems. As a rule of thumb, emery column in the norm table ought to include results from about 5C children. This means that it may take a large number of children to generate the 'wrinkle' that your group needs. The age groupings used in the OISC norms were the Largest ranges considered Legitimate. This means that about 100 children of roughly the same age would be the smallest number that you could use to produce two-group Local norms (either two age groups or two groups of the same age).

Collecting the Data

Ethical Issues. For most agencies, collecting the data will be a matter of searching through records. This means that the first step is to evaluate the ethical implications of such a search. In most agencies, agency staff have access to records as a matter of course, an no problem will arise to the records are searched by a staff member There may be situations where it is most appropriate for individual staff to search their own records only and summarize the relevant data before passing them on to a co-ordinator. In cases where an outsider is to search the records, the issue of consent of the clients becomes a serious problem that must be carefully evaluated. Another issue is the confidentiality of data. By and Large, it the name and birth date are re. Dyed, the data will be unidentifiable. In any case, ethical evaluation of procedures ought to be the first step in the method.

Identifying Groups

Before starting the record search, you wilt need to identify the groups that you will use for your norms. As a rule of thumb you will need 50 children per group. Some of your groups will be pre-determined because of the age characteristics. The most appropriate age groupings will be those used in the DISC groupings be much wider cases where you would manual. In no case should your age than those used there. There may be like narrower age groups or groups that cover a different span of ages, but are the same width. In these cases, your results ought to be different than the norms in the DISC Manual, because you have changed the way you have subdivided the data.

Once you have established the number of age groups required, it is important to estimate the number of subjects available and their age distribution. Assuming 50 subjects per age group, this will give you some sense of how you could divide your subjects. In a small pre-school setting, you may be able to generate only one group at one age level. In a larger setting you may be able to generate four or more at several age Levels.

Once you know how .any groups you can have, you can decide how you want to divide your subjects. Boys versus girls is an obvious example. Other criterion might be by referral source by parental characteristics, by prior pre-school experience, or any characteristic that is Likely to be important based on your own experience in your setting.

Coding data

Coding the data should be relatively simple, but may be more or less time consuming, depending on how your data records are organized. Your most appropriate procedure is to establish three kinds of records. One kind of record is the DISC form used for testing along with other kinds of confidential information about the child. A second kind of record is a test of children's names and subject numbers that are assigned to the children in an arbitrary manner. 1his list should be considered as confidential as the first kind of records mentioned above. The third kind of record is the raw data set for the norms. These records consist of the child's subject number, the child's age at testing, the group he or se will be assigned to and the eight DISC scores. You may want a more detailed data set for more sophisticated analyses discussed later, include the child's birth date in these data, it possible to identify a given child, confidentiality.

The virtue of this procedure is that you the third records form a data set that can be shared or analyzed by someone outside et the institution without violation of confidentiality. Using the subject number and the second set of records will allow children to be backtracked to fire out more about specific cases, to check for errors, or to modify the grouping procedure. This recording procedure can be facilitated somewhat by collecting and recording the three types of record at the time of testing. Where there is limited access to computing facilities, the third set of data use for norms may be recorded on index cares to facilitate sorting of the children into groups at a later date.

Analyzing the Data

The data analysis is quite simple. Sort the groups that you have chosen. Within each group, subjects by score on the first DISC scale (Fine Motor). Sort the children into sort the Record how many children got a score of O. then 1. then 2. and so on until you reach 27. Now sort the cards again by the score on the second DISC scale, then the third and so on. In the end, you wilt have eight lists of DISC scores indicating how many children got a given score: the £t £g0S Lists.

The next step is to generate the cumulative frequencies List. On a new sheet of paper, record how many children got 0 as a score. Then record how many children got 0 or I in the second row (i.e., the score to the '0' row on the cumulative frequencies List plus the score or the '1' row from the raw frequencies list). When this is finished, the cumulative frequencies list will indicate how many children got the score indicated at the beginning of the row, or a lower score.

The next step is to generate a cumulative percentage distribution. For every entry in the cumulative frequency distribution, divide by the number of children in the group, and multiply by 100. The result is the percentage of children who obtained a given score or less on a DISC scale. This table your local norm for the group you have seen working with. When a new child is tested, he or she can be compared with this table and the performance stated as being as good or better than X% of the children of this type in your experience.

Further Analyses

Mean and Standard deviation. The mean and standard deviation are the baste statistics used by Psychologists and other social science types to describe groups of scores. The mean to more familiar to most people when called an average. To get the mean of a list of numbers simply add the List (to get the 'sum") and divide by the number of entries in the List.

Standard deviations will be unfamiliar to people with no statistical background. The standard deviation is a measure of how variable a list of numbers happens to be. For example/ the mean daily temperature to a desert valley could be the same as the mean daily temperature on a tropical island. The island will have a very constant daily temperature but the desert will be very cold at night and very warm by day. The standard deviation of the hourly temperatures will be greater for the desert than for the island, even though the means are the same.

The standard deviation is not too as you are careful. If you don't know already a statistics hard to calculate so long know how already, it is text. What follows is a simple explanation. You can check your understanding by trying to compute the standard deviation of the following numbers: &, 5, 7, 9. 10. The mean is 7 and the standard deviation is 2.55. Make a vertical list of scores on a piece of paper. To the right of each score write the squared score. Find the sum of the scores (on the Left) n the sum of the squared scores (on the right). Make sure you keep track of which is which. Square the sum of the scores (i.e., the sum on the Left). Divide the sum of the squared scores (i.e.. the one on the right) scores. Subtract this quantity (i.e.. scores divided by the number of scores) the scores (i.e., the value on the Left, by the number of the sum of the squared from the squared sum of squared). Divide this difference by one less than the number of subjects. What you have now is the variance. The square root of the variance is the standard deviation. Or any DISC scale, the standard deviation ought to be roughly 1.5. If you Get less than 0.5 or more than 5.0 make sure you haven't computed the values incorrectly.

Testing Differences

Once you have computed the means and standard deviations of your norm groups, you may wish to test to see if there is a difference between two of your groups, or between one of your groups and another group such as the equivalent group to the manual. Your best procedure would be to go to an elementary statistics textbook and follow the procedure for a t-test for two sample means. If you have trouble with this, try to find someone who knows the material well enough to help you. {e.g., a psychology graduate with an Honours B.A. or higher degree.)

Checking Items

There may be certain items on the test that you believe are poorly 'placed' for your population (i.e., too easy or too hard). If you think that this is the case, read over the item instructions try carefully to ensure that it is being administered carefully. If this is the case, you can check the item by expanding the data collected in the third type of record identified above.

The data needed for checking an item are, at Least, whether the item was passed or failed, and what the child's score was on the scale that includes the 1rem. With these data, you can generate a table that Indicates the percentage of children who pass the target item as a function of how well they did on the scale. The table would indicate what percentage of all the children who got I on a given scale passed the target item, then what percentage of children who got 2 passed the target item an so on. While the simplest procedure would be to use the obtained score on the relevant score, it is better to exclude the contribution of the target item. Thus if a child passes the target item the score on the scale ought to be reduced by one and if the child fails the target item the score should remain unchanged.

A good item will show an increase in the percentage of children passing the item as their scores on the relevant scale Increase. Thus item 15 on the Fine Motor scale should be passed more often by children who passed 15 of the other items than by children who passed 14 of the other items. Properly ordered items will show about 50% of children passing item #X when they get a X-1 of the other items correct Thus, about 50% of children who got 14 other items correct should get item number 15 correct. If the difference from 50% is substantial, you may have an interesting difference in your population or in your administration of the item.

Discussion

A simple set of has been outlined for the local norms offer the potential to: adapt the test to populations not accommodate the norms in the manual, identify differences between your population and other populations of children, identify problems to your administration of the DISC, and identify problems in your administration of the ISC, and identify problems in the D[$C 1rems themselves. It is our hope that a number of agencies will be able to develop and share Local norms.

References

Brown, F.G. (1983) Principles of Educational and Psychological Testing (3rd Edition) Holt. Reinhart and Winston. New York.

Mainland, M. and Amdur, J. (1982). The Diagnostic Inventory for Screening Children. K-W Hospital, Kitchener, Ontario.

Welkowitz, J., Ewen, R.E., and Cohen, J. (1976). Introductory Statistics for the Behavioural Sciences, (2nd Edition). Academic Press, New York.



Back to Mainland Consulting


Web page design by Jerry Walsh
jwalsh@bigfoot.com