Evaluation or Research
My project represents evaluation rather than research. The reason being, my project is to help the stakeholders understand occurrences within the cycle of the program. In this case, the success of the PBS program and positive behavior modification taking place in the participants. It would be hard to try and control the variables if this were research, and the only manipulation that would take place would be the need to change something in the program itself. The main goal of the evaluator for this project is to collect information and provide feedback. Through summative assessments and frequent meetings, adjustments can be made to improve the program along the way.
There are two ways to look at the question of sampling with my project. When I consider that every student in this particular school will be involved in the evaluation, the answer would be no, there is no sampling. But if I look at it in a larger sense, i.e. this school being the only school in the county implementing this program, than my answer could be yes. For the purpose of this assignment, I am going with the latter.
Since there are tens of thousands of students in the Bay County school district, but only a few hundred at Patterson Elementary who are participating, that makes them a sample. After reading the chapter, I believe I lean more towards a Judgment Sample if I had to label it. I do this simply because all of the students in this sample are low income, minority, one parent families. They do not represent the rest of the county as a whole, but can represent those who are in most need of this program.
YouTube Video on Sampling Methods
In the video the teacher discusses what a sample, or as he describes it a survey, is and why we need to use one. He covers the meaning of population, which is, would include everyone that could be involved in the survey, but since a population would be too large to conduct an effective evaluation, we use a sample, or smaller snapshot of the population of a whole. He also talks about the mean or the average, of the information given in the sample. It always makes more sense to use a sample when the population is very large for both time and, in some cases, costs.
YouTube Video on Simple Random Sampling
Simple random sampling simply means that every item or individual has an equal chance of getting selected for the group. The presenter refers to this as putting names in hat. As long as the names are well shuffled, then each has the same probability of getting picked. However, this type of method can be tainted by human error. Another example of SRS is to use a random digit table where the outcome is random and independent of other outcomes. Lastly, he discussed using a random number generator where all human elements are taken away and a machine helps pick the sample. Also discussed in the video are the pros and cons to simple random sampling. The pros are that these methods are easy to use, and unbiased. The cons to using this sort of sampling is that the variances could be large, that there must be a population reference frame, and that the sample you end up with may not be the best representation of the population.
Answer questions in two ways:
1. What are the differences between quantitative and qualitative?
Quantitative is data that is collected through observation and findings are presented through words and dialogue instead of numbers.
Qualitative is data that is collected by numerical information such as tests, counts, and measurements. The results are given in the form of numbers.
For the Scenario the qualitative data are things like interests, attitudes, and skills of the residents. The example of qualitative data in the scenario was the sign up sheet that gave the number of people engaged in activities.
2. What are the levels of data you might encounter?
Nominal Data is the type of data that is based on one principle such as type, gender, or age.
Ordinal Data is also the type of data based on one principle. This type of data conveys some sort of rank or order. The book describes a scale as being a good example for ordinal data.
Interval Data have rank like ordinal data, but they also have intervals. Most numerical scales have intervals. This type of data contains no absolute zero and therefore the so there can be no assumption between negative and positive numbers.
Ratio Data is data a combination of nominal data, ordinal data, and interval data, with the addition that in ratio data there is an absolute zero. Examples are weight and height.
In the scenario nominal data, such as age and gender of the residence, can be used as well as the name of the activities that they are involved in. Ordinal data could be found on the sign up sheet to see which activities ranked higher in opinion of the residence. Interval data could be used when ranking the level of health benefits in a certain program. Ratio Data could be used when determining overall health such as percentage of weight lost or affects on blood pressure.
3. What are some instruments you might use or develop?
Interview: Gather information by asking question to gain information for qualitative data. In the scenario the residents could be interviewed to see how they feel about the program and how it affects them.
Scales: can be used to gather nominal data, which can be done by having the staff and residents rate their experience with a program.
Sentence Completion: This is done by giving the participant a complete a sentence, thought or problem. For the scenario it would be a good tool to ask participants how, in their opinion, would the improve conditions.
Tests: To gain quantitative data through administering tests that is given and measured the same way every time. The participants could take a test to have a measurement of skills acquired through different activities.
Observational Data: In the book it is described as a method used by evaluators to assess program delivery. It can be unobtrusive, where the person being observed does not know that they are being observed, and obtrusive where the do know. In the scenario the participants activity and involvement can be observed either obtrusively or unobtrusively.
This Data Analysis tutorial is presented by The University of the West of England, and contains a great deal of detail of the process of data analysis. It also gives in depth examples and information of qualitative and quantitative analysis. For instance when it discusses issues of qualitative analysis such as trustworthiness, credibility, transferability, dependability and conformability. It also goes in depth with issues concerning quantitative analysis such as validity and reliability.
When doing an evaluation, it is just as important to know how as well as what you are evaluating. In the readings this week the focus was on how to best evaluate a program. Both discussed the steps in creating a formative and effective evaluation. Chapter 5 in the course text mainly dealt with choosing an evaluation method, and giving details on each method to help the reader decide which method would be best for their given program. In the article reading, it shows how these methods are implemented through examples, and a deeper description of how to arrive at formative and summative questions.
One thing that the readings bring out is that even though there may be one evaluator, creating an evaluation is a group effort. It shows how evaluators need to meet with stakeholders to figure out their concerns and input on what sort of questions need to be asked during the evaluation. If not careful, some evaluators might try to make the evaluation a one-person job by designing relevant questions, but missing over key components that participants want to address. No evaluation is perfect. Money and time constraints can cause loss of opportunity to collect more information and make it hard to go as in depth as desired.
For the project chosen for this course, the evaluation model that would serve the best would be the Decision Making Model. This model was chosen because the main reason for doing this evaluation on the PBS program is to see if it is something the school will continue to use in the future for their discipline plan. They have had a discipline plan in place for sometime with lack luster results, so the stakeholders are looking to replace the former plan with the PBS plan hopefully on a permanent basis.
Another reason to use this model is for the wide variety of sources to collect data. Quantitative and qualitative can and should be used to get a wide scope of how useful the program is. By doing interviews with the participants the evaluator can get a feel as to what extent they use the program as well as what their attitude can be. Quantitative assessment are brought in to the assessment through records of the numbers of referrals as well as good behavior recognition. It also lends itself to formative as well as summative assessment. The program needs to be assessed throughout the year for adjustments, and a summative assessment at the end of the year to see if this indeed is the program that will be instated on a permanent basis.