It will be interesting to see how this scales. I am the director of a small, rural library with a single paid position – mine – and it is part time. We operate on volunteers and have a very challenging budget so have no “staff” people to assist with studies and evaluations. I’ll chime in more later on as I digest this avenue of investigation. Having run a stat lab as a grad student in a former life, this used to be fun and I guess still is! Evaluation of program design seems like a natural partner to this.
Thank you! As a first-timer, really dissecting the evaluation process has been helpful. I look forward to exploring it further with the group. For those who may be interested, my question is a simple one “What could we do to increase participation in our ACL Book Club?” Due to COVID, it’s gone by the wayside, but before then, there was great interest and correspondence (email/text) but when it came time to actually meet, no luck. It seems everyone wants a book club, but no one wants to participate IN a book club! Perhaps another perspective would be to consider attendance in a percentage, rather than expecting 100%? When you are as small as we are, every person counts. Ways to increase participation is a chief focus and I believe if I would pre-evaluate my programs with many of these questions, it would help me plan better, certainly with more measurable outcomes.
Holly,
Thanks for posting your question and the context. This will be a great evaluation question for you to plan to answer. You have the program and are defining what participation is. Who is your target audience and how will you reach them to ask about participating? For example, do you have people who signed up that you would ask? Or is this a more general question to the public? You might also start to think about which collection methods you might use. Surveys? Interviews? Focus group?
This was awesome! I’m really excited, as I’ve been asking myself how my system can better evaluate our children’s programming, particularly in STEM/STEAM areas. My question is really specific, as we started purchasing sets of Bee-Bots for use in different branches – I’ve been very curious as to how we can make these things as valuable to patrons as possible! This has also been helpful as it helped me think that I may have to change some of my breakout choices around to fit observation in. Thanks a ton for this!
Hello Linda and Sharon! Just finished this preparation for the LIVE event. Thank you for the handholding. Ok my question has been for many months if not years (we turned five in October) so are a new library for our community of 9,000 ballpark people….my question is more about service hours rather than a program, but I really need to know the answer and I have been struggling with it for the last five years now. We have grown our hours to a total of 61 hours a week and the growth of staff has not kept up with the expansion of hours. This is my question. “Are the library service/operating hours conducive to patron needs and staffing availability?” Am I on the right track here? We are operating currently and have been for the last five years at a 10% staff deficit and an additional 16 hours (40%) over the Gold level standard of 40 hours required by our state for our population. I am not trying to cut back hours per se, but I am concerned about staff burnout and the excessive work load of 8 a.m. to 8 p.m. for a small community. It to me seems inefficient and overstriving and unnecessary for our community needs. I would love to hear from Sharon and Linda on this and see if I am wrong to ask this question. I also feel like I need the data (which I have the participation of during these times) to show my argument as my City Administration does not think it is a big deal. They say if one person is here in the library (we staff) need to be here. I just think this needs some reevaluation for the benefit of staff to avoid burnout, not to mention I feel it has become a safety issue. Thank you! CAn’t wait to tackle this properly.
Hi Tracy! I think that your question works well, and certainly understand your concern about staff burnout. And, your use of data is appropriate for demonstrating the issues of overwork/understaffing. One question – do you have a door counter, and if not, do you have any other means of analyzing usage patterns across each day of the week? If so, I would recommend analyzing those data to determine priority hours for staffing, based on demand, as well as to make the argument for hours that could be cut because of limited usage.
Leah shared your email with me, and we will respond with some information about how you could make the case for more staff.
Hi Tracy! I think that your question works well, and certainly understand your concern about staff burnout. And, your use of data is appropriate for demonstrating the issues of overwork/understaffing. One question – do you have a door counter, and if not, do you have any other means of analyzing usage patterns across each day of the week? If so, I would recommend analyzing those data to determine priority hours for staffing, based on demand, as well as to make the argument for hours that could be cut because of limited usage.
Leah shared your email with me, and we will respond with some information about how you could make the case for more staff.
Wow. There is a lot of great information presented in these three videos. When discussing the Outcomes, I understand your definition as level of services and materials provided and how ‘number of events offered’ fits that definition; however, I feel like that fits better as an input. How do others feel about that or what characteristic sets that factor apart from an input?
Hi Paul – thanks for asking this question. The level of services and materials provided are actually outputs, not outcomes. Inputs would be the resources (money, staff) put into providing a certain level of services, materials circulated, etc. And then outcomes would the be effect that the services/materials have on users. How do they change, in terms of attitudes, knowledge, behavior, etc.?
I am a new librarian and remember going through these input, output and outcome exercises in library school. Unfortunately, they don’t seem to be playing much of a role in the real world. In my head, I can already hear my overworked colleagues say, “You want me to do what? I don’t have time.” Plus, when it comes to data collection, our marketing department likes to keep tight control. Finally, this evaluation process seems to be better suited to ongoing programs, that is programs that run as a series, not one-off programs. Is that a correct perspective?
Hi Beth – thanks for your comments and questions. Whether or not inputs, outputs, outcomes, and/or other data play a role in the “real world” is very dependent on the library. Some libraries are active users of data for purposes including advocacy, decision-making, and improvement. I recommend taking a look at the case studies available in the Public Library Association’s Project Outcome toolkit for examples of data use in libraries: https://www.projectoutcome.org/surveys-resources/case-studies (you need to sign up for a free account to access the case studies).
We’re going to discuss issues such as convincing overworked colleagues to collect data in a couple RIPL sessions – the Change Management breakout and the “Data Geeks Disperse!” general session.
And, regarding your question about what evaluation is best suited for, it can be used for one-off programs as well as series. Whether a library is offering a one-off program or a series, they should have objectives for these programs in terms of what participants will gain from them (for example, learning something new, developing a skill, etc.). So, evaluation can be used to determine whether the program (one-off or series) achieved these objectives.
I am struggling to work with a small, rural library that receives little use. I want to help the board evaluate why it is used so little by the community. I can see obvious issues, but they need to see this process to understand what is not working. If I can do this during the conference I believe I can return and help them carry out a great evaluation of the library to see where they need to make drastic changes – by using data they collect.
Is this just going to be too general of a question? Or can I do this? It is such a need to know for this board. If they see it, I believe they will institute changes.
The question my director would like me to look at is which Readers Advisory tools/services we are currently providing provide a worthwhile bang for the buck.
-Grab bags with curbside pickup (new since COVID)
-Online form for personalized recommendations (new since COVID)
-Email recommendations to target populations based on Orange Boy segments (used in a very limited way currently)
-Subscription RA emails based on genre (LibraryAware/NextReads)
-RA Database: Novelist
-Phone or in-person RA interviewing and recommendations
-Book Bites (podcasts)
-Book Discussions
-Hot Authors List (upcoming titles shared in print, on website and via NextReads subscription)
-Social Media “Librarian Recommended” posts
-Displays
Possibly too much to look at at once? Should I just focus on one or two?
Hi Robyn – I would recommend starting with one service. This will allow you to gain experience with the evaluation process and then refine it for subsequent services. You could select the service based on which one best targets one of the groups you are interested in (families w/young children, etc.). Let me know what you decide!
I work for a state library agency, and I don’t interact with the public. I’m struggling to determine a suitable evaluation project. Any suggestions would be welcome. Thank you.
Hi Kelly – great question! Previous RIPL participants from state libraries or regional consortia have tended to take one of the following approaches:
1 – Put themselves in a public librarian’s shoes and developed an evaluation plan for a public library program or service, which they’ve used as a basis for training staff in their state
2 – Evaluated a program or service within their own agency – for example, Talking Book Library, evaluations of the professional development or consulting services staff provide, services such as state park pass checkout or resource kits, etc.
Please let me know if you have additional questions.
Thanks, Linda. I’m going to focus on PLAS training. My question – which I’m concerned is too narrow – is “Are written instructions an effective training and support model for PLAS training?”
Thanks for the excellent overview. The evaluation question I want to pursue is, “How well are our adult technology classes meeting the learning needs of adult learners with limited computer literacy in our community?” Part of the evaluation process seems obvious to me–to ask attendees to self report on the outcomes of whether they acquired skills, confidence, or are more likely to use technology after attending our tech classes. But that’s not the main thing I want to know, based on the WHY of my evaluation question. The WHY is that our staff report a lot of requests from patrons for technology classes, but relative to to patron inquiries we have fairly low actual attendance at classes. I’m less clear on how to evaluate why that gap is occurring and how we can close that gap. I suspect that the reason is either the tech topics we’re offering, the times the classes are offered, or the mode of instruction. Would a survey of patrons who inquire about tech classes be a better tool?
Thank you! These videos were a great overview of the evaluation process. I love the Cola explanation for input, output and outcomes!
It will be interesting to see how this scales. I am the director of a small, rural library with a single paid position – mine – and it is part time. We operate on volunteers and have a very challenging budget so have no “staff” people to assist with studies and evaluations. I’ll chime in more later on as I digest this avenue of investigation. Having run a stat lab as a grad student in a former life, this used to be fun and I guess still is! Evaluation of program design seems like a natural partner to this.
Thank you! As a first-timer, really dissecting the evaluation process has been helpful. I look forward to exploring it further with the group. For those who may be interested, my question is a simple one “What could we do to increase participation in our ACL Book Club?” Due to COVID, it’s gone by the wayside, but before then, there was great interest and correspondence (email/text) but when it came time to actually meet, no luck. It seems everyone wants a book club, but no one wants to participate IN a book club! Perhaps another perspective would be to consider attendance in a percentage, rather than expecting 100%? When you are as small as we are, every person counts. Ways to increase participation is a chief focus and I believe if I would pre-evaluate my programs with many of these questions, it would help me plan better, certainly with more measurable outcomes.
Holly,
Thanks for posting your question and the context. This will be a great evaluation question for you to plan to answer. You have the program and are defining what participation is. Who is your target audience and how will you reach them to ask about participating? For example, do you have people who signed up that you would ask? Or is this a more general question to the public? You might also start to think about which collection methods you might use. Surveys? Interviews? Focus group?
This was awesome! I’m really excited, as I’ve been asking myself how my system can better evaluate our children’s programming, particularly in STEM/STEAM areas. My question is really specific, as we started purchasing sets of Bee-Bots for use in different branches – I’ve been very curious as to how we can make these things as valuable to patrons as possible! This has also been helpful as it helped me think that I may have to change some of my breakout choices around to fit observation in. Thanks a ton for this!
Hello Linda and Sharon! Just finished this preparation for the LIVE event. Thank you for the handholding. Ok my question has been for many months if not years (we turned five in October) so are a new library for our community of 9,000 ballpark people….my question is more about service hours rather than a program, but I really need to know the answer and I have been struggling with it for the last five years now. We have grown our hours to a total of 61 hours a week and the growth of staff has not kept up with the expansion of hours. This is my question. “Are the library service/operating hours conducive to patron needs and staffing availability?” Am I on the right track here? We are operating currently and have been for the last five years at a 10% staff deficit and an additional 16 hours (40%) over the Gold level standard of 40 hours required by our state for our population. I am not trying to cut back hours per se, but I am concerned about staff burnout and the excessive work load of 8 a.m. to 8 p.m. for a small community. It to me seems inefficient and overstriving and unnecessary for our community needs. I would love to hear from Sharon and Linda on this and see if I am wrong to ask this question. I also feel like I need the data (which I have the participation of during these times) to show my argument as my City Administration does not think it is a big deal. They say if one person is here in the library (we staff) need to be here. I just think this needs some reevaluation for the benefit of staff to avoid burnout, not to mention I feel it has become a safety issue. Thank you! CAn’t wait to tackle this properly.
Hi Tracy! I think that your question works well, and certainly understand your concern about staff burnout. And, your use of data is appropriate for demonstrating the issues of overwork/understaffing. One question – do you have a door counter, and if not, do you have any other means of analyzing usage patterns across each day of the week? If so, I would recommend analyzing those data to determine priority hours for staffing, based on demand, as well as to make the argument for hours that could be cut because of limited usage.
Leah shared your email with me, and we will respond with some information about how you could make the case for more staff.
I’m excited to see how you decide to tackle this!
Great overview and exercises. Looking forward to the virtual event. Thank you
Hi Tracy! I think that your question works well, and certainly understand your concern about staff burnout. And, your use of data is appropriate for demonstrating the issues of overwork/understaffing. One question – do you have a door counter, and if not, do you have any other means of analyzing usage patterns across each day of the week? If so, I would recommend analyzing those data to determine priority hours for staffing, based on demand, as well as to make the argument for hours that could be cut because of limited usage.
Leah shared your email with me, and we will respond with some information about how you could make the case for more staff.
I’m excited to see how you decide to tackle this!
Wow. There is a lot of great information presented in these three videos. When discussing the Outcomes, I understand your definition as level of services and materials provided and how ‘number of events offered’ fits that definition; however, I feel like that fits better as an input. How do others feel about that or what characteristic sets that factor apart from an input?
Hi Paul – thanks for asking this question. The level of services and materials provided are actually outputs, not outcomes. Inputs would be the resources (money, staff) put into providing a certain level of services, materials circulated, etc. And then outcomes would the be effect that the services/materials have on users. How do they change, in terms of attitudes, knowledge, behavior, etc.?
I am a new librarian and remember going through these input, output and outcome exercises in library school. Unfortunately, they don’t seem to be playing much of a role in the real world. In my head, I can already hear my overworked colleagues say, “You want me to do what? I don’t have time.” Plus, when it comes to data collection, our marketing department likes to keep tight control. Finally, this evaluation process seems to be better suited to ongoing programs, that is programs that run as a series, not one-off programs. Is that a correct perspective?
Hi Beth – thanks for your comments and questions. Whether or not inputs, outputs, outcomes, and/or other data play a role in the “real world” is very dependent on the library. Some libraries are active users of data for purposes including advocacy, decision-making, and improvement. I recommend taking a look at the case studies available in the Public Library Association’s Project Outcome toolkit for examples of data use in libraries: https://www.projectoutcome.org/surveys-resources/case-studies (you need to sign up for a free account to access the case studies).
We’re going to discuss issues such as convincing overworked colleagues to collect data in a couple RIPL sessions – the Change Management breakout and the “Data Geeks Disperse!” general session.
And, regarding your question about what evaluation is best suited for, it can be used for one-off programs as well as series. Whether a library is offering a one-off program or a series, they should have objectives for these programs in terms of what participants will gain from them (for example, learning something new, developing a skill, etc.). So, evaluation can be used to determine whether the program (one-off or series) achieved these objectives.
I am struggling to work with a small, rural library that receives little use. I want to help the board evaluate why it is used so little by the community. I can see obvious issues, but they need to see this process to understand what is not working. If I can do this during the conference I believe I can return and help them carry out a great evaluation of the library to see where they need to make drastic changes – by using data they collect.
Is this just going to be too general of a question? Or can I do this? It is such a need to know for this board. If they see it, I believe they will institute changes.
Thoughts?
Hi Jeanne – Thanks for emailing me about this – please let me know if you have any other questions.
The question my director would like me to look at is which Readers Advisory tools/services we are currently providing provide a worthwhile bang for the buck.
-Grab bags with curbside pickup (new since COVID)
-Online form for personalized recommendations (new since COVID)
-Email recommendations to target populations based on Orange Boy segments (used in a very limited way currently)
-Subscription RA emails based on genre (LibraryAware/NextReads)
-RA Database: Novelist
-Phone or in-person RA interviewing and recommendations
-Book Bites (podcasts)
-Book Discussions
-Hot Authors List (upcoming titles shared in print, on website and via NextReads subscription)
-Social Media “Librarian Recommended” posts
-Displays
Possibly too much to look at at once? Should I just focus on one or two?
Hi Robyn – I would recommend starting with one service. This will allow you to gain experience with the evaluation process and then refine it for subsequent services. You could select the service based on which one best targets one of the groups you are interested in (families w/young children, etc.). Let me know what you decide!
I work for a state library agency, and I don’t interact with the public. I’m struggling to determine a suitable evaluation project. Any suggestions would be welcome. Thank you.
Hi Kelly – great question! Previous RIPL participants from state libraries or regional consortia have tended to take one of the following approaches:
1 – Put themselves in a public librarian’s shoes and developed an evaluation plan for a public library program or service, which they’ve used as a basis for training staff in their state
2 – Evaluated a program or service within their own agency – for example, Talking Book Library, evaluations of the professional development or consulting services staff provide, services such as state park pass checkout or resource kits, etc.
Please let me know if you have additional questions.
Thanks,
Linda
Thanks, Linda. I’m going to focus on PLAS training. My question – which I’m concerned is too narrow – is “Are written instructions an effective training and support model for PLAS training?”
Actually – I think this works perfectly, unless there is another training/support model that you want to evaluate at the same time.
Thanks for the excellent overview. The evaluation question I want to pursue is, “How well are our adult technology classes meeting the learning needs of adult learners with limited computer literacy in our community?” Part of the evaluation process seems obvious to me–to ask attendees to self report on the outcomes of whether they acquired skills, confidence, or are more likely to use technology after attending our tech classes. But that’s not the main thing I want to know, based on the WHY of my evaluation question. The WHY is that our staff report a lot of requests from patrons for technology classes, but relative to to patron inquiries we have fairly low actual attendance at classes. I’m less clear on how to evaluate why that gap is occurring and how we can close that gap. I suspect that the reason is either the tech topics we’re offering, the times the classes are offered, or the mode of instruction. Would a survey of patrons who inquire about tech classes be a better tool?