Back to Evaluation Team home

Evaluation Team Meeting
01/12/05
present: Anne Gruel, Jennifer May, Michael Bowman
minutes taken by Jennifer May

Prior to meeting we discussed via email just looking at what Vancouver Public Library (VPL) measures - as a starting point - see if there are any glaring things missing we feel we should be counting, or perhaps things we don't feel are important to count, or are unable to count, and then prioritize them for our service. This would be useful for a number of reasons: we have a great report from VPL outlining their process; Caleb already provided L-Net's numbers to VPL when they conducted their survey (so we can easily look at numbers historically); and for the future it creates a kind of standardization for different VR services to compare themselves by...

So, we started the meeting reviewing VPL's performance measures as stated in their presentation titled "We've Counted It - Now What?: Establishing Performance Targets for a Virtual Reference Service"∞

- Chat Questions Per Capita (# of questions/population of service area)
- Chat Questions Per Hour (# of questions/total hours of service)
- Chat vs. Total Reference Activity (total questions received in all formats/total chat questions)
- Cost Per Transaction (total cost of chat service/total questions answered)
- Time Per Transaction (average transaction length)
- Capacity Analysis (total staff minutes/average transaction length)
- Repeat Users (total unique users/total repeat users)
- User Satisfaction (exit survey responses)

We decided our first step should be to review the measures and choose our priorities.

Discussion of Measures:
- Chat Questions Per Capita

• break it down a bit more, by zip code/region and look at how well we are serving different regions (as this was one of the selling points of VR, to better serve remote regions)
• choose 4-6 regions

- Chat Questions Per Hour

• already being done

- Chat vs. Total Reference Activity

• Caleb has created a ranking of L-Net vs. other libraries in regard to reference transactions (estimate) - keep doing it
• real numbers are dependant on how other libraries are tracking reference transactions
  • OSU and Jackson pick a sample week
  • PSU has been counting every hour since reorg, but not currently using numbers
  • MCL counts every hour...

- Cost Per Transaction

• Look at LSTA grant monies and divide by total questions answered

- Time Per Transaction

• already being done
• later in meeting discussed this in connection with Ref Tracker (see below)

- Capacity Analysis

• not discussed, will not be looking into at this time

- Repeat Users and User Satisfaction

• both important and need to be looked into further (see below for further discussion)

Other measurements of value:
- Level of Information Needed (from general) or Grade Level (from Kids and Teens) - possibly refer to as "User Profile"
- Ref Tracker?

• should we look at how many chat questions end up being placed in Ref Tracker?
• look at response time/turnaround time for answering Ref Tracker questions
  • people who come in using chat are expecting quick turnaround
  • this could fall under User Satisfaction, and could help us improve our service
• look at what type of questions go to Ref Tracker
  • subject specialty vs. local policy
  • could lead to better referral

- Look at # of sources used during chat?

• would require transcript analysis - perhaps falls under Training Team?

User Satisfaction Discussion:
- L-Net's response rate to the exit survey is low

• Did VPL look at response rate to exit surveys in their study?
  • If so, study libraries with higher response rates and see what techniques they are using
  • If not, contact libraries (same ones from VPL survey for starters), determine response rates and go from there

Decision on Priorities - or, what we think we need to have before April Board Meeting:
- Per Capita by Region
- Cost Per Transaction
- User Satisfaction (includes Repeat User and Exit Survey)

Action Items:
- Anne to contact chairs of other L-Net Teams and see what they need to do their work
- Michael to speak with Caleb about our priority measures (as determined today) and how we get those numbers generated
- Jennifer to begin looking at User Satisfaction exit surveys
- Jennifer to invite Caleb to next meeting on January 19 at 8:30am

Back to Evaluation Team home