How instant is instant messaging compared to web-based chat in virtual reference?

Two of the most common kinds of things we hear from patrons are "you are too slow!" and "you are so fast!". What does it mean?

There are all kinds of ways to measure speed in synchronous virtual reference. Are patrons responding to the total time of the session? How long she had to wait before talking to a librarian? How much time it takes the librarian to respond after the patron sends a message? Just the general feel of the software?

I looked at two sets of records from different virtual reference platforms and a set of my own instant message logs to see if instant messaging is really more "instant" than web-based chat.

Apples and oranges warning: I don't provide virtual reference over instant messaging, so this is a comparison of virtual reference with mostly professional conversations. If you offer IM virtual reference and you keep logs and want to share your data, see the footnote for how I examined Trillian's XML logs.

I examined 227 logs of IMs with co-workers and an odd friend or patron.

I examined 1,226 chat transcripts from L-net in February, 2006. QuestionPoint/24-7 was using an "eGain" platform.

I examined 1,124 chat transcripts from L-net in December, 2006. QuestionPoint/24-7 was using a "Flash Chat" platform.

I wanted to find out two things:

1. Does the length of the session vary by platform?
2. Does the time in between messages vary by platform?

For each group, I looked at each session and counted the total session time, the number of messages sent by the librarian and the number of messages sent by the patron.

The idea is to only track actual messages between the patron and librarian, so I unscientifically subtracted a fixed number based on the number of automatic messages in each kind of QuestionPoint Transcript; any IM system messages in the Trillian logs were likewise not counted.

I then eliminated transcripts for which either the patron or the librarian sent less than one message and did some math.

I looked at means, medians, modes, minimums, maximums and standard deviations, but found the medians most helpful, so as to eliminate the extremes. For example, I did not want to average in a session recorded as having lasted 24 hours. For IM, I counted the beginning and end of the session as the difference between the time the first and last message were sent.

I think it's interesting that the two QuestionPoint sets are pretty close in total session time but otherwise different. If reference in web-based chat still takes 15-16 minutes, perhaps patrons and librarians just send as many messages as they can in that period.

Here are some pictures demonstrating the difference between the three services. Each point on a graph represents a session. It's place on the x-axis represents the number of messages sent. It's place on the y-axis represents the mean time between messages for that session.

The tick-marked sections on the x-axis represent 10 messages. Each section on the y-axis represents 60 seconds.

Instant Messaging

QuestionPoint eGain

Note: I only have time data by minute for these sessions, so every session's length is a multiple of 60, and that's how come we see that cool distribution pattern on the lower right. Reference question: does anyone know what that kind of pattern is called?

QuestionPoint Flash Chat

What the pictures show is that a lot of IM sessions have messages trading back at least every 10 seconds, even if there are 70 or 80 messages in the session. There are a few in QP/eGain, but none at all in QP/Flash Chat.

But back to apples and oranges, what does this mean? Are reference chats longer than IM chats, or is IM really 'instant'?

Request: If you offer IM reference and have some logs to share or analyze, please get at me and/or write a paper about it.

Why web-based chat is slow

My hypothesis here is that IM is more "instant" than web-based chat. It was more than anecdote that led me to that idea.

OCLC says their server response time is about .02 seconds per query (or per message sent). This is very good, but it only measures the time it takes the QuestionPoint server to take a message and add it to the database - there is additional time for the remote librarian's computer to send the message to OCLC and for the remote patron's computer to learn that a new message exists and process it.

In QuestionPoint, the time to "learn that a new message exists" is key. The patron's screen refreshes every six seconds (or so), so the actual time for a message to be sent and received can be increased by up to six seconds, depending on the patron's luck.

I have no idea how often the librarian's Flash Chat interface is querying the QP server, but the same issue could exist there as well.

At the ALA Midwinter QuestionPoint users group meeting, OCLC made hints that an AJAX-based patron interface may be available next fall, which I'm guessing means early December. At that time, perhaps the patron's interface will query the server more than once every six seconds - that will definitely improve the response time for the patron's browser, even if it slows down the server somewhat.

Footnote - How I extracted session time and message counts from Trillian logs

I used Microsoft Word, Firefox and Microsoft Excel. You can use Internet Explorer if you don't have Firefox, as long as it is version 5 or higher and supports XSL transformations.

First, I concatenated my logs into one long text file and changed the file extension to 'xml'. Copy and paste worked well, but there are geek tricks you can use too.

Second, I restructured the Trillian logs so that each transcript is part of a separate XML node. I made a backup copy of the file and opened it file in Microsoft Word for a little find-and-replace:

<session type="start"

for

</imsession><imsession><session type="start"

Notice that it doesn't remove any of the existing coding, so don't do it twice.

Then I moved the extra </imsession> at the beginning to the end of the file, so each transcript was in an <imsession></imsession> wrapper.

I put the whole document inside another wrapper, <data></data> and saved the file as an XML document.

Third, I created an XSL file to extract the data I wanted (session time, number of messages). You can download mine if you want.

Fourth, I added a reference to my XSL file at the beginning of my XML Trillian log by adding the lines

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="im.xsl"?>

Fifth, I opened the XML Trillian logs in Firefox. The XSL file formatted them as a table with the data I wanted, the number of messages in the session (sessions with only one party are eliminated), the time of the first message in the bunch, the time of the last incoming message, and the time of the last outgoing message.

Sixth, I could have tried to figure out how to get XSL to do the sums and averages for me, but since I am better at Excel than XSL (say that out loud), I copied and pasted the table to Excel and did all my sums and averages there.

Then I made graphs and told you about it.