RFD 38
Journal Club
Sign in
published

Journal Club

This RFD can be accessed by the following groups:[public]
RFD
38
Updated

Origins

Journal clubs trace their origins to the 19th century, when they were used by physicians to keep current on medical literature. Since then, they have played a pivotal role for physicians in particular, and for the biosciences more generally. The literature on journal clubs themselves cites three essential purposes, concisely summarized by Mark Linzer in a survey of journal club history[linzer]:

It is remarkable to note that all three of the historical goals of medical journal clubs (to 'keep up' with the literature, to impact on clinical practice and to teach critical reading skills) were mentioned in our survey with almost equal frequencies.

Goals

For Oxide, a journal club serves similar purposes to those distilled by Linzer: to keep us abreast of the other work in our domain, to keep our own reading skills sharp (and to model them for more junior engineers), and (especially) to broaden and inform our own engineering thinking, as described in [rfd5].

Experience

The literature on journal clubs (and in particular <a href="https://github.com/oxidecomputer/papers/blob/master/engineering/how-to-run-an-effective-journal-club.pdf">Deendadaylan et al.'s survey</a>) cites some common problems with journal clubs: keeping discussion germane, keeping attendance up, and making sure that everyone has read the papers being discussed. While these problems seem to transcend medicine, the specific solutions used over the years in the health sciences may have less relevance for us in software engineering. To that end, it’s worth discussing previous experiences in software engineering in particular.

Experience at Joyent

In experimenting with a journal club at Joyent, we had a modicum of success, but it ultimately fizzled: after having had monthly cadence for most of a year, Joyent’s journal club slowly became more and more irregular until disappearing entirely.

There were a couple of clear challenges that mirror those seen elsewhere:

  1. Lack of discussion leadership. We tried to rotate leadership of a discussion, but when "looking for volunteers", it seemed to always fall to the same names. This problem was exacerbated by the fact that discussion leadership had become burdensome due to in part to lack of preparation of participants (described below).

  2. Fixed monthly cadence. The monthly cadence was seemingly never the right interval: at times it was too infrequent (in particular, with the benefit of hindsight, a more timely journal club at some key junctures might have resulted in more informed development), and at times too frequent. The lesson is that the cadence should be — as much as possible — dynamic: there are times that a team will want to be reading and discussing the literature very intensely, and times that they will naturally be focused on other things.

  3. Lack of preparation. Regrettably, we found lack of preparation to become rampant: participants would arrive at journal club having not read the paper in advance. This wasn’t misbehavior per se in that the meeting was deliberately not exclusive to those having read the paper (and participants would be upfront about having not read the paper but wishing to benefit from the discussion), but it would invariably result in the leader of the discussion summarizing the work — which itself would take much of the alloted time (and effectively punished those who had read the paper). This reduced journal club to being more of a lecture than a conversation, which in turn made leading a discussion overly burdensome.

Experience at Delphix

One additional lesson from Delphix was that keeping the burden low for the prospective coordinators is essential. We drifted into paralysis in a surprising way. We had a few organizers in a row who were extremely diligent in their preparation. They presented excellent slides that thoroughly and helpfully explained the content of the papers discussed. Each successive presentation was of higher and higher quality. As a result…​

  • the space for discussion decreased

  • the burden on participants to have read the paper decreased

  • and the expectations for what it took to coordinate became insurmountable and no one was willing to coordinate.

It may feel like it runs counter to our value of rigor, but having the coordinator not be more prepared for the discussion than anyone else is important to encourage all forms of participation.

Experience at Transposit

The additional lesson from Transposit was to cultivate a culture of curiosity even in the face of an urgent need to ship. We only made space for papers that were highly related to areas of the product; the result was that often interest was too narrow and often the discussion happened too late to impact the product.

Experience with Papers We Love

Several of us have been involved with (that is, attended and/or presented at) Papers We Love. This is a kind of internet journal club, and has resulted in many fruitful and interesting discussions! An important aspect of Papers We Love is that its sessions are recorded — they can be enjoyed worldwide and long after the fact.

Mechanism

The goals of Journal Club should be to inform engineering decisions and to stay abreast of published research — without falling into the traps experienced in our various past lives, and with mechanisms that can reasonably scale with team size and be sustainable over time. Importantly, Journal Club is optional: if a body of work is considered to be so important as to be mandatory reading, Journal Club is the wrong vehicle. Similarly, the cadence of Journal Club should be dynamic, and should be initiated by those wishing to discuss papers that they have read.

Paper selection

Any paper to be a candidate to be discussed in Journal Club must be in the Oxide papers repository. To initiate a discussion, a GitHub issue should be opened with the description "Journal Club on <Name of paper>". This issue must be opened by someone who has read the paper, and should link to the paper in the repo. The issue should indicate if the submitter it willing to coordinate the discussion. (This amounts to taking responsibility for scheduling and getting any results recorded; see below.) The GitHub issue is also an opportunity to explicitly tag those who might also be interested ("I think @jonvonn mentioned that he and @alovelace both wanted to read this paper?").

Those who are potentially interested but haven’t yet read the paper can express their interest by GitHub emoji (e.g., a +1), but once an interested party has read the paper and is ready to discuss, they should indicate that by explicitly and affirmatively commenting on the issue, e.g. "I have read this paper; count me in!" If the original submitter has not indicated that they are willing to coordinate the discussion, this is an also an opportunity to volunteer to coordinate.

Once two (or more!) people have read the paper and agreed to discuss it, the designated coordinator should schedule a time for a convening of Journal Club to discuss.

Scheduling

Scheduling can be the most difficult part of a conversation; here are some guidelines for Journal Club.

First, it does not need to be too far in the future: all participants have read the paper, and there is value to having discussion sooner rather than later!

Second, one of the surprising common themes of the literature on journal clubs was that the successful ones involve food or beverage. We deliberately keep lunches unscheduled at Oxide, which offers a conveniently available time for Journal Club! While a Journal Club does not strictly need to be scheduled during lunch Pacific time, this is encouraged: it’s a time that is likely to be available, and is generally workable in different time zones. If lunch is unavailable or undesirable for some reason, aim for a time when people might be able to have a beverage in hand — caffeinated or otherwise.

A time should be proposed by the coordinator on the GitHub issue (one that doesn’t conflict with another Journal Club), and an entry on the Journal Club calendar should be created.

Meeting

Anyone who has read the paper and expressed their interest on the GitHub issue can attend the Journal Club. Requiring that one has read the paper in order to participate might seem harsh at first, but it keeps the time spent together focussed on discussing the work rather than explaining it.

The coordinator should make sure that any who can attend in the designated time are accommodated (i.e., setting up a video conference). On that note: Journal Club should always be recorded. Our nascent experience with Journal Club has shown these discussions to be incredible valuable. Recording Journal Club has the obvious benefit of allowing others to later read the paper and enjoy the fruits of the discussion, but there is a subtler benefit too: when the content is highly technical, even those who attended Journal Club can often benefit from repeated listenings.

Discussion

In terms of the discussion itself, it can hopefully become meaty relatively quickly: everyone should come to the meeting with questions or observations about the paper. While the questions to ask will (certainly) vary from subdomain to subdomain and work to work, here are some questions to potentially consider:

  • What is the most important result of the paper?

  • What are the most surprising aspects of the paper?

  • What are the aspects of the system described or evaluated that make it particularly relevant to us?

  • What are the aspects of the system described or evaluated that limit the applicability to us?

  • What wisdom does the paper contain that we can reflect in our systems?

  • In what areas has the paper inspired us to do further investigation?

  • Are there papers in the bibliography that merit reading?

Summarizing

After Journal Club, the coordinator (or designee) should close the GitHub issue on the reports repo, indicating that the conversation took place, linking to the recording. If there are particularly germane conclusions from the discussion (other papers to read, other people at Oxide who should read the paper, particular wisdom that we wish to incorporate in our systems, etc.), those should be denoted on the issue.

Feedback

After a Journal Club, if there is abstract feedback on Journal Club itself (e.g., on the goals or mechanics), it should be directed to this RFD. If there is concrete feedback on a particular session, it should be directed to the GitHub issue for that session of Journal Club.

References