Memory Recall Experiment

In this experiment, we asked programmers to recall the location of refactoring on pie menus.

• This file contains 5 slides shown to participants before the experiment began. The test administrator explained each slide to give the participants an idea of what to expect during the experiment.
• This file contains the training and test packet given to participants. Pages 1-9: control group training packet. Pages 10-19: control group test packet. Pages 20-28: experimental group training packet. Pages 28-38: experimental group test packet.
• This file contains the results of the experiment. The column "demogrpahics.Subject" contains subject identifiers: numerals are from the design patterns class, A-F are research assistants, and M-Q are commercial developers. Treatment=TRUE when a subject is in the treatment group.

OSCON 2007 Survey

In this survey, we asked professional programmers about their opinions of the refactoring tools.

• This file contains 2 pages that the interviewer used while interviewing people. Note that the last question was only asked of the last 11 interviewees. Also, note that this paper was only exposed to the interviewees briefly, and served as a cheat-sheet for the interviewer to use, not as a verbatim script.
• This page contains screencasts shown to interviewees. Also, links to checkout the original tool are here. (Be sure to checkout version 645).
• This file contains the results of the survey.

How We Refactor, and How We Know It: Methodology

In this set of experiments, we characterized how programmers refactor. Below, you'll find the materials that we used to do it:

• The Database. This contains the data, and queries, which we used to do much of our analysis. Note that we removed the Toolsmith's refactoring tool usage for privacy reasons. Some of the tables of interest:

  tool_usage.  This table contained data from 
  .refactorings directories from four Toolsmiths.
  

  projects.  This table contains the 41 Eclipse projects 
  which tool_usage referred to.
  

  eclipse_dev_commits.  All files changed (extracted from
  CVS) for all projects listed in projects.
  

  eclipse_dev_inspected.  A list of all refactorings that
  (and some non-refactorings) that we detected by 
  manually comparing Eclipse versions.
  

  udc_usage.  A list of Eclipse commands, and how many
  times and by how many users they were executed.
  

  random_numbers.  Pre-defined random numbers, used
  for randomizing global commit ids.
  

  The following queries are also notable:

  update_global_commits_using_ratzinger.  A query that
  updates the _global_commits_of_interest table
  and checks hasRefactoringComment, according
  to Ratzinger's commit-classifying methodology.
  

  commmits_labeled.  The pool from which 20 commits
  matching Ratzinger's keywords were drawn.
  

  commmits_unlabeled.  The pool from which 20 commits
  not matching Ratzinger's keywords were drawn.
  

• Data Tools. This Eclipse project is an Eclipse plugin that we used to do some of our analysis. Specifically, we used CVSGroupingGenerator to group CVS revisions in the database into single commits, DigRefactoringGroupGenerator to batch .refactoring-style tool uses in the database, MurphyGroupGenerator to do the same for command-style tool uses, and PutRefactoringsInAccess to take .refactoring directories and import them into our access database.
• Eclipse CVS Tool. We used this tool easily check out Eclipse projects. Essentially, you open up Eclipse with this plugin running, select several carriage return separated global-ids in an editor, select the parent of all Eclipse CVS projects selected in the Repository View, right-click, and choose Checkout.../All Eclipse Projects. This will fetch the commit dates from the database, and checkout those projects before and after the commit.
• The Projects. This contains two sets of 20 randomly selected Eclipse versions that we checked out from CVS. Unzip and import these into your workspace. Projects are ordered by global commit id, which also happens to sorted by author, then sorted chronologically. The numbered prefix indicates the commit, then BEFORE or AFTER indicates whether the project is before or after the checkin, then the project name. Note that some global IDs have more than 2 projects; in these cases, several projects were committed.
• Experimenter's Notebook. This is a preliminary working document that we used to formulate and carry out our experiments. Note that, being preliminary, many of the tables that it references are not up to date, and thus this document should be considered historical, rather than definitive.
• Our Normalization Procedure. In the paper, we discussed a normalization procedure for some reported data. To explain the procedure we used for calculating, we'll give the intuitive explanation and an example calculation below:
  
  

  In section 3.5, we wish to estimate how many pure-refactoring commits were made to CVS. Recall that previously, we sampled 20 Labeled projects and 20 Unlabeled projects, and we know that 6 Labeled commits were pure-refactoring and 0 Unlabeled commits were pure-refactoring. Naively, might simply do the addition (6+0) and divide over the total commits to get the estimate: 6/40 = 15%. However, this is a good estimate for our sample, but a bad estimate for the population as a whole, because our 20-20 sample was drawn from two unequal strata. Specifically, in this naive estimate, we are giving too much weight to the 6 pure-refactoring commits, because Labeled commits only account for about 10% of total commits. So what do we do?

  Instead of the naive approach, we normalize our estimate for the relative proportions of Labeled (~10%) to Unlabeled commits (~90%). The following calculation gives the normalized result:

  6 is the number of Labeled pure-refactoring commits.
  0 is the number of Unlabeled pure-refactoring commits.
  290 is the number of Labeled commits.
  2498 is the number of Unlabeled commits.
  (6/20)*(290/(290+2498)) + (0/20)*(1-290/(290+2498)) = 0.0312051649928264
  

  And thus, we estimate that about 3% of commits contained pure-refactorings.

TSE Extension

• The Projects.
• The Database.
• The Survey. This survey was sent to developers of the projects that we studied.

Code Smell Experiment: Methodology

Blank Experimenter's Notebook. The experiment administrator filled this out as the experiment progressed.

Smell Cards. The 8 cards given to participants to familiarize them with smells.

Result Database. Contains 3 tables:

finding.  Each record was a Java file (indicated by 
'firstOrSecond') inspected by the programmer using 
a tool or manually ('usedTool' boolean).  Numbers in 
the last 8 columns indicate how many times they noticed 
that smell.  For instance, subject 4 inspected code 8 
with the help of a tool and didn't see any data clumping, 
but did notice 6 instances of feature envy.

questionnaire.  *_important records the middle major 
column in the post-experiment questionnaire, while 
*_obey records the right major column in the 
post-experiment questionnaire.

subjects. Demographics from the subjects.  Job 
descriptions are removed to preserve anonymity.

Result Notes. This document contains transcriptions of my handwritten notes from the experiment. Note that there may be some transcription errors.

We used code from Vuze and the Java libraries:

Basic Violated Precondition Experiment

In this experiment, we asked programmers to select code using 3 different selection tools, and interpret the meaning of violated Extract Method preconditions using either Refactoring Annotations or error messages.

• Blank Experimenter's Notebook. The experiment administrator filled this out as the experiment progressed.
• Data. An Excel spreadsheet containing timings and other measurements, subject opinions, and demographics.
• Code. An Eclipse project containing the code that programmers selected and analyzed for precondition violations. Look for the following strings in the code: "SELECT" for if statement selection and "EXTRACT" for attempted Extract Methods.

Generalized Violated Precondition Experiment

In this experiment, we asked programmers to interpret the meaning of violated refactoring preconditions using either Refactoring Annotations or error messages.

• Blank Experimenter's Notebook. The experiment administrator filled this out as the experiment progressed.
• Results Database. A database containing the results of the experiment. Tables include:

  subjects.  Demographic information about the 10
  subject programmers.
  

  timings.  How long it to complete 8 refactoring analysis
  tasks.  Two records per subject; one for Refactoring
  Annotations, one for error messages.
  

  numerical_results.  The number of correct, missed,
  neutral, and irrelevant code fragments identified
  for each violation interpretation task.
  

  questionnaire.  Results of the post-experiment questionnaire.
  

• Code. 32 PDF files containing example code used in the simulated refactorings. Each PDF was printed out and attached to a flip chart, and includes selected code and an error message or Refactoring Annotation.