• Scary feedback: Motivation is not good. 
• • Conflicting points… too broad
  • Identify 2-3 key issues
  • Matt: Choosing a research topic, which has been worked on a lot
  • Willem: Nugget was missing in the talk.

• Feedback: Look into related work
  
• • How do you make a fair comparison to something which is not really related?
  • Matt: Feature subsumption
  • Willem: Mention the goal of the tool in the paper clearly. If the reviewer is in doubt, they might comment in that way.

• DON'T generalize or over-sell. 
• • Don’t do it in Academia. State your claims accurately.
  • Better to undersell and over deliver than the reverse

• Acknowledge the limitations — Good for reviewers
  

• Summarize your work in simple sentences
• • What are you doing? e.g. solving Foo
  • What is your technique? e.g., static analysis
  • What us your insight? e.g., take advantage of big data

•  How to write a paper - It’s like fishing
• • FLOW:
  • • Abstract - get the reader interested
    • 1st two pages - get them involved
    • Rest of the paper - how you do it for real
  • If you wait too long to give the punchline, the reviewer will be LOST
  • You should have a KILLER motivating example
  • Threats to validity is not important.
  • Limitations should be discussed in the discussion and mentioned in the conclusion
  • Shing-Chi: Reviewer’s thinking process
  • • First see if the paper interests me
    • Then think about interesting points and come back to the paper
  • Reviewers STOP reading if they sense it’s weak

• DS presentation too technical?
• • Make the presentation clear. What is done / What is missing?

• A good presentation mirrors a good paper
  
• • Motivating example — reader should clearly know what you are doing?
    

• How to improve evaluation
• • Try to do as much as you can do — Using student subjects is better than nothing.
  • Possibly use mechanical turk
  • Compare to state-of-art

• Building huge systems
• • Examples: Benchmarks in other areas. Physicists take a long time 
  • Effort can be valuable in the end. e.g., JPF is huge but a new feature might be 10 LOC
  • Work with your advisor to plan the timelines

• What to do when you receive short feedback?
• • If there is an obvious error, report to PC chair
  • Everyone gets bad reviews once in a while. Luck!

• Reviewer selection process
• • ICSE takes 5 days
  • Reviewer bidding is based on the Title mainly and sometimes abstract 

• Dissertation best practices
• • Consult local standards - varies across university, size/style
  • Tie together the papers along the story if you can
  • Have your papers in the dissertation and clearly emphasize what is YOUR contribution (and playing down the shared contribution of others).

• How are PhD’s evaluated
• • Top conference papers - perception of quality. More is not always better
  • Universities evaluate professors using the same metric.