Research Methods in Computer Science
Research Methods in Computer Science
CS is unusual among technical fields: its primary publication venues are conferences, not journals. The research moves fast — what appears in NeurIPS 2024 today was rejected from ICML 2023 yesterday. Understanding how research works lets you read papers critically, extract real insights, and distinguish breakthroughs from incremental work.
The CS Research Landscape
CS research divides into subfields with distinct venues and cultures:
Theory (algorithms, complexity, cryptography): primarily published in STOC, FOCS, SODA, CRYPTO. Mathematical proofs are the primary contribution. Rarely empirically evaluated. Reviewer emphasis on novelty and correctness.
Systems (OS, networks, distributed systems, storage): SOSP, OSDI, NSDI, USENIX ATC. Emphasis on real implementations, measured performance, evaluation on realistic workloads. "Systems plumbers" prize practicality.
Machine Learning/AI: NeurIPS, ICML, ICLR, AAAI. Empirical results on benchmarks, theoretical analysis secondary. Fastest-moving field — arXiv preprints dominate, papers are months ahead of conferences.
Programming Languages: POPL, PLDI, OOPSLA. Type theory, program analysis, compilers. Mix of formal proofs and empirical evaluation.
Human-Computer Interaction: CHI. User studies, qualitative and quantitative human evaluation. Ethical review for studies involving humans.
Security: IEEE S&P, USENIX Security, CCS, NDSS. CVEs, formal security models, attacks and defenses. Responsible disclosure adds complexity.
Anatomy of a CS Paper
Most CS papers follow this structure:
Abstract: 150-250 words. Problem statement, approach, key result, significance. Reading only the abstract tells you 70% of what you need to decide if it's relevant.
Introduction: Motivates the problem, states contributions explicitly (often as a bulleted list), and outlines the paper structure. The contribution list is the paper's core claim.
Related Work: positions the paper among prior work. Read this to understand what's novel vs incremental. "To our knowledge, X is the first..." is a strong claim to verify.
Technical Content: the meat — architecture, algorithm, proof, or system design. May be the hardest to read but skip liberally on first pass if you don't need to implement it.
Evaluation: experiments, benchmarks, or proofs. For empirical papers, this is critical — is the experimental setup fair? Are baselines strong? Are results statistically significant?
Conclusion: summarizes contributions, limitations, and future work. Limitations are often undersold.
Reading Papers Effectively
Three-pass reading (Keshav, 2007):
- First pass (5-10 minutes): title, abstract, section headings, figures, conclusion. Decide: is this relevant? What are the claims?
- Second pass (1 hour): read carefully, skipping proofs. Understand the key ideas, examine figures, note unfamiliar terms.
- Third pass (4-5 hours): fully understand every step. Attempt to re-derive proofs, identify assumptions, evaluate experiments critically.
Most papers warrant only a first or second pass. Do a third pass only for foundational papers in your area or papers you intend to build upon.
Evaluating claims critically:
- What problem is being solved? Is it the right problem?
- What are the assumptions? Are they realistic?
- Are comparisons fair? Are baselines the best current alternatives?
- Is the improvement statistically significant? What are the confidence intervals?
- Would the approach generalize to settings different from the paper's experiments?
- What is not shown? Papers present their system in the best light.
Experimental Design in CS
A good CS experiment is:
Controlled: only one variable changes at a time. When comparing system A vs system B, hardware, dataset, configuration, and metrics should be identical. Lack of control is the most common experimental flaw.
Representative: benchmarks should reflect real workloads. Micro-benchmarks measure peak performance but not realistic behavior. The SPEC CPU benchmarks are industry standards for CPU performance; TPC-C for databases; MLPerf for ML.
Statistically rigorous: report means, standard deviations, and confidence intervals. Run experiments multiple times with different seeds. Report the median, not just the maximum. Overfitting to random seeds is a documented ML research pathology.
Reproducible: provide code, datasets, and exact hyperparameters. Most ML papers now submit code to OpenReview. Systems papers submit artifacts for evaluation. Despite this, the ML reproducibility crisis persists: different hardware, library versions, and initialization seeds often produce meaningfully different results.
Ablation studies are essential for papers proposing multi-component systems. Without ablations, you cannot tell which components matter. "We achieve 10% improvement" could mean 8% came from one component and the other four collectively added 2%.
The Peer Review Process
A submitted paper goes to an area chair (AC) who assigns 3-5 reviewers. In double-blind review, neither reviewers nor authors know each other's identities. Reviewers submit scores (e.g., 1-10) and written reviews. Authors respond to reviews in a rebuttal. AC makes a recommendation; program chairs decide.
Typical acceptance rates: NeurIPS ~26%, ICML ~22%, ICLR ~32%, SOSP ~18%, PLDI ~22%. Rejection is the norm. The best papers are often rejected once.
Common rejection reasons: lack of novelty ("similar to prior work X"), insufficient evaluation, missing ablations, unclear writing, overclaiming, or missing important baselines.
arXiv preprints: most ML and theory research appears on arXiv months before peer review. This accelerates knowledge dissemination but also means many non-peer-reviewed claims circulate as though established. "Recent papers claim X" ≠ "X is established."
Staying Current
Key strategies: follow top venues on Semantic Scholar/Papers With Code. Set arXiv keyword alerts. Subscribe to researcher Twitter/Bluesky. Read survey papers for comprehensive overviews. Attend or watch conference talks. Follow reproducibility reports.
Papers With Code (paperswithcode.com) tracks state-of-the-art on benchmarks with links to implementations — invaluable for ML. Semantic Scholar provides citation graphs and AI-powered recommendations. Google Scholar for citation counts and author pages.
Understanding research culture makes you a better engineer: you can evaluate vendor claims, prototype new techniques, and distinguish marketing from substance.