How the Rating System Works

At Paperstars, we believe that research should be judged by its quality — not by where it's published or how often it's cited.

That's why we've built a structured, weighted rating system that reflects the key components of rigorous research. Each review is more than just a star — it's a breakdown of the paper's strengths and weaknesses across core criteria.

📋 The Seven Evaluation Criteria

Each paper is assessed across seven components, with different levels of importance (weights). For each criterion, reviewers select one of three rubric options.

Component	What We're Asking	Weight
Title	Was the title appropriate, slightly misleading, or exaggerated?	1
Methods	Were the methods robust and appropriate for the question?	4
Statistical Analysis	Was the statistical analysis appropriate, or were there signs of p-hacking or misuse?	4
Data Presentation	Were the figures clear and appropriate, or potentially misleading?	3
Discussion	Were the results interpreted appropriately and reasonably?	2
Limitations	Were the limitations clearly acknowledged and discussed?	2
Data Availability	Were the data and code shared openly and transparently?	4

Each of these components is rated on a simple scale:

✅ Strong / Appropriate – Full credit (1 point)
⚠️ Mid-level / Minor issues – Partial credit (0.5 points)
❌ Weak / Inadequate – No credit (0 points)

🧮 How It All Comes Together

Each component's score is weighted based on its importance (e.g., methods and statistical analysis carry more weight than the title). The system then calculates a final percentage score.

This score maps onto a star rating:

Score Range	Star Rating
81–100%	⭐⭐⭐⭐⭐ (5 stars)
51–80%	⭐⭐⭐⭐ (4 stars)
41–50%	⭐⭐⭐ (3 stars)
21–40%	⭐⭐ (2 stars)
1–20%	⭐ (1 star)

💡 Reviewers can also manually adjust their final star rating if they feel the calculation doesn't fully reflect their overall impression.

🧾 A Structured Review Process

To support transparency and consistency, reviewers are guided with structured prompts, ensuring reviews are:

Thoughtful and complete
Focused on the actual content of the paper
Helpful for readers and fair to authors

A minimum word count ensures reviews provide real insight, not empty praise or vague criticism.

🔐 Why It Matters

This isn't just about scoring papers — it's about building a culture of thoughtful evaluation, open science, and academic accountability.

By breaking down quality into clear, weighted components, we avoid the pitfalls of simplistic star ratings or inflated citation metrics. We believe this is a better way to assess scientific contribution — and we hope you'll be part of it.