The plan I initially wanted to follow would have me scrape headlines from BuzzFeed, the New York Times, The Hindu and Scoopwhoop, assemble my own corpus, and run it through Bi-LSTM and GRU architectures. II wanted to measure click-through rate, time spent reading, and an upvote or downvote at the end of every article. Almost none of that survived.

The custom corpus went first mainly due to time. The bigger problem was that building it meant sitting down and personally scoring every headline on a 1–5 clickbait scale. There is no reason anyone should take my labels seriously. The Webis Clickbait Corpus 2017 contains 38,517 posts from 27 US publishers, each judged by five Mechanical Turk annotators, and years of published work benchmarked against it. Using it meant my numbers could be placed next to somebody else's. That was worth more than owning the data.

The baseline that should worry you

Webis 2017 leans about three to one toward non-clickbait: 14,777 against 4,761 in the training split.

Before running anything real, I trained a model on nothing at all. A constant vector with no features, text or signal. It scored 0.76 accuracy.

Everything after that had to be read against that number. A model can be right about three quarters of the time on this dataset while having learned nothing, because guessing "non-clickbait" is a reasonable strategy when three quarters of the corpus is non-clickbait. Macro-F1 on that empty baseline was 0.43, and that is the figure that actually separates a working model from a lucky one. Every result below is macro-averaged for this reason.

The features I assumed mattered

I engineered eight features from each headline: character count, word count, whether it opened with who/what/when/why/where/how, adjective count, exclamation marks, question marks, slang terms, and uppercase ratio. Then I trained models on restricted subsets to see which linguistic dimension was carrying the weight.

Question-based features scored 0.18 macro-F1. Below the empty baseline, with recall on the clickbait class close to zero. The rhetorical question is the single most cited marker of clickbait in the popular understanding of it, and on 38,000 real posts it was worse than useless on its own. They appear too rarely to classify anything.

Style features, the exclamation marks and capitals and slang, reached 0.47. Barely above a model trained on nothing.

The best shallow feature was headline length. Count-based features alone hit 0.60. How many characters the thing is.

Dropping half the dataset

Webis gives you two text fields: postText, the tweet as it was posted, and targetTitle, the original article headline. I started with both. Combining them was the strongest feature-engineered configuration I had at 0.62, beforeI dropping postText entirely.

The reasoning was about the reader rather than the model. postText varies with platform constraints and whoever ran the social account that day. targetTitle is the thing a person actually sees before deciding to click. A detector trained on the field the reader never encounters first is solving a slightly different problem than the one I claimed to be solving. Giving up two points of macro-F1 to model the real decision seemed like the right trade, and it made the later A/B study logical.

TF-IDF vectors from that one field lifted Logistic Regression to 0.72 which none of my eight hand-built features did. Rebalancing the training set through undersampling, repeated across five random seeds so I could report a standard deviation rather than a single hopeful run, held the best classical models around 0.73–0.74. A 1D CNN on tokenised titles reached 0.76 on test.

The danger of a good validation score

Fine-tuning RoBERTa-base on titles alone produced 0.796 validation macro-F1 and 0.689 on test. That gap is the most honest thing in the results chapter.

I then added the handcrafted features back, one group at a time, to see whether any of them added something the transformer had missed. Count features helped by boosting test macro-F1 to 0.706 whereas style features did not. And the variant with everything stacked on top scored the highest validation figure in the entire study, 0.807, then fell to 0.680 on test. The best looking model I trained was the second-worst model I trained.

The confusion matrices explain what the count variant actually bought. Title-only RoBERTa caught 1,686 of 4,515 clickbait headlines in the test set while falsely flagging 1,324. Adding length features caught 3,331 and falsely flagged 3,495. It roughly doubled detection and nearly tripled the false alarms. Over-flagging was teacheing readers to ignore the badge and under-flagging made the badge is decorative.

Putting it in front of people

I wanted to make an interface to test the model. Thirty headlines across three rounds, five picks per round, drawn from an even split of clickbait and non-clickbait.

The control group saw plain text. The experimental group saw the same headlines with a coloured badge carrying a band and a percentage: Unlikely (0.30), Likely (0.60), Highly Likely (0.75) and Clickbait above that. Across the thirty headlines the bands fell 18 / 4 / 4 / 4, which meant most badges were reassuring and only a few were warnings. Selections were logged with timestamps and group assignment.

I had nineteen student complete it. Ten in control and nine in experimental. University guidelines restricted recruitment to students at the university. The study collected no personal data and participants were anonymous by UID.

Control participants averaged 8.1 clickbait picks out of 15, with several hitting 10. The experimental group's median dropped to 7 and the spread widened. Two people in the experimental group picked 11 and 12.

They saw the warnings and clicked the flagged headlines anyway, more than anyone in the control group did. With nineteen participants I cannot tell you whether that is reactance, curiosity, distrust of the model or two individuals having a normal day. It is the finding I would most want to chase with a real sample, because a labelling system that advertises the most manipulative headlines to a subset of readers is doing the opposite of its job.

The technical result is that RoBERTa with headline length is the best configuration I produced, around 0.71 macro-F1, well short of the high-90s figures that appear in papers using small single-domain corpora. The more useful result is that most of what people believe identifies clickbait, the punctuation and the capitals and the rhetorical questions, does almost no classification work at scale, and that showing someone a probability score is not the same as changing their mind.

What I would do differently

If I were to visit my project, the evaluation would like have to come from one script. Every number in this write-up should be emitted by the code into a single results file, and the write-up should read from that file rather than from numbers copied across at different points in the project. I would tune the decision threshold rather than accepting 0.5, because the real question was never which model scores highest, it was where to sit on the precision-recall curve. I would also probably test on an out-of-domain set, since the published failure mode for models fine-tuned on Webis collapses when they leave it. On the study side, I would run label vs. label instead of labels vs. nothing. Comparing a badged interface to a bare one only tells you that badges do something. The two people who clicked more when warned suggest the more useful experiment is whether a percentage, a band name, or a plain icon produces different behaviour.

Code: github.com/vivekach7/clickbait-detection-nlp