Case study · NLP · Solo project
BERT financial sentiment — optimising for the class that matters
A three-class sentiment classifier for financial text whose first version looked good on accuracy and was useless at the one job that matters: catching bad news. The fix was cheap; the interesting part is what it cost and what the evaluation cannot yet say.
- Result
- Negative-class recall 28.6% → 87.4% on a 1,169-sentence validation split; overall accuracy 81.5% → 79.4%
- My part
- Everything: data loading, training loop, class weighting, early stopping, evaluation
- Model
bert-base-uncased(110M parameters) with a three-way classification head, fine-tuned end to end- Data
- 5,842 labelled financial sentences and headlines; 53.6% neutral, 31.7% positive, 14.7% negative
- Stack
- PyTorch, Hugging Face Transformers, scikit-learn, pandas
- Code
- github.com/M-Rodani1/bert-financial-sentiment
The model that scored 81.5% accuracy missed 71% of negative sentences. The model that scores 79.4% catches 87% of them.All figures are from one 80/20 validation split; see Limitations.
Problem
Classify short financial sentences as negative, neutral or positive. The dataset is imbalanced — only one sentence in seven is negative — and the costs are asymmetric. For anything downstream of a sentiment signal (screening news, flagging holdings, prioritising what a person reads), a missed negative is expensive and a false alarm is cheap. A model that quietly ignores the minority class can still post a respectable accuracy, which is exactly what happened first.
Data and split
5,842 sentences labelled by sentiment, drawn from financial news and micro-blog text, loaded from a single CSV. A random 80/20 split with random_state=42 gives 4,673 training and 1,169 validation sentences. The split is not stratified, so the validation class mix (175 negative, 622 neutral, 372 positive) differs slightly from the overall proportions. Class weights are computed from the training labels only.
What was wrong with the first model
The original fine-tune used plain cross-entropy and reported 81.5% accuracy. Per-class metrics — added afterwards, which was the real mistake — showed negative recall of 28.6% and negative F1 of 0.41. With 54% of examples neutral, the cheapest way to reduce average loss is to become good at neutral and positive and treat negative as noise. Accuracy rewarded that.
Method
- Weighted loss. Cross-entropy with per-class weights inversely proportional to training frequency:
[2.27, 0.62, 1.05]for negative, neutral, positive. - Early stopping. Validation loss monitored each epoch, patience 3, minimum improvement 0.001; the best checkpoint is restored.
- Optimisation. AdamW at 2 × 10⁻⁵, linear warm-up over the first 10% of steps then linear decay, batch size 8, sequence length 128, up to 10 epochs.
- Evaluation. Precision, recall and F1 per class on the validation split after every epoch, not just accuracy.
| Epoch | Train loss | Validation loss | Decision |
|---|---|---|---|
| 2 | 0.375 | 0.447 | Best — saved |
| 3 | 0.265 | 0.567 | Worse |
| 4 | 0.207 | 0.628 | Worse |
| 5 | 0.189 | 0.745 | Stopped, reverted to epoch 2 |
Training loss kept falling while validation loss rose from epoch 3: with 4,673 examples a 110M-parameter model overfits within a few passes, so the stopping rule is doing real work rather than decoration.
Results
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| Negative | 52.8% | 87.4% | 65.8% | 175 |
| Neutral | 94.3% | 71.2% | 81.1% | 622 |
| Positive | 81.2% | 89.3% | 85.0% | 372 |
| Accuracy | — | — | 79.4% | 1,169 |
| Metric | Original | Improved | Change |
|---|---|---|---|
| Negative recall | 28.6% | 87.4% | +58.8 pts |
| Negative F1 | 0.41 | 0.66 | +0.25 |
| Overall accuracy | 81.5% | 79.4% | −2.1 pts |
The trade-off
Negative precision is 52.8%: roughly half of the sentences flagged negative are not. Neutral recall fell to 71.2% because borderline neutral sentences now get pulled towards negative. That is the intended direction of the trade — the weights move the decision boundary so that the model errs on the side of flagging — but it is a trade, and the right operating point depends on what sits downstream. As a screen feeding a person or a second-stage model, high recall with moderate precision is what you want. As an automated signal acting on its own, it would generate too many false alarms, and the answer would be threshold tuning on the negative logit rather than heavier class weights.
The general lesson is the one every imbalanced-classification project relearns: aggregate accuracy hides which class is being served, and the metric to optimise is a decision about costs, not a default.
Limitations
One random split. The same 1,169 sentences were used for early stopping and for the reported metrics, so the numbers are mildly optimistic; there is no locked test set. The repository lists k-fold cross-validation as future work, and I have labelled the split "validation" here rather than "test" for that reason.
The split is unstratified, and 175 negative examples is a small denominator: the 87.4% recall is 153 of 175 sentences, so a handful of examples moves it by a point.
The base model is general-domain BERT. A FinBERT comparison is listed as future work in the repository and has not been run, so nothing here says whether domain pre-training would help.
Sentence-level sentiment is not a trading signal. Nothing in this project tests whether the labels relate to returns, and the dataset mixes sources with different label conventions.
No deployment or latency work was done, and no calibration analysis of the output probabilities exists.
What I would do next
- Lock a stratified test set, then run stratified 5-fold CV on the rest and report per-class metrics with confidence intervals.
- Compare
ProsusAI/finbertandbert-base-uncasedunder identical training, which is the experiment the current claim quietly assumes. - Sweep the negative-class decision threshold and publish the precision–recall curve so the operating point can be chosen per use, instead of baking it into the loss.
- Error analysis by source and by sentence length; the micro-blog sentences are likely where the false negatives live.