Making Language Model for Hate Speech Type Determination

Hate Speech Detection

Date: 2023.07.27 ~ 2023.08.04

Writer: 9tailwolf


Introduction

There is a many way to classify type of hate speech. I choose some I set some criteria for classify sentences.

  • Gender
  • Age
  • Obscenity
  • Insult
  • Disability
  • Political
  • Religion
  • Race/Region
  • Job
  • Violence

In this page, I will made Language Model for classify above conditions.


Base Model

In the previous page, I tested the models that suitable for sentence classification(hate speech detection). As a result, CNN++2 Model is the best model for classification, witch consist of CNN with 3 hidden BERT layers. Below is a CNN++2 Model.

class KcELECTRA_CNN3(nn.Module):
    def __init__(self, len_size, kernal, filter_size, stride, labels, active):
        super(KcELECTRA_CNN3,self).__init__()
        self.KcELECTRA = AutoModel.from_pretrained("beomi/KcELECTRA-base", output_hidden_states=True)
        self.Dropout_default = nn.Dropout(0.1)
        self.Convs = nn.ModuleList([nn.Conv2d(in_channels = 3, out_channels = filter_size, kernel_size = (i,768), padding = ((i-1)//2,0)) for i in kernal])
        self.Relu = nn.ReLU()
        self.Pooling = nn.MaxPool1d(kernel_size=len_size//stride, stride = stride, padding = (len_size//stride-1)//2)
        self.Flat = nn.Flatten()
        self.Linear = nn.Linear(len_size // stride * len(kernal) * filter_size , labels)
        self.Active = active
            
    
    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
        output = self.KcELECTRA(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)[1][-3:]
        output = torch.transpose(torch.cat(tuple([t.unsqueeze(0) for t in output]), 0),0,1) # batch * 3 * encoder * 768
        x = [self.Relu(self.Dropout_default(Conv(self.Dropout_default(output)).squeeze(3))) for Conv in self.Convs]
        pool = torch.cat([self.Pooling(self.Dropout_default(i)) for i in x],1)
        flat = self.Flat(self.Dropout_default(pool))
        l = self.Linear(self.Dropout_default(flat))
        return self.Active(l)

I just set a parameters(len_size, kernal, filter_size, stride, labels) for below models.


Hate Speech Detection Model

Previous page is a model for finding hate and offensive. I use that model.

I use Selectstar Open Datasets and Kocohub Datasets for training model. Selectstar datas are consist of 100,000 labeled text datas without netural datas. For the training dataset, clean sentences from Kocohub data, non-target hate data from Selectstar were labeled as 0, and 4000 targeted hate data from Selectstar were labeled as 1. Following is the structure of a training set.

  • Gender : 4514 target, 4000 non-target hate, 3486 clean data (98% of Accuracy)
  • Age : 2135 target, 3379 non-target hate, 3486 clean data (96% of Accuracy)
  • Obscenity : 2398 target, 4116 non-target hate, 3486 clean data (93% of Accuracy)
  • Insult : 4514 target, 4000 non-target hate, 3486 clean data (91% of Accuracy)
  • Disability : 1792 target, 2722 non-target hate, 3486 clean data (95% of Accuracy)
  • Political : 4514 target, 4000 non-target hate, 3486 clean data (91% of Accuracy)
  • Religion : 1177 target, 1337 non-target hate, 3486 clean data (99% of Accuracy)
  • Race/Region : 4514 target, 4000 non-target hate, 3486 clean data (92 of Accuracy)
  • Job : 4514 target, 4000 non-target hate, 3486 clean data (88% of Accuracy)
  • Violence : 4514 target, 4000 non-target hate, 3486 clean data (89% of Accuracy)

The following is a parameters for Hate Speech Detection Model.

  • len_size : 50
  • kernal : (1,3,5)
  • filter_size : 12
  • stride : 8
  • labels : 1
  • active : nn.Sigmoid


Additional : Sentimental Analysis Model

Sentimental Analysis is a good for knowing the atmosphere of community. I thought it is necessary to check the sentiment of the sentence.

I use AI Hub for training model.

Data is consist of 43991 sentence with 7 labels, netural, angry, sadness, disgust, surprise, fear, and happiness. When I Checking the data, the number of labels is unbalanced. So I deleted 6000 of netural sentences, and 8000 of sadness sentences. As a result, I use 29991 datas for training model.

The following is a parameters for Sentimental Analysis Model.

  • len_size : 50
  • kernal : (1,3,5)
  • filter_size : 12
  • stride : 8
  • labels : 7
  • active : nn.LogSoftmax(dim=1)