Detecting the Primary User Emulation Attack Using the Logistic Regression and MLE

Abstract— Cognitive radio technology aims at solving the problem of the spectrum scarcity by enabling secondary users to access the idle frequency channel without causing harmful interference to the primary user. However, this technology is subject to numerous severe cyber-attacks that threaten its security. The primary user emulation attack is one of these cyber-attacks in which a malicious user emulates the legitimates primary user signal to prevent other secondary users from accessing the vacant frequency channel. This attack is attracting a considerable interest due to its negative impact on the network. By exploiting the advances in the machine learning area, we propose in this paper the logistic regression with Maximum Likelihood Estimation and gradient ascent to detect and defend against such an attack. This supervised algorithm is trained and extensively tested on a dataset generated based on a simulated primary user emulation attack. Our findings show that the proposed algorithm can detect effectively this attack and it outperforms some relevant techniques existing in the literature, such as Support Vector Machine and Artificial Neural Network, with a probability of detection of 99.5% and a probability of false alarm of 0.6%.

Keywords—Cognitive radio; primary user emulation attacker; Logistic regression; Maximum Likelihood Estimation; Support Vector Machine; Artificial Neural Network

I.     Introduction

Cognitive radio (CR) is an emerging technology which addresses the problem of spectrum scarcity by enabling secondary users (SUs) to use the vacant frequency channels without causing interference to the primary users [1, 2]. However, the CR network is exposed to a wide variety of severe attacks that can disturb the normal network operations. One of these attacks is the primary user emulation (PUE) attack. In this attack, the attacker mimics the legitimate primary user (PU) signal to prevent other secondary users from accessing the idle frequency channel [3, 4]. There two types of the PUE attack, selfish and malicious. In the former, the attacker emulates the PU signal to prevent other legitimate SUs from accessing the channel and uses it selfishly to transmit its signal. In the latter, the attacker mimics the PU signal and prevent purposelessly other SUs from using the free channel [5, 6].

Due to its harmful impact on the CR network, several approaches have been proposed to detect and cope with such an attack [7-22]. For instance, some authors have proposed energy detection and feature-based techniques such as autocorrelation and matched filter. In [11], the authors proposed a detection technique based on energy-detection. In their approach, secondary users measure the power level of the received signal and compare it to that from a legitimate PU. Thus, they can verify the source of the signal and detect any potential PUE attacker. In [12], the authors proposed a detection technique based on the characteristics of wireless channels. Since the statistical property of a wireless channel between a transmitter and a receiver is unique, this feature is used as a radio fingerprint to detect the primary user emulation attacker.

Other techniques have been proposed to detect the PUE attacker by estimating its position and comparing it to the known position of the PU [13-17]. For example, the authors in [13, 14] proposed a time difference of arrival (TDoA) based approach to estimate the position of the PUE attacker. Using the time elapsed between the transmission of the signal and the reception of the reply, the location of the transmitter can be estimated and compared to the known position of the PU then used to decide if the transmitter is a legitimate PU or a PUE attacker. In another paper [16], the authors proposed a detection technique based on the trilateration and the received signal strength. In their localization-based approach, the authors use the particle swarm optimization technique to improve the estimated location of the transmitter and increase the probability of detection of the PUE attacker.

Although these techniques can detect the PUE attacker, they suffer from a number of limitations. For instance, the energy-detection technique [11] are not efficient in distinguishing between signal and noise, and also they suffer from a high level of probability of false alarm [18]. In addition, the feature-based techniques [12] are inefficient in identifying effectively the primary user signal characteristic since the noise introduced by the hardware is random. Moreover, TDoA-based technique [13] requires a tight synchronization between the transmitters and the receivers, which is challenging. Furthermore, the technique proposed in [15] is based mainly on the received signal strength that can be impacted by the noise and the multipath phenomenon, which can lead to inaccurate results.

By taking advantages of the advances in the machine learning area, some authors have applied these techniques to further increase the detection accuracy of the PUE attacker and decrease the probability of false alarm. For instance, several Support Vector Machine (SVM) based techniques have been proposed [19-21]. In [19], the authors proposed an SVM based approach for classifying the source of the incoming signal and detecting any potential attacker. For training their proposed model, the authors generated a dataset based on a simulated PUE attack. Using the received signal strength and the position of the PU, SUs can verify the source of the incoming signal so they can decide if the transmitter is an attacker or a legitimate PU. After the training phase, the SVM based algorithm defines boundaries around the PU location, thus any signal coming from a transmitter which is located outside those boundaries is considered as an attacker.

Similarly, the authors in [20] proposed a technique based on zoom-FFT (Fast Fourier Transform) and support vector data description (SVDD) which is inspired by SVM. In their approach, zoom-FFT is used to estimate the accuracies of pilot and symbol rate as a radio-frequency fingerprint (RFF), then the SVDD classifier is used to construct a boundary around the PU objects, which are the RFF, and enclose these objects within a minimum sphere. Thus, SVDD can sense the fine differences of the RFF then distinguish between the PU attacker signal and that of the PU.

Another technique based on one-class SVM has been proposed in [21]. In their paper, the authors suggested an anomaly detection framework for the cognitive radio network. They divide the problem of detecting the PUE attack into two sub-problems. The first case is where no authorized signal is present and the second one is where an authorized signal is present. In the first case, they used the existing cooperative sensing schemes, including the hard and soft decision combining, to investigate the impact of signal path loss on their detection performances. In the second case, they proposed three approaches for detecting anomalous signals by making use of the characteristics of radio propagation. These approaches are linearity check-given-location, one-class support vector machine, and calibrating power.

Other authors have proposed Artificial Neural Network (ANN) based techniques for detecting the PUE attack. For example, the authors in [22] proposed an approach using energy detection, cyclostationary calculation, and ANN algorithm. In their approach, the existing users in the frequency band are localized using the energy detection. Next, the cyclostationary calculation technique is used to represent the features of the users’ signal, and then these features are fed to the neural network for detecting the PUE attack. In addition, the authors in [23] used the artificial neural network with the radio-frequency fingerprinting for detecting the PUE attack. Their approach is based on two steps: training and testing the ANN. During the training step, a radio-frequency fingerprint profile is created for each transmitter using its signal, then a three-layer neural network classifier is generated. In the testing step, a radio-frequency fingerprint is created as a test signal, then it is fed to the neural network before comparing it to the established profiles of the transmitter.

Although these SVM and ANN based techniques produce satisfactory results in terms of detecting the PUE attack, our proposed approach outperforms these techniques with a high probability of detection and a low probability of false alarm. In addition, the framework proposed in [21] for detecting unauthorized spectrum usage in a cognitive radio network is designed to detect anomalies in general but it does not consider specific attack such as the primary user emulation attack. Moreover, the technique proposed in [23] shows satisfactory results in the ad hoc cognitive radio network, but the main drawback of this technique is that it is based on the radio-frequency fingerprinting, which can be impacted by the noise, and it can lead to producing inaccurate results.

In this paper, we propose logistic regression with the Maximum Likelihood Estimation and the gradient ascent as a machine learning algorithm for detecting the primary user emulation attacker. Logistic regression is a supervised machine learning algorithm widely used for classification and especially in binary classification. To the best of our knowledge, this algorithm has never been used before for detecting the primary user emulation attacks in a cognitive radio network. For training and testing this algorithm, we propose and simulate a primary user emulation attack model in order to generate the required dataset with the appropriate features. The proposed technique is compared to two relevant machine learning algorithms existing the literature which are Support Vector Machine (SVM) and the Artificial Neural Network (ANN). This comparison is based on several metrics including the probability of detection, the probability of false alarm, the probability of miss detection, the accuracy.

The remainder of this paper is organized as follows: section II describes the proposed approach based on logistic regression, Maximum Likelihood Estimation, and the gradient ascent. In addition, it presents the attack model adopted for simulating and detecting the PUE attack. Section III gives the features generated and used for detecting the PUE attack, in addition to the metrics used for comparing the proposed approach with SVM and ANN based techniques. The simulation results are given and discussed in the same section. The last section draws some conclusions and future work.

II.    Methodology

A.    Proposed Approach

Detecting the primary user emulation attack can be approached as a classification problem, more specifically as a binary classification problem. When the signal is emitted by a legitimate primary user, then this event is classified as a normal event. Otherwise, if the signal is transmitted by an attacker, this event can be classified as an attack. In this paper, we select the logistic regression as a classifier with Maximum Likelihood Estimation and gradient ascent. Logistic regression (LR) is a supervised machine learning algorithm used for classification, especially for binary classification [24, 25]. Let

• j
• dpu_suj= distance between the

  PUand a given

  SUj

• dt_suj= distance between a transmitter

  tand a given

  SUj

• i=integer between 1 and

  N

• j=integer between 1 and

  SUs

• // Normal Event: the transmitter is the PU, there is no PUE attack
• TPos= PUPos
• for

  ifrom 1 to

  N/2

• for each secondary user in

  SUs

• Calculate

  dpu_suibetween

  SUPosjposition and the

  PUPos

• Calculate

  dt_suibetween

  SUPosjand

  TPosusing the Free Path Loss equation

• If (

  |dpu_sui-

  dt_sui|>=

  τ) then

  1. The transmitter is an attacker
• else
  1. The transmitter is the legitimate PU
• end for
• end for
• // Attack Event: the transmitter is a PUE attacker, there is a PUE attack
• TPos=

  PUEPos

• for

  ifrom

  (N/2)+1 to

  N

• for each secondary user in

  SUs

• Calculate

  dpusuibetween

  SUPosjposition and the

  PUPos

• Calculate

  dt_suibetween

  SUPosjand

  TPosusing the Free Path Loss equation

• If (

  |dpu_sui-

  dt_sui|>=

  τ) then

•           The transmitter is an attacker
• else
•           The transmitter is the legitimate PU
• end for
• end for
• end

In this case, the PUE attacker is emulating the PU signal features to prevent other SUs from accessing the idle frequency channel. During each event, each SU can independently detect the PUE attack by computing the distance between the comparing the distance between its own position and that of the PU, then compare it to the estimated position of the transmitter. The distance,

dpu_sui,between the position of a secondary user

iand the PU position is calculated using the following equation:

Where

(xp,yp)are the position coordinates of the primary user and

(xsui,ysui)are the position coordinates of a secondary user

i. Based on the received power of the transmitter, each secondary user can verify the source of the incoming signal using the Free Path Loss formula, which is given by:

Where

Pris the received power,

Ptis the transmitted power,

Gris the antenna gain of the receiver,

Gtis the antenna gain of the transmitter,

λis the wavelength, and

dt_suiis the distance between a secondary user

iand the transmitter

t. The equation (14) can be expressed in dB as:

Since the received signal can be subject to the noise in a real wireless network environment, the equation (15) can be written as:

Where

niis a white Gaussian noise which follows the normal distribution

N(0, σ2). Using the equation (16), the distance,

dt_sui, can be estimated as:

Where

If the difference between

ddt_suiand

dpu_suiis greater or equal to a predefined threshold

τthen the transmitter is an attacker, otherwise, it is a legitimate primary user.

Based on this model a number of features have been purposefully selected to help SU in detecting effectively the PU and they are described in Table I. As it can be seen, these features include the coordinates of the users, for instance, the coordinate

x,yof the PU and the PUE attacker. They include also the noisy received power at each SU and the estimated distance between the users.

III.   Results and discussion

Based on the proposed primary user emulation attack model, 2000 instances were generated for training and testing the proposed classifier. Each instance has 29 features as inputs and one output class. Based on this dataset, the proposed approach is compared with two relevant algorithms existing in the literature used for detecting the primary user emulation attack which are Support Vector Machine (SVM) and Artificial Neural Network (ANN). For the SVM algorithm, several kernels are selected including linear, polynomial, Radial Basis Function (RBF), polynomial, and sigmoid kernel [27]. Regarding the ANN, two types of network are selected, the first one with one hidden layer and 100 neuron nodes, and the second one with two hidden layers each one with 100 neuron nodes.

A.    Metrics

For evaluating the performance of the proposed approach along with SVM and ANN, several metrics. These metrics include the probability of detection, the probability of miss detection, the probability of false alarm, and the accuracy.

The probability of detection,

Pd,corresponds to the number of times where the primary user emulation attacker signal correctly classified as an attack divided by the total number of attacks in the data set [28]. It is given by:

Pd= Number of PUE detectionsTotal number of attacks

*100

Regarding the probability of false alarm,

Pfa,it corresponds to the number of times where the legitimate primary user signal is wrongly classified as an attack divided by the total number of normal instance in the data set [28]. It is expressed as:

Pfa= Number of false detected attacksTotal number of normal instances

*100

For the probability of miss detection,

Pm,it corresponds to the number of times where an attack is incorrectly classified as a normal signal divided by the total number of attacks in the testing data set [28]. It is given by:

Pm= Number of miss detections Total number of attacks

*100

The accuracy corresponds to the number of instances that are correctly classified by the total number of instances in the testing dataset [28]. It expressed as:

Accuracy= Number of correclty classfied instances Total number of instances

*100

B.    Results

The proposed approach was implemented using Python [29] and the experiments were conducted on an Intel Core(TM) i7-6700 CPU @ 3.40GHz 3.41 GHz, 16 GB RAM with a windows 10 64-bit Operating system.

To avoid the under-fitting and overfitting issues in the proposed approach along with increasing the accuracy and the probability of detection, we used a regularization parameter (λ) with several values and the results are given in the Figures 1 and 2. Figure 1 depicts the accuracy of the proposed approach as a function of the regularization parameter (λ). As it can be seen from this figure, the accuracy of the proposed approach increases with the increase of the λ value. For instance, when the value of λ is equal to 0.001 the accuracy is equal to 98.5%, and when the value of λ increases to 0.01 the accuracy reaches the peak, which is 99.5%. Figure 2 shows the probability of the detection and the probability of false alarm as a function of the λ parameter. As it can be seen from this figure, the probability of detection increases when the λ parameter’s value increases. For example, when the λ value is equal to 0.001, the corresponding probability of detection is equal to 99.1%, and when λ values rises to 0.02, the probability of detection reaches 99.5%. Figure 2 shows also the probability of false alarm as a function of the λ parameter. As it can be observed, when the λ value increases, the probability of false alarm slightly decreases. For instance, when λ is equal to 0.001, the probability of false alarm is equal to 1.1%, when the λ value increases to 0.01 the probability of false alarm decreases to 0.6% and it remains stable.

For a better overall performance, a cross-validation technique is used to select the number of folds suitable for SVM, ANN, and the proposed approach. With the cross-validation, the dataset is split into k smaller parts (k-folds) and the models are trained on k-1 of folds and tested on the remaining of the data set [30]. In addition, the comparison is conducted with various variance of SVM and ANN. For instance, two types of ANN are selected, with one is hidden layer and with two hidden layer. Both of these types include 100 neuron nodes with the activation function Relu. Regarding SVM, several kernels are selected including linear, Radial Basis function (RBF), polynomial, and sigmoid kernel.  The results of this comparison based on various number of cross-validation folds is given in Figure 3.

Figure 3 shows the accuracy of the proposed approach, the ANN with two different number of hidden layers, and SVM with various kernels, as a function of the number of cross-validation folds. As one can see, the accuracy of the proposed approach changes slightly with the increase of the number of folds. For instance, with 2 folds, the accuracy is equal to 99.3% and it increases to 99.4%, 99.5%, 99.5% with a number of fold corresponding to 3, 5, and 10 folds, respectively. For ANN with one hidden layer, the highest accuracy result, which is 97%, is given by 20 folds. For ANN with two hidden layers, the accuracy reaches the peak, which is 98.7%, with a number of fold corresponding to 10 folds. Regarding SVM with a linear kernel, the highest accuracy values is obtained with 5 folds. For SVM with a Radial Basis function (RBF) kernel, the highest accuracy is obtained with 2 folds and the lowest values is given by 10 folds. For the third type of SVM with a polynomial kernel, the highest accuracy value, which is 54.8%, is achieved with 2 folds. For the last type of SVM with a sigmoid kernel, 2 folds give the highest accuracy which is 85.7%.

After selecting the suitable number of folds for the proposed approach, ANN with two different hidden layer, and SVM with several kernels, a comparison between these models in terms of the accuracy, probability of detection, probability of false alarm, and the probability of miss detection is given in the Table II. The first column of this table shows the probability of detection, as it can be seen, the proposed approach has highest value, which is 99.5%, followed by ANN with two hidden layers, and SVM with the linear kernel which have 98.6% and 97.5%, respectively. From the second column of this table, it can be seen that the proposed approach has the lowest probability of false alarm, which is 0.6% corresponding to 12 miss detected events in the dataset, followed by SVM with the linear kernel, which has 2.5% corresponding to 50 miss detected events, then ANN with two hidden layers, which has 1.3% corresponding to 26 miss detected events. This result suggests that the proposed approach produces less probability false alarm comparing to the other algorithm which helps the SUs in increasing their chance to access the idle frequency channels.

The third column of this table shows the probability of miss detection. as it can be observed, the proposed approach has 0.5%, which is the lowest value, the ANN with two hidden layers has 1.4%, and SVM with linear kernel reports 2.5%. The fourth column of this table shows the accuracy. As it can be seen, the proposed approach produces the highest accuracy, which is 99.5%, followed by ANN with two hidden layers which has an accuracy of 98.7%, and SVM with the linear kernel which has an accuracy of 97.5%. These results suggest that the proposed approach can classify accurately the attack and normal event comparing to ANN and SVM with different types, which helps the SUs in detecting more accurately the primary user emulation attacker.

IV.   Conclusion

In this paper, we propose logistic regression classifier with the Maximum Likelihood Estimation and the gradient ascent as a supervised machine learning algorithm for detecting the primary user emulation attack in the cognitive radio network. For training and testing the model, a dataset is generated based on a primary user emulation attack scenario in which two events are considered, normal event an attack event. Based on these events, several training examples is generated with 29 features. These features are selected purposefully to help secondary user in detecting the primary user emulation attacker more accurately. These features include the coordinate of the primary user position, the coordinate of the primary user emulation attacker position, the coordinate of the secondary users, and the power received signal at each secondary user.  After training the proposed classifier, the regularization parameter is used in order to avoid the underfitting and overfitting issues along with increasing the accuracy and the probability of detection. The proposed approach is compared to the artificial neural network (ANN) with a different number of hidden layers, and with Support Vector Machine (SVM) with several kernels.

The proposed approach is compared to different types of ANN and SVM. For ANN, two hidden layers produces more accurate results than the one with one hidden layer. Regarding SVM, the linear kernel outperforms Radial Basis Function, polynomial, and sigmoid kernel in terms of accuracy, the probability of detection. Comparing the proposed approach with ANN with two hidden layers, and SVM with the linear kernel, the proposed approach outperforms these classifiers in terms of the probability of detection, accuracy, the probability of false alarm, and the probability of miss detection. For instance, the proposed approach reports the highest accuracy corresponding to 99.5%, followed by ANN with two hidden layers which has 98.7% and then SVM with a linear kernel which has 97.5%. Regarding the probability of false alarm, the proposed approach has the lowest value, which is 0.6%, followed by ANN with two hidden layers which reports 1.3%, then SVM with the linear kernel which has a probability of false alarm corresponding to 2.5%. As a future work, a real primary user emulation attacks using the Universal Software Radio Peripheral will be simulated. In addition, other attacks including Jamming attack, and spectrum sensing data falsification attack will be considered to generate a large dataset and develop an anomaly detection system for the cognitive radio network.

References

1. N. Kaabouch, Handbook of Research on Software-Defined and Cognitive Radio Technologies for Dynamic Spectrum Management. IGI Global, 2014.
2. M. T. Masonta, M. Mzyece, and N. Ntlatlapa, “Spectrum decision in cognitive radio networks: A survey,” IEEE Commun. Surv. Tutor., vol. 15, no. 3, pp. 1088–1107, 2013.
3. M. R. Manesh and N. Kaabouch, “Security Threats and Countermeasures of MAC Layer in Cognitive Radio Networks,” Ad Hoc Netw., 2017.
4. A. A. Sharifi, M. Sharifi, and M. J. M. Niya, “Secure cooperative spectrum sensing under primary user emulation attack in cognitive radio networks: Attack-aware threshold selection approach,” AEU-International J. Electron. Commun., vol. 70, no. 1, pp. 95–104, 2016.
5. M. Bouabdellah, N. Kaabouch, F. El Bouanani, and H. Ben-Azza, “Network layer attacks and countermeasures in cognitive radio networks: A survey,” J. Inf. Secur. Appl., vol. 38, pp. 40–49, 2018.
6. W. R. Ghanem and M. Shokair, “Defense against selfish PUEA in cognitive radio networks based on hash message authentication code,” Int. J. Electron. Inf. Eng., vol. 4, no. 1, pp. 12–21, 2016.
7. R. K. Sharma and D. B. Rawat, ‘Advances on security threats and countermeasures for cognitive radio networks: A survey’, IEEE Commun. Surv. Tutorials, 2015, vol. 17, no. 2, pp. 1023–1043
8. R. Yu, Y. Zhang, Y. Liu, S. Gjessing, and M. Guizani, ‘Securing cognitive radio networks against primary user emulation attacks’, IEEE Netw., 2015, vol. 29, no. 4, pp. 68–74
9. D. Pu and A. M. Wyglinski, ‘Primary-user emulation detection using database-assisted frequency-domain action recognition’, IEEE Trans. Veh. Technol., 2014, vol. 63, no. 9, pp. 4372–4382
10. N. T. Nguyen, R. Zheng, and Z. Han, ‘On identifying primary user emulation attacks in cognitive radio systems using nonparametric bayesian classification’, IEEE Trans. Signal Process., 2012, vol. 60, no. 3, pp. 1432–1445
11. F. Jin, V. Varadharajan, and U. Tupakula, “Improved detection of primary user emulation attacks in cognitive radio networks,” in International Telecommunication Networks and Applications Conference (ITNAC), 2015, pp. 274–279.
12. W. L. Chin, C. L. Tseng, C. S. Tsai, W. C. Kao, and C. W. Kao, “Channel-Based Detection of Primary User Emulation Attacks in Cognitive Radios,” in IEEE 75th Vehicular Technology Conference (VTC Spring), 2012, pp. 1–5.
13. W. R. Ghanem, M. Shokair, and M. I. Desouky, “An improved primary user emulation attack detection in cognitive radio networks based on firefly optimization algorithm,” in 33rd National Radio Science Conference (NRSC), 2016, pp. 178–187.
14. O. León, J. Hernández-Serrano, and M. Soriano, ‘Cooperative detection of primary user emulation attacks in CRNs’, Comput. Networks, 2012, vol. 56, no. 14, pp. 3374–3384
15. R. Sultana and M. Hussain, “Mitigating Primary User Emulation Attack in Cognitive Radio Network Using Localization and Variance Detection,” in Proceedings of First International Conference on Smart System, Innovations and Computing, Springer, Singapore, 2018, pp. 433–444.
16. F. W. Fassi, Y. Arjoune, H. El Ghazi, N. Kaabouch, B. A. ElMajd, “A particle swarm optimization based algorithm for primary user emulation attack detection,” Computing and Communication Workshop and Conference (CCWC) IEEE, 2018.
17. J. Wang, J. Chen, and D. Cabric, ‘Cramer-Rao bounds for joint RSS/DoA-based primary-user localization in cognitive radio networks’, IEEE Trans. Wirel. Commun., 2013, vol. 12, no. 3, pp. 1363–1375
18. M. R. Manesh, M. S. Apu, N. Kaabouch, and W. C. Hu, “Performance evaluation of spectrum sensing techniques for cognitive radio systems,” in IEEE 7th Annual Ubiquitous Computing, Electronics Mobile Communication Conference (UEMCON), 2016, pp. 1–7.
19. S. A. Selvi and M. Sundararajan, “SVM based Two Level Authentication for Primary user Emulation Attack Detection,” Indian J. Sci. Technol., vol. 9, no. 29, 2016.
20. Z. Luo, C. Lou, S. Chen, S. Zheng, and S. Li, “Specific primary user sensing for wireless security in IEEE 802.22 network,” in 2011 11th International Symposium on Communications Information Technologies (ISCIT), 2011, pp. 18–22.
21. S. Liu, Y. Chen, W. Trappe, and L. J. Greenstein, “ALDO: An Anomaly Detection Framework for Dynamic Spectrum Access Networks,” in IEEE INFOCOM, 2009, pp. 675–683.
22. D. Pu, Y. Shi, A. V. Ilyashenko, and A. M. Wyglinski, “Detecting Primary User Emulation Attack in Cognitive Radio Networks,” in IEEE Global Telecommunications Conference – GLOBECOM, 2011, pp. 1–5.
23. S. U. Rehman, K. W. Sowerby, and C. Coghill, “Radio-frequency fingerprinting for mitigating primary user emulation attack in low-end cognitive radios,” IET Commun., vol. 8, no. 8, pp. 1274–1284, Feb. 2014.
24. D. W. Hosmer Jr, S. Lemeshow, and R. X. Sturdivant, Applied logistic regression, vol. 398. John Wiley & Sons, 2013.
25. A. Y. Ng and M. I. Jordan, “On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes,” in Advances in neural information processing systems, 2002, pp. 841–848.
26. I. J. Myung, “Tutorial on maximum likelihood estimation,” J. Math. Psychol., vol. 47, no. 1, pp. 90–100, 2003.
27. L. Wang, Support vector machines: theory and applications, vol. 177. Springer Science & Business Media, 2005.
28. I. Almomani, B. Al-Kasasbeh, and M. Al-Akhras, “WSN-DS: a Dataset for intrusion detection systems in wireless sensor networks,” J. Sens., 2016.
29. F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, no. Oct, pp. 2825–2830, 2011.
30.  P. Refaeilzadeh, L. Tang, and H. Liu, “Cross-validation,” in Encyclopedia of database systems, Springer, 2009, pp. 532–538.