IMPROVING ADAPTABILITY OF DECISION TREE FOR MINING BIG DATA

Big data has become a popular research topic since the data explosion in the past decade. An efficient analytical methodology provides a way of discovering the potential value from big data. Sampling technique is unsuitable any more that the full data will tell the truths. To this end, the data mining algorithm shall be robust to imperfect data, which may lead to tree size explosion and detrimental accuracy problems. In this paper, we propose an incremental optimization mechanism to solve these problems. The mechanism is called Optimized Very Fast Decision Tree (OVFDT) that possesses an optimized node-splitting control mechanism. Accuracy, tree size and the learning time are the significant factors, which contribute to a much-improved algorithm in performance. Naturally a bigger tree size takes longer computation time. OVFDT is a pioneer model equipped with an incremental optimization mechanism that seeks for a balance between accuracy and tree size for data stream mining. OVFDT operates incrementally by a test-then-train approach. Two new methods of functional tree leaves are proposed to improve the accuracy that the tree model makes a prediction for a new data stream in the testing phase. The optimized node-splitting mechanism controls the tree model growth in the training phase. The experiment supports our claim that OVFDT is an optimal tree structure in both numeric and nominal datasets.