With regard to the scale of the financial transactions and the extent of the healthcare industry, it is one of the ideal systems for fraud. Therefore, suitable identifying fraud data is still one of the challenges facing the healthcare providers, although there are several fraud detection algorithms. In the paper, the BIRCH clustering algorithm, as one hierarchical clustering algorithm, is hybridized with a chemical reaction optimization algorithm (CRO). The BIRCH with linear time complexity is able for clustering large scale data and identifying their noises and the CRO, as one of new meta-heuristic algorithm inspired by the chemical reactions in the real world, explores the search space with a dynamic population size based on four reactions such as on-wall ineffective collision, decomposition, inter-molecular ineffective collision and synthesis. Due to the improved BIRCH-CRO removes the internal clustering process of the classic BIRCH and determines the optimal values of its main parameters, it causes that the computational time decreases and accuracy and precision of detecting fraud data increase since its experimental results is compared with the exist unsupervised algorithms. Also, the proposed fraud detection algorithm has the ability to perform on online data and large scale data, and given the obtained results, it provides a proper performance.