Tuberculosis (TB) is an infectious disease caused by the bacteria Mycobacterium tuberculosis or simply M. tuberculosis. It is primarily an infection of the lungs, but it can also affect other parts of the body. TB is one of the leading causes of death in developing countries, although most are preventable if diagnosed early and treated. Among the available tools, Sputum smear microscopy is widely used for TB diagnosis. rnManual TB screening is tedious work and prone to error due to workload and a dearth of properly trained technicians, manual recognition of the bacillus from the microscopic image takes a long time and requires expert handling of the equipment for the TB identification. rnTo overcome the manual detection issues and develop an automatic TB diagnosis model, we used deep neural networks. We proposed an automatic TB diagnosis and segmentation model composed of Mask R-CNN, Hungrain Algorithm, and Hard example mining for the microscopic image. The proposed model works in a sequential manner where it first detects, classifies, and segments the bacillus objects then the Hungarian Algorithm and Hard example mining is used to further enhance the performance and overcome the problem of high False Positive rate. rnWe carried out experiments to evaluate the performance of our proposed model, we used the metrics of recall, precision, and F-score. We collected the sputum images ZNSM-iDB dataset which is publicly available dataset in the internet and used it for both training and testing. Our experimental results show values of 99.25%, 91.04%, 94.96% for recall, precision, and f-score respectively. which is a significant improvement by the proposed approach compared to existing methods, thus helping in more accurate disease diagnosis.