Building an AI-based Healthcare Solution

A US-based healthcare center asked us to develop a robust AI-based system that could track ovarian follicles across video frames, detect them in ultrasound images, and measure them. Our team developed an efficient and complex solution that employs artificial intelligence (AI) algorithms. The delivered system has shown astonishing accuracy and helped our client accelerate and simplify their doctors’ routine as well as provide high-quality services to their patients.

Therapists at the client’s organization had to constantly pause ultrasound videos to detect and mark maximum horizontal and vertical diameters of follicles. Later, they had to manually measure the detected follicles. This was an extremely time-consuming process. The client’s challenge was to automatically detect, segment, and measure follicles in ultrasound images. To solve this challenge, our client valued detection accuracy over speed. It was essential to achieve accurate results and detect as many follicles as possible. During the discovery phase, we highlighted a few challenges that needed to be solved before starting system development: Matching pixels with object sizes — We couldn’t build object measurement functionality without knowing the size of one pixel in an ultrasound image. During a consultation with the client, we learned there are hatch marks in all ultrasound images spaced 5 millimeters apart. We found a way to detect these hatch marks and accurately calculate the pixel size prior to creating algorithms for follicle measurement. Researching and preparing specific data — We needed to know how and where follicles are located in images. Also, we required a high-quality dataset containing information that we lack the qualifications to supply. Building an efficient solution was impossible without this dataset. To acquire all the data we needed and gain an in-depth understanding of how doctors work with this data, we had extensive consultations with the client. Then, we summarized requirements for annotating data and our client used these requirements to prepare a highquality dataset. Lack of environment — Another challenge was testing our neural network’s performance without an environment either we or our client had set up.

Since building an AI-based healthcare solution requires thorough preparation, we started with researching the nuances of the client’s challenges. We needed to understand how doctors work with data on follicles and interpret ultrasound images before automating the detection and measurement of follicles. Once we received all the required information from our client, we planned the further development process, paying attention to possible pitfalls. Then, our team moved on to designing and developing the concept for the future system. We developed a complex system that includes: 1. A neural network for follicle detection. 2. Pre- and post-processing modules that use hand-crafted machine learning and computer vision algorithms for follicle segmentation and measurement. 3. A post-processing module that generates reports with analytics and statistics on system precision, recall, etc. The project consisted of three main stages: 1. Preparing data and consulting with the client There were several challenges we needed to handle before actually starting work on the solution concept. The most essential step was preparing data. First, we had to focus on creating a high-quality dataset since the client only had raw data. We explained the importance of data quality to the client and provided them with full guidance regarding how to create a proper dataset. A dataset is a set of pairs — in our case, ultrasound images with annotations for each image, where the annotations contain the information we need to detect with our AI system. Datasets are used to train neural networks and check the accuracy of their results. So to train our client’s system properly, we needed high-quality and consistent images with accurate annotations. Since our clients’ therapists have expertise in analyzing ultrasound images, our client was able to create a perfect dataset. Once the initial dataset was ready, we started development of the AI system for healthcare. Second, we needed to define the way the solution would measure objects in ultrasound images. Doctors manually measure physical parameters such as the diameter and perimeter of follicles. But AI algorithms can’t do this properly, since they process image pixels. Therefore, we needed to know the size of one pixel in an ultrasound image. Our client informed us that every ultrasound image has hatch marks on one side with a distance of 5 millimeters between hatches. 2. Developing algorithms for detecting and measuring follicles The next step was creating the part of the solution to detect follicles. In the AI industry, detecting objects in images is called instance segmentation. During thorough research, we explored the best instance segmentation solutions available and found that the most relevant architecture for our task was a region-based convolutional neural network with a residual network backbone called Mask R-CNN. We developed additional layers on top of Mask R-CNN to perform tracking and measurements on all detected follicles. Then, we implemented an image filtering algorithm to measure objects. It helped us detect the hatch marks and calculate the number of pixels between them. Knowing the size of one pixel allowed us to calculate the size of any object in an image. 3. Implementing additional modules in the system We added several more helpful modules to provide our client with the opportunity to work with different types of data. With these modules, therapists can detect follicles in separate images, folders with image, and videos in various formats. Our additional goal was to create a user-friendly interface. We marked the input to our system as “Video file” and the output as “Detected and measured follicles.” In their traditional workflow, doctors receive data in the form of an ultrasound video and then pause it to mark follicles in images. We automated this process by using a module to automatically split a video into images. Then, we simplified the final concept of a system: one application detects follicles and another measures them. The system saves doctors’ time by eliminating the need to spend lots of time manually examining dozens of images. Thus, doctors can analyze video files quickly, re-check the system’s results, and pay more attention to patients. Our client is currently testing the system in their own fertility treatment center. They are also going to test our solution in more clinics once it’s approved by the United States Food and Drug Administration (FDA). To receive approval, our client needs to pass certification for a medical device. To help them with this process, Our team is documenting all the technical details required for submission to the FDA.