Overview


In order to cultivate E. coli in the laboratory, it takes at least 8 hours to cultivate the bacteria in the culture dish, and after that, it must be taken out at a fixed time to measure the OD value in order to know the growth cycle of the bacteria, which is costly in terms of time and labor. Our device uses camera platform + deep learning + Google spreadsheet + LineBot to automatically detect and record the results of the E. coli culture dish, and can remind the experimenter to collect the bacteria at the growth cycle of E. coli required by the experimenter.

Model Training


Data Preparation

The 250 photos taken by phone are used as database, where the ratio of training set, test set and validation set is about 8:1:1, and the number is 220:15:15 respectively; Lablme is used to label the colonies in the data set, and the colonies that can be used are labeled as colony, and the colonies that are fused or contacted together cannot be used, so they are labeled as wcolony.

Fig. 1) Separate colonies into two categories

Fig. 2) picture set

Fig. 3) Labeling with labelimg

Adjustment of parameters

The reason for using Yolov5m is that the model of Yolov5s is smaller, and no matter how to adjust the parameters, the accuracy will not be greater than 85%; if we use a larger model such as Yolov5x, there will be problems of overfitting and slow running speed, so we choose Yolov5m with a moderate model size. Epochs was set to 200 because after several tests, we found that the accuracy and loss were converged around this position, and more tests would not increase the accuracy. We found that increasing the resolution does not increase the accuracy, so we can improve the training speed by decreasing the resolution.

Fig. 4) The accuracy of the model

Fig. 5) Demonstration of the model in the validation set

Model training

The training is performed using the above model tuning parameters and the completed weights are transferred to the notebook computer for use.

Detecting System


The camera automatically takes pictures, the frequency of taking pictures is 15 seconds once, and then the Arduino rotating platform rotates 60 degree. The photos are saved to the set folder, and the colonies on the plate would be detect by our well trained model. After data processing, the data will be uploaded to GoogleSheet, and it could be return to the users by LineBot.

Device Demonstration

Currently, our LineBot presents experimental datas and URL for our wiki and community websites to users, and we are still developing more advanced functions.

Results


Model Accuracy

From the graph of PR curve, we can see that the map of this model is 0.669, and from the graph of P_curve, we can see that the best accuracy of this model is around 90%, and in the graph of confusion_matrix, we can notice that the accuracy of prediction in colony is higher, but the accuracy of prediction in wcolony is lower only 0.58. This is probably due to the lack of data in wcolony and the fact that there is a whole tray full of colony photos in the test set, which may cause its accuracy to decrease.

Actual Use Case

In the actual use scenario, the whole system has been tested by us and works well. The coordinates recorded on the spreadsheet, after the conversion of the scatter diagram and the original photo, the colony locations are quite accurate, but the effect is not good in the case of too many colonies; the RGB value is not much different from the actual color after the check; the Arduino can also control the turntable normally to achieve the effect of monitoring several dishes at the same time; the colony size is determined by the box size, so there may be some errors. Linebot can warn users normally, and can also use additional functions such as IGEM team website and IGEM website normally, but there is a formatting problem in the data return part, and more adjustment should be made.

Conclusion


Our device verified that it is feasible to accelerate the experiment and reduce the manpower for the purpose of cultivating E. coli in a laboratory environment, but there are many areas that need to be optimized, such as:

  1. Increase the accuracy of the system:
    The number of photos in the database can be increased to improve
  2. Increase the accuracy of colony size detection:
    We may need to use models with higher accuracy such as mask-rcnn.
  3. Improve the experience of using LineBoT:
    Collect more user feedback to update
  4. Google spreadsheet uploading problem:
    Build a lab-specific database to replace it.
  5. Miniaturization of the system body:
    The whole system is currently using laptops as the carrier for computing and cameras, but later we can use jetson nano or industrial computers to reduce its size and shrink the model to achieve the purpose of miniaturization.

Based on the above points, we hope to optimize this system in the future and make it into a complete system that can be used in a laboratory environment to speed up laboratory experiments and simplify the experimental process.


    Reference

  1. V. Sittakul, W. Khotwongsa, Y. Poolthep and S. Pasakawee. (2019). On-site Data Storage via Website or LineBOT. IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), 409-412. https://ieeexplore.ieee.org/document/8953086
  2. Shamash, M., & Maurice, C. F. (2021). OnePetri: Accelerating Common Bacteriophage Petri Dish Assays with Computer Vision. PHAGE (New Rochelle, N.Y.), 2(4), 224–231. https://doi.org/10.1089/phage.2021.0012
  3. Moyeed Abrar. (2019). Interfacing a Servomotor with Arduino uno Microcontroller. https://1library.net/document/qvrdgj0y-interfacing-a-servomotor-with-arduino-uno-microcontroller.html
  4. Python official website. https://www.python.org/
  5. ultralytics. yolov5 repository on github. https://github.com/ultralytics/yolov5
  6. pytorch official website. https://pytorch.org/
  7. Arduino official website. https://www.arduino.cc/
  8. heartexlabs. labelImg repository on github. https://github.com/heartexlabs/labelImg
  9. Visual Studio Code official website. https://code.visualstudio.com/
  10. Heroku: Cloud Application Platform. https://devcenter.heroku.com/
  11. Line developers official website. https://developers.line.biz/en/
  12. Connecting Google Sheet with crawled data using python. https://www.learncodewithmike.com/2020/08/python-write-to-google-sheet.html
  13. Google Cloud. https://console.cloud.google.com/welcome?project=fast-metric-57015&supportedpurview=project