One of the most exciting intersections of our technology today is that of augmented reality (AR) and machine learning (ML). Today, we'll dive into a practical example of how we can use Apple's ARKit and Core ML to build a simple, yet effective danger detector.
Our app is built using SwiftUI, and its primary view is a simple AR view. It uses the ARView from RealityKit to provide a camera feed for AR.
import SwiftUI
import RealityKit
import ARKit
import Vision
struct ContentView : View {
var body: some View {
ARViewContainer().edgesIgnoringSafeArea(.all)
}
}
The main logic of our application is handled in the ARViewController. We initialize the AR view, load the Core ML model (we're using MobileNet for object detection), and setup the Vision request.
class ARViewController: UIViewController, ARSessionDelegate {
var arView: ARView!
var visionRequests = [VNRequest]()
var classificationLabel: UILabel!
let dispatchQueueML = DispatchQueue(label: "com.hw.dispatchqueueml")
override func viewDidLoad() {
super.viewDidLoad()
// Setup ARView
arView = ARView(frame: self.view.frame)
.view.addSubview(arView)
classificationLabel ()
classificationLabel.frame (x: , y: , width: .view.frame.width, height: )
classificationLabel.numberOfLines
classificationLabel.textAlignment .center
classificationLabel.textColor .white
classificationLabel.backgroundColor .black.withAlphaComponent()
.view.addSubview(classificationLabel)
configuration ()
arView.session.run(configuration)
model (for: ().model) { }
classificationRequest (model: model, completionHandler: .handleClassification)
.visionRequests [classificationRequest]
loopCoreMLUpdate()
}
We continuously update the Core ML model with data from the AR view. This is handled in loopCoreMLUpdate() and updateCoreML() functions.
func loopCoreMLUpdate() {
dispatchQueueML.async {
self.updateCoreML()
self.loopCoreMLUpdate()
}
}
func updateCoreML() {
guard let pixelBuffer = arView.session.currentFrame?.capturedImage else { return }
var requestOptions:[VNImageOption : Any] = [:]
if let cameraIntrinsicData = CMGetAttachment(pixelBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil) {
requestOptions = [.cameraIntrinsics: cameraIntrinsicData]
}
let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up, options: requestOptions)
do {
try imageRequestHandler.perform(self.visionRequests)
} catch {
print(error)
}
}
The classification happens in handleClassification(). We handle any error, get the results from the observations and classify them based on the objects. We have a pre-defined set of dangerous objects that if detected will change the label to red and display "Danger", otherwise it will display "Safe" in green.
func handleClassification(request: VNRequest, error: Error?) {
guard let observations = request.results else {
print("no result")
return
}
let dangerousObjects = ["knife", "meat cleaver", "cleaver", "chopper", "hatchet", "ax", "switchblade"]
let classifications = observations
.compactMap({ $0 as? VNClassificationObservation })
.filter({ $0.confidence > 0.3 }) // confidence level - higher more accurate
.map({ ($0.identifier, $0.confidence) })
let dangerousClassifications = classifications.filter {
dangerousObjects.contains(where: $0.0.lowercased().contains)
}
let status: String
let color: UIColor
if !dangerousClassifications.isEmpty {
status = "Danger"
color = .red
} else {
status = "Safe"
color = .green
}
DispatchQueue.main.async {
let classificationTexts = classifications.map { }
.classificationLabel.text classificationTexts.joined(separator: )
.classificationLabel.textColor color
}
classification classifications {
()
}
}
}
: {
(: ) -> {
()
}
( : , : ) {}
}
Safe Snake:

Dangerous Knife:

Using ARKit and Core ML, we've built a simple but efficient danger detector. This is a very rudimentary example of how powerful and versatile these tools can be when combined. In future projects, you might consider training your model on a wider array of dangerous objects, or even using a more powerful model for better accuracy. The possibilities are virtually limitless with this tech stack!
Please consider Buying Me A Coffee. I work hard to bring you my best content and any support would be greatly appreciated. Thank you for your support!