Crash on calling `poseModel?.predict` function for uploaded model

A brief summary of the issue you’re encountering:


I get this EXC_BREAKPOINT(code=1) error when I use my custom mlmodel in the app.

I was wondering what I am missing? Why does the predict function crash?

  • The version of the Fritz SDK, testing device, iOS version, and Xcode version. (e.g Fritz v5.3.1, iPhone 11 on iOS 13.4, Xcode 11.4):
    Fritz iOS SDK v5.3.7
    iPhone XR
    iOS 13.5
    Xcode 11.5

  • Any relevant code snippets that exemplify the issue:

I have the following code for the uploaded model:

extension MobilenetV2: SwiftIdentifiedModel {
   static let modelIdentifier = "<secret code>"
   static let packagedModelVersion = 1
}

I am using the same demo code as in the iOS testing examples, but with this minor change:

lazy var poseModel = FritzVisionPosePredictor<BodySkeleton>(
model: MobilenetV2()
)

This is my skeleton

public enum BodySkeleton: Int, SkeletonType {

public static let objectName = "body"

case nose
case leftEye
case rightEye
case leftEar
case rightEar
case leftShoulder
case rightShoulder
case leftElbow
case rightElbow
case leftWrist
case rightWrist
case leftHip
case rightHip
case leftKnee
case rightKnee
case leftAnkle
case rightAnkle
}
  • Any important or relevant context about the code snippet and/or the solution you’re seeking:

I am using the OpenPose trained model posenet_v2 converted to mlmodel format from a frozen pb file taken from here - https://github.com/ildoonet/tf-pose-estimation/tree/master/models/graph/mobilenet_v2_large.

The mlmodel output format may not be as expected with a string to double dictionary and classLabel as the two outputs (this is how tfcoreml converted the pb file). I’ve got an output vector of weird size too in the pb model (1x38x38x57).

  • Any relevant error messages/logs:

Thanks for the help!

Hello! First off, thank you for the very detailed, easy to read error report.

The issue here is that your pose model architecture is not compatible with the Fritz SDK. Our SDK is designed to work with custom pose estimation models trained with the Fritz AI Studio. Specifically, the output tensors of the model must match what the SDK is expecting so that the proper post-processing can be applied. The model you have converted from Open Pose is a different architecture and is thus incompatible with the SDK.

You can still use Fritz AI to manage your model (ship new versions, collect performance data, etc), but you will not be able to use the FritzVisionPosePredictor. Instead, you can follow this guide to initialize your Core ML model with Fritz and implement the proper post-processing for Open Pose models separately.

Alternatively, you can train your own Fritz compatible pose estimation model with Fritz AI Studio.

I hope this helps!

What is the required output shape for a Fritz CoreML model? Also what are the output feature names? Thanks in advance!