robotic_sdk.ml_models package

Subpackages

• robotic_sdk.ml_models.free_spot_2d package
  • Submodules
  • robotic_sdk.ml_models.free_spot_2d.free_spot module
  • Module contents
• robotic_sdk.ml_models.grasp package
  • Subpackages
    • robotic_sdk.ml_models.grasp.interfaces package
      • Submodules
      • robotic_sdk.ml_models.grasp.interfaces.contact module
      • Module contents
    • robotic_sdk.ml_models.grasp.models package
      • Submodules
      • robotic_sdk.ml_models.grasp.models.contact module
      • Module contents
    • robotic_sdk.ml_models.grasp.utils package
      • Submodules
      • robotic_sdk.ml_models.grasp.utils.geom_utils module
      • robotic_sdk.ml_models.grasp.utils.msg_utils module
      • robotic_sdk.ml_models.grasp.utils.rosbag_utils module
      • robotic_sdk.ml_models.grasp.utils.vis_utils module
      • Module contents
  • Submodules
  • robotic_sdk.ml_models.grasp.collinear_point module
  • robotic_sdk.ml_models.grasp.debug_logger module
  • robotic_sdk.ml_models.grasp.detection module
  • robotic_sdk.ml_models.grasp.geom_utils module
  • robotic_sdk.ml_models.grasp.grasp module
  • robotic_sdk.ml_models.grasp.grasp_2f module
  • robotic_sdk.ml_models.grasp.grasp_pose module
  • robotic_sdk.ml_models.grasp.msg_utils module
  • robotic_sdk.ml_models.grasp.pick_point module
  • robotic_sdk.ml_models.grasp.projection_logger module
  • robotic_sdk.ml_models.grasp.segmentation module
  • robotic_sdk.ml_models.grasp.segmentation_logger module
  • robotic_sdk.ml_models.grasp.smooth_point module
  • robotic_sdk.ml_models.grasp.top_line_point module
  • Module contents
• robotic_sdk.ml_models.grounded_sam_2 package
  • Subpackages
    • robotic_sdk.ml_models.grounded_sam_2.utils package
      • Submodules
      • robotic_sdk.ml_models.grounded_sam_2.utils.build_sam module
      • robotic_sdk.ml_models.grounded_sam_2.utils.mask_dictionary_model module
      • robotic_sdk.ml_models.grounded_sam_2.utils.misc module
      • robotic_sdk.ml_models.grounded_sam_2.utils.sam2_video_predictor module
      • Module contents
  • Submodules
  • robotic_sdk.ml_models.grounded_sam_2.grounded_sam2_tracking module
  • Module contents

Submodules

robotic_sdk.ml_models.anomaly_detection module

class robotic_sdk.ml_models.anomaly_detection.AnomalyDetectionModel(model_name: str, parameters: dict)

Bases: Model

A class representing the anomaly detection model.

__init__(model_name: str, parameters: dict)

Initializes the AnomalyDetectionModel.

Parameters:

• model_name (str) – The name of the model.

• parameters (dict) – A dictionary of parameters for the model.

load_model(model_path: str)

Load the model from the specified path.

Parameters:

model_path (str) – The path to the model file.

Returns:

The instance of the model.

Return type:

self

perform_inference(data_x)

Perform inference using the model.

Parameters:

data_x – The data to perform inference on.

Returns:

The result of the inference.

save_model(model_path: str)

Save the model at the specified path.

Parameters:

model_path (str) – The path to save the model file.

Returns:

The instance of the model.

Return type:

self

train_model(data_x) → None

Train the model with the provided data.

Parameters:

data_x – The training data.

robotic_sdk.ml_models.gpt module

robotic_sdk.ml_models.grounded_segmentation module

class robotic_sdk.ml_models.grounded_segmentation.GroundedSegmentationModel(model_name: str = 'GroundedSegmentationModel', parameters: dict | None = None)

Bases: Model

A class representing the GroundingDINO + Segment Anything Model (SAM).

__init__(model_name: str = 'GroundedSegmentationModel', parameters: dict | None = None)

Initializes the GroundedSegmentationModel.

Parameters:

• model_name (str) – The name of the model.

• parameters (dict) – A dictionary of parameters for the model.

detect(image: Image, labels: List[str], threshold: float = 0.3) → List[DetectionResult]

Use Grounding DINO to detect a set of labels in an image in a zero-shot fashion.

Parameters:

• image (Image.Image) – The input image.

• labels (List[str]) – The list of labels to detect.

• threshold (float) – The detection threshold.

Returns:

The detection results.

Return type:

List[DetectionResult]

load_model()

Load the model from the specified path.

Returns:

The instance of the model.

Return type:

self

perform_inference(image_bgr: ndarray, labels: List[str], threshold: float = 0.3, max_box_area: int = 1000)

Perform inference on the given image to detect and annotate objects.

Parameters:

• image_bgr (np.ndarray) – The input image in BGR format.

• labels (List[str]) – The list of labels to detect.

• threshold (float) – The detection threshold.

• max_box_area (int) – The maximum area for detections.

Returns:

The detection results.

Return type:

List[DetectionResult]

save_model(model_path: str) → None

Save the model at the specified path. Currently not implemented.

Parameters:

model_path (str) – The path to save the model file.

segment(image: Image, detection_results: List[DetectionResult]) → List[DetectionResult]

Use Segment Anything (SAM) to generate masks given an image and a set of bounding boxes.

Parameters:

• image (Image.Image) – The input image.

• detection_results (List[DetectionResult]) – The detection results with bounding boxes.

Returns:

The detection results with masks.

Return type:

List[DetectionResult]

train_model() → None

Train the model. Currently not implemented.

robotic_sdk.ml_models.grounded_segmentation.get_boxes(results: List[DetectionResult]) → List[List[List[float]]]

Extract bounding boxes from detection results.

Parameters:

results (List[DetectionResult]) – The detection results.

Returns:

A list of bounding boxes.

Return type:

List[List[List[float]]]

robotic_sdk.ml_models.grounded_segmentation.refine_masks(masks: BoolTensor) → List[ndarray]

Refine the masks by converting them to numpy arrays.

Parameters:

masks (torch.BoolTensor) – The input masks.

Returns:

The refined masks as numpy arrays.

Return type:

List[np.ndarray]

robotic_sdk.ml_models.metrics module

robotic_sdk.ml_models.metrics.calculate_iou(detections: list, ground_truth_mask: ndarray)

Calculate the Intersection over Union (IoU) score for a list of detections against a ground truth mask.

Parameters:

• detections (list) – A list of detection objects, where each object contains a mask attribute. The mask attribute is assumed to be a binary 2D numpy array representing the detected area.

• ground_truth_mask (np.ndarray) – A binary 2D numpy array representing the ground truth area for comparison.

Returns:

The average IoU score for all detections.

Return type:

float

robotic_sdk.ml_models.model module

class robotic_sdk.ml_models.model.Model(model_name: str, parameters: dict | None = None)

Bases: ABC

Abstract base class for all models.

__init__(model_name: str, parameters: dict | None = None)

Initializes the Model instance.

Parameters:

• model_name (str) – The name of the model.

• parameters (dict) – The hyperparameters for the model.

Raises:

ValueError – If the configuration is not set before creating any model instances.

abstract load_model(model_path: str) → None

Loads the model from the given path.

Parameters:

model_path (str) – The path to the model file.

Returns:

None

abstract perform_inference(**kwargs)

Abstract method to perform inference.

Parameters:

**kwargs – Arbitrary keyword arguments.

Returns:

None

abstract save_model(model_path: str) → None

Abstract method to save the model at the given path.

Parameters:

model_path (str) – The path to save the model file.

Returns:

None

abstract train_model() → None

Abstract method to train the model.

Returns:

None

robotic_sdk.ml_models.model_info module

robotic_sdk.ml_models.qwen2vl module

robotic_sdk.ml_models.segment_anything module

class robotic_sdk.ml_models.segment_anything.SegmentAnythingModel(model_name: str, parameters: dict)

Bases: Model

A class representing the Segment Anything Model (SAM) used for panoptic segmentation.

__init__(model_name: str, parameters: dict)

Initializes the Model instance.

Parameters:

• model_name (str) – The name of the model.

• parameters (dict) – The hyperparameters for the model.

Raises:

ValueError – If the configuration is not set before creating any model instances.

evaluate_model(evaluation_data)

Evaluate the model using Intersection over Union (IoU) metric.

Parameters:

evaluation_data (list) – A list of tuples where each tuple contains the image path and corresponding ground truth mask path.

Returns:

Evaluated metrics.

Return type:

dict

load_model(model_path: str)

Load the model from the specified path.

Parameters:

model_path (str) – Model path.

perform_inference(image_path: str, min_area: int | None = None, max_area: int | None = None, confidence: float | None = None, polygons: list | None = None, frame_resolution_wh=None)

Perform inference on the given image to detect and annotate objects.

Parameters:

• image_path (str) – Path to the input image file.

• min_area (int, optional) – Minimum area threshold for detections.

• max_area (int, optional) – Maximum area threshold for detections.

• confidence (float, optional) – Confidence threshold for detections.

• polygons (list, optional) – List of polygons for zone-based filtering.

• frame_resolution_wh (tuple, optional) – Frame resolution width and height.

Returns:

Original image, annotated image, result object, detections.

Return type:

tuple

train_model() → None

Abstract method to train the model.

Returns:

None

robotic_sdk.ml_models.utils module

robotic_sdk.ml_models.utils.o3d_to_ros2_pointcloud(pcd: PointCloud, frame_id: str = 'map') → PointCloud2

Convert Open3D point cloud to ROS2 PointCloud2 message.

robotic_sdk.ml_models.zero_shot_segmentation module

Module contents