利用OpenCV實現實時影象識別和影象跟蹤
影象識別什麼是影象識別影象識別，是指利用計算機對影象進行處理、分析和理解，以識別各種不同模式的目標和對像的技術。根據觀測到的影象，對其中的物體分辨其類別，做出有意義的判斷。利用現代資訊處理與計算技術來模擬和完成人類的認識、理解過程。一般而言，一個影象識別系統主要由三個部分組成，分別是：影象分割、影象特徵提取以及分類器的識別分類。
其中，影象分割將影象劃分為多個有意義的區域，然後將每個區域的影象進行特徵提取，最後分類器根據提取的影象特徵對影象進行相對應的分類。實際上，影象識別和影象分割並不存在嚴格的界限。從某種意義上，影象分割的過程就是影象識別的過程。影象分割著重於物件和背景的關係，研究的是物件在特定背景下所表現出來的整體屬性，而影象識別則著重於物件本身的屬性。
影象識別的研究現狀影象識別的發展經歷了三個階段：文字識別、數字影象處理與識別、物體識別。
影象識別作為計算視覺技術體系中的重要一環，一直備受重視。微軟在兩年前就公佈了一項里程碑式的成果：它的影象系統識別圖片的錯誤率比人類還要低。如今，影象識別技術又發展到一個新高度。這有賴於更多資料的開放、更多基礎工具的開源、產業鏈的更新迭代，以及高效能的AI計算晶片、深度攝像頭和優秀的深度學習演算法等的進步，這些都為影象識別技術向更深處發展提供了源源不斷的動力。
其實對於影象識別技術，大家已經不陌生，人臉識別、虹膜識別、指紋識別等都屬於這個範疇，但是影象識別遠不只如此，它涵蓋了生物識別、物體與場景識別、影片識別三大類。發展至今，儘管與理想還相距甚遠，但日漸成熟的影象識別技術已開始探索在各類行業的應用。
Android影象識別相關技術OpenCV 基於BSD許可（開源）發行的跨平臺計算機視覺庫，可以執行在Linux、Windows、Android和Mac OS作業系統上。 輕量級而且高效——由一系列 C 函式和少量 C++ 類構成，同時提供了Python、Ruby、MATLAB等語言的介面，實現了影象處理和計算機視覺方面的很多通用演算法TensorFlow TensorFlow是一個深度學習框架，支援Linux平臺，Windows平臺，Mac平臺，甚至手機移動裝置等各種平臺。 TensorFlow提供了非常豐富的深度學習相關的API，可以說目前所有深度學習框架裡，提供的API最全的，包括基本的向量矩陣計算、各種最佳化演算法、各種卷積神經網路和迴圈神經網路基本單元的實現、以及視覺化的輔助工具、等等。YOLO YOLO (You Only Look Once)是一種快速和準確的實時物件檢測演算法。 YOLOv3 在 TensorFlow 中實現的完整資料管道。它可用在資料集上來訓練和評估自己的目標檢測模型。……基於OpenCV實現介紹使用OpenCV來實現指定影象識別的DEMO：
實現思路①開啟應用的同時開啟攝像頭 ②對實時攝像頭拍攝的影象封裝成MAT物件進行逐幀比對：
獲取目標特徵並針對各特徵集獲取描述符獲取兩個描述符集合間的匹配項獲取參考影象和空間匹配影象間的單應性當影象矩陣符合單應性時，繪製跟蹤影象的輪廓線程式碼部分許可權設定AndroidMainifest.xml<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-feature android:name="android.hardware.camera" />
<uses-feature
android:name="android.hardware.camera.autofocus"
android:required="false" />
<uses-feature
android:name="android.hardware.camera.flash"
android:required="false" />
許可權提示方法
private void requestPermissions() {
final int REQUEST_CODE = 1;
if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {
ActivityCompat.requestPermissions(this, new String[]{
Manifest.permission.CAMERA, Manifest.permission.WRITE_EXTERNAL_STORAGE},
REQUEST_CODE);
}
}
介面設計activity_img_recognition.xml<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:opencv="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:id="@+id/activity_img_recognition"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context="com.sueed.imagerecognition.CameraActivity">
<org.opencv.android.JavaCameraView
android:id="@+id/jcv"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:visibility="gone"
opencv:camera_id="any"
opencv:show_fps="true" />
</RelativeLayout>
主要邏輯程式碼CameraActivity.java 【相機啟動獲取影象和包裝MAT相關】因為OpenCV中JavaCameraView繼承自SurfaceView，若有需要可以自定義編寫extends SurfaceView implements SurfaceHolder.Callback的xxxSurfaceView替換使用。
package com.sueed.imagerecognition;
import android.Manifest;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.os.Bundle;
import android.util.Log;
import android.view.Menu;
import android.view.MenuItem;
import android.view.SurfaceView;
import android.view.View;
import android.view.WindowManager;
import android.widget.ImageView;
import android.widget.RelativeLayout;
import android.widget.Toast;
import androidx.appcompat.app.AppCompatActivity;
import androidx.core.app.ActivityCompat;
import androidx.core.content.ContextCompat;
import com.sueed.imagerecognition.filters.Filter;
import com.sueed.imagerecognition.filters.NoneFilter;
import com.sueed.imagerecognition.filters.ar.ImageDetectionFilter;
import com.sueed.imagerecognition.imagerecognition.R;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;
import org.opencv.android.JavaCameraView;
import org.opencv.android.OpenCVLoader;
import org.opencv.core.Mat;
import java.io.IOException;
// Use the deprecated Camera class.
@SuppressWarnings("deprecation")
public final class CameraActivity extends AppCompatActivity implements CvCameraViewListener2 {
// A tag for log output.
private static final String TAG = CameraActivity.class.getSimpleName();
// The filters.
private Filter[] mImageDetectionFilters;
// The indices of the active filters.
private int mImageDetectionFilterIndex;
// The camera view.
private CameraBridgeViewBase mCameraView;
@Override
protected void onCreate(final Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
//init CameraView
mCameraView = new JavaCameraView(this, 0);
mCameraView.setMaxFrameSize(size.MaxWidth, size.MaxHeight);
mCameraView.setCvCameraViewListener(this);
setContentView(mCameraView);
requestPermissions();
mCameraView.enableView();
}
@Override
public void onPause() {
if (mCameraView != null) {
mCameraView.disableView();
}
super.onPause();
}
@Override
public void onResume() {
super.onResume();
OpenCVLoader.initDebug();
}
@Override
public void onDestroy() {
if (mCameraView != null) {
mCameraView.disableView();
}
super.onDestroy();
}
@Override
public boolean onCreateOptionsMenu(final Menu menu) {
getMenuInflater().inflate(R.menu.activity_camera, menu);
return true;
}
@Override
public boolean onOptionsItemSelected(final MenuItem item) {
switch (item.getItemId()) {
case R.id.menu_next_image_detection_filter:
mImageDetectionFilterIndex++;
if (mImageDetectionFilters != null && mImageDetectionFilterIndex == mImageDetectionFilters.length) {
mImageDetectionFilterIndex = 0;
}
return true;
default:
return super.onOptionsItemSelected(item);
}
}
@Override
public void onCameraViewStarted(final int width, final int height) {
Filter Enkidu = null;
try {
Enkidu = new ImageDetectionFilter(CameraActivity.this, R.drawable.enkidu);
} catch (IOException e) {
e.printStackTrace();
}
Filter akbarHunting = null;
try {
akbarHunting = new ImageDetectionFilter(CameraActivity.this, R.drawable.akbar_hunting_with_cheetahs);
} catch (IOException e) {
Log.e(TAG, "Failed to load drawable: " + "akbar_hunting_with_cheetahs");
e.printStackTrace();
}
mImageDetectionFilters = new Filter[]{
new NoneFilter(),
Enkidu,
akbarHunting
};
}
@Override
public void onCameraViewStopped() {
}
@Override
public Mat onCameraFrame(final CvCameraViewFrame inputFrame) {
final Mat rgba = inputFrame.rgba();
if (mImageDetectionFilters != null) {
mImageDetectionFilters[mImageDetectionFilterIndex].apply(rgba, rgba);
}
return rgba;
}
}
ImageRecognitionFilter.java【影象特徵過濾比對及繪製追蹤綠框】
package com.nummist.secondsight.filters.ar;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.opencv.android.Utils;
import org.opencv.calib3d.Calib3d;
import org.opencv.core.Core;
import org.opencv.core.CvType;
import org.opencv.core.DMatch;
import org.opencv.core.KeyPoint;
import org.opencv.core.Mat;
import org.opencv.core.MatOfDMatch;
import org.opencv.core.MatOfKeyPoint;
import org.opencv.core.MatOfPoint;
import org.opencv.core.MatOfPoint2f;
import org.opencv.core.Point;
import org.opencv.core.Scalar;
import org.opencv.features2d.DescriptorExtractor;
import org.opencv.features2d.DescriptorMatcher;
import org.opencv.features2d.FeatureDetector;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;
import android.content.Context;
import com.nummist.secondsight.filters.Filter;
public final class ImageDetectionFilter implements Filter {
// The reference image (this detector's target).
private final Mat mReferenceImage;
// Features of the reference image.
private final MatOfKeyPoint mReferenceKeypoints = new MatOfKeyPoint();
// Descriptors of the reference image's features.
private final Mat mReferenceDescriptors = new Mat();
// The corner coordinates of the reference image, in pixels.
// CvType defines the color depth, number of channels, and
// channel layout in the image. Here, each point is represented
// by two 32-bit floats.
private final Mat mReferenceCorners = new Mat(4, 1, CvType.CV_32FC2);
// Features of the scene (the current frame).
private final MatOfKeyPoint mSceneKeypoints = new MatOfKeyPoint();
// Descriptors of the scene's features.
private final Mat mSceneDescriptors = new Mat();
// Tentative corner coordinates detected in the scene, in
// pixels.
private final Mat mCandidateSceneCorners = new Mat(4, 1, CvType.CV_32FC2);
// Good corner coordinates detected in the scene, in pixels.
private final Mat mSceneCorners = new Mat(0, 0, CvType.CV_32FC2);
// The good detected corner coordinates, in pixels, as integers.
private final MatOfPoint mIntSceneCorners = new MatOfPoint();
// A grayscale version of the scene.
private final Mat mGraySrc = new Mat();
// Tentative matches of scene features and reference features.
private final MatOfDMatch mMatches = new MatOfDMatch();
// A feature detector, which finds features in images.
private final FeatureDetector mFeatureDetector = FeatureDetector.create(FeatureDetector.ORB);
// A descriptor extractor, which creates descriptors of
// features.
private final DescriptorExtractor mDescriptorExtractor = DescriptorExtractor.create(DescriptorExtractor.ORB);
// A descriptor matcher, which matches features based on their
// descriptors.
private final DescriptorMatcher mDescriptorMatcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMINGLUT);
// The color of the outline drawn around the detected image.
private final Scalar mLineColor = new Scalar(0, 255, 0);
public ImageDetectionFilter(final Context context, final int referenceImageResourceID) throws IOException {
// Load the reference image from the app's resources.
// It is loaded in BGR (blue, green, red) format.
mReferenceImage = Utils.loadResource(context, referenceImageResourceID, Imgcodecs.CV_LOAD_IMAGE_COLOR);
// Create grayscale and RGBA versions of the reference image.
final Mat referenceImageGray = new Mat();
Imgproc.cvtColor(mReferenceImage, referenceImageGray, Imgproc.COLOR_BGR2GRAY);
Imgproc.cvtColor(mReferenceImage, mReferenceImage, Imgproc.COLOR_BGR2RGBA);
// Store the reference image's corner coordinates, in pixels.
mReferenceCorners.put(0, 0, new double[]{0.0, 0.0});
mReferenceCorners.put(1, 0, new double[]{referenceImageGray.cols(), 0.0});
mReferenceCorners.put(2, 0, new double[]{referenceImageGray.cols(), referenceImageGray.rows()});
mReferenceCorners.put(3, 0, new double[]{0.0, referenceImageGray.rows()});
// Detect the reference features and compute their
// descriptors.
mFeatureDetector.detect(referenceImageGray, mReferenceKeypoints);
mDescriptorExtractor.compute(referenceImageGray, mReferenceKeypoints, mReferenceDescriptors);
}
@Override
public void apply(final Mat src, final Mat dst) {
// Convert the scene to grayscale.
Imgproc.cvtColor(src, mGraySrc, Imgproc.COLOR_RGBA2GRAY);
// Detect the scene features, compute their descriptors,
// and match the scene descriptors to reference descriptors.
mFeatureDetector.detect(mGraySrc, mSceneKeypoints);
mDescriptorExtractor.compute(mGraySrc, mSceneKeypoints, mSceneDescriptors);
mDescriptorMatcher.match(mSceneDescriptors, mReferenceDescriptors, mMatches);
// Attempt to find the target image's corners in the scene.
findSceneCorners();
// If the corners have been found, draw an outline around the
// target image.
// Else, draw a thumbnail of the target image.
draw(src, dst);
}
private void findSceneCorners() {
final List<DMatch> matchesList = mMatches.toList();
if (matchesList.size() < 4) {
// There are too few matches to find the homography.
return;
}
final List<KeyPoint> referenceKeypointsList = mReferenceKeypoints.toList();
final List<KeyPoint> sceneKeypointsList = mSceneKeypoints.toList();
// Calculate the max and min distances between keypoints.
double maxDist = 0.0;
double minDist = Double.MAX_VALUE;
for (final DMatch match : matchesList) {
final double dist = match.distance;
if (dist < minDist) {
minDist = dist;
}
if (dist > maxDist) {
maxDist = dist;
}
}
// The thresholds for minDist are chosen subjectively
// based on testing. The unit is not related to pixel
// distances; it is related to the number of failed tests
// for similarity between the matched descriptors.
if (minDist > 50.0) {
// The target is completely lost.
// Discard any previously found corners.
mSceneCorners.create(0, 0, mSceneCorners.type());
return;
} else if (minDist > 25.0) {
// The target is lost but maybe it is still close.
// Keep any previously found corners.
return;
}
// Identify "good" keypoints based on match distance.
final ArrayList<Point> goodReferencePointsList = new ArrayList<Point>();
final ArrayList<Point> goodScenePointsList = new ArrayList<Point>();
final double maxGoodMatchDist = 1.75 * minDist;
for (final DMatch match : matchesList) {
if (match.distance < maxGoodMatchDist) {
goodReferencePointsList.add(referenceKeypointsList.get(match.trainIdx).pt);
goodScenePointsList.add(sceneKeypointsList.get(match.queryIdx).pt);
}
}
if (goodReferencePointsList.size() < 4 || goodScenePointsList.size() < 4) {
// There are too few good points to find the homography.
return;
}
// There are enough good points to find the homography.
// (Otherwise, the method would have already returned.)
// Convert the matched points to MatOfPoint2f format, as
// required by the Calib3d.findHomography function.
final MatOfPoint2f goodReferencePoints = new MatOfPoint2f();
goodReferencePoints.fromList(goodReferencePointsList);
final MatOfPoint2f goodScenePoints = new MatOfPoint2f();
goodScenePoints.fromList(goodScenePointsList);
// Find the homography.
final Mat homography = Calib3d.findHomography(goodReferencePoints, goodScenePoints);
// Use the homography to project the reference corner
// coordinates into scene coordinates.
Core.perspectiveTransform(mReferenceCorners, mCandidateSceneCorners, homography);
// Convert the scene corners to integer format, as required
// by the Imgproc.isContourConvex function.
mCandidateSceneCorners.convertTo(mIntSceneCorners, CvType.CV_32S);
// Check whether the corners form a convex polygon. If not,
// (that is, if the corners form a concave polygon), the
// detection result is invalid because no real perspective can
// make the corners of a rectangular image look like a concave
// polygon!
if (Imgproc.isContourConvex(mIntSceneCorners)) {
// The corners form a convex polygon, so record them as
// valid scene corners.
mCandidateSceneCorners.copyTo(mSceneCorners);
}
}
protected void draw(final Mat src, final Mat dst) {
if (dst != src) {
src.copyTo(dst);
}
if (mSceneCorners.height() < 4) {
// The target has not been found.
// Draw a thumbnail of the target in the upper-left
// corner so that the user knows what it is.
// Compute the thumbnail's larger dimension as half the
// video frame's smaller dimension.
int height = mReferenceImage.height();
int width = mReferenceImage.width();
final int maxDimension = Math.min(dst.width(), dst.height()) / 2;
final double aspectRatio = width / (double) height;
if (height > width) {
height = maxDimension;
width = (int) (height * aspectRatio);
} else {
width = maxDimension;
height = (int) (width / aspectRatio);
}
// Select the region of interest (ROI) where the thumbnail
// will be drawn.
final Mat dstROI = dst.submat(0, height, 0, width);
// Copy a resized reference image into the ROI.
Imgproc.resize(mReferenceImage, dstROI, dstROI.size(), 0.0, 0.0, Imgproc.INTER_AREA);
return;
}
// Outline the found target in green.
Imgproc.line(dst, new Point(mSceneCorners.get(0, 0)), new Point(mSceneCorners.get(1, 0)), mLineColor, 4);
Imgproc.line(dst, new Point(mSceneCorners.get(1, 0)), new Point(mSceneCorners.get(2, 0)), mLineColor, 4);
Imgproc.line(dst, new Point(mSceneCorners.get(2, 0)), new Point(mSceneCorners.get(3, 0)), mLineColor, 4);
Imgproc.line(dst, new Point(mSceneCorners.get(3, 0)), new Point(mSceneCorners.get(0, 0)), mLineColor, 4);
}
}
實現效果圖確認允許許可權：
實時追蹤指定影象
結語本文只實現了需要提供完整原圖進行比對才能實現影象識別，還有許多更加智慧方便的識別技術和方法，比如：HOG、SIFT、SURF 等方法經由正負樣本庫進行訓練後可以從影象中提取一些特徵，並透過特徵確定物體類別。OpenCV庫中也仍有很大一部分的功能在本文中未能進行實踐，亟待今後繼續探索和研究。更多Python知識請關注我分享更多！
本文轉載於：https://blog.csdn.net/qq_42871911/article/details/111460677?ops_request_misc=%25257B%252522request%25255Fid%252522%25253A%252522160880932216780310190875%252522%25252C%252522scm%252522%25253A%25252220140713.130102334.pc%25255Fall.%252522%25257D&request_id=160880932216780310190875&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~first_rank_v2~times_rank-23-111460677.pc_search_result_no_baidu_js&utm_term=opencv%E4%BA%BA%E8%84%B8%E8%AF%86%E5%88%AB