Lane detection for ADAS application with sliding windows technics

nguyenrobot
10 min readOct 21, 2020
Lane Finding demo, Image Credit : nguyenrobot

We will make a line detection algorithm which could be used for ADAS (Advanced Driving Assistance System) applications. Of course, many calibrations and testings should be done before to get our algorithm matured at industrial reliability. Further, the performance and the reliability of our codes need to be improved to be able to implement on Raspberry or Nvidia Jetson scale systems, which is the final objective at the end of my series of tutorials.

Line detection in this tutorial will cover :

  • Line detection of ego vehicle’s current lane
  • Line detection of ego vehicle’s next lane (next-left side and next-right lane)
  • Confidence level of each detected line
  • Line-type of each detected line
  • Lane-changing signal
  • Curve-fitting by 3-rd polynomial

In this tutorial, basic and fundamental technics of computer vision are applied :

  • Camera’s distorsion correction
  • Color-space transform [RGB to HSL]
  • Image pre-processing in HSL space
  • Bird-eye view transform
  • Gradient detection with Sobel Operator
  • Curve fitting with sliding-windows

**Github repos :
https://github.com/nguyenrobot/lane_detection_advanced_sliding_windows

#Part 1 : Technics

1.1 ADAS’s needs on lane-finding

An ADAS system would have these main components :

  • Camera’s distorsion correction
  • Color-space transform [RGB to HSL]
  • Image pre-processing in HSL space
  • Bird-eye view transform
  • Gradient detection with Sobel Operator
  • Curve fitting with sliding-windows

Among these ADAS components, LKA intensively relied on line detection and object tracking layers (another tutorial in my series will take charge of object tracking). Hence, LKA will need more information than just simple detected lines’ equations :

  • Confidence level of detected lines : to planify and maintain control phase, the controller needs to know about the detection quality
  • Line-type : basically, the controller needs to know if detected line is a solid (lane-crossing forbiden) or dashed to make reasonings. Furthuremore, some other types are yellow-solid, road-edge, barrier, double-line; they essentially to decide when to trigger the control phase, so the sensibility of the system
  • Lane change signal : although ADAS controller could make reasonings based on detected lines’ equations to know if a lane-crossing occurs, roughly a lane-crossing signal
  • Curve-fitting of detected lines by a 3rd polynomial. So, why a 3rd degree polynomial x = dy³ + cy² + by + a ?

*from a, we can estimate lateral distance to the line

*from b, we can estimate heading-angle of the ego vehicle

*from c, we can estimate the line curvature

*from d, we can estimate the curvature’s gradient

  • Next-lane information

1.2 Camera’s distorsion correction

I use the Undistort function from OpenCV library to do the job. You can find all about complex mathematics behind it in OpenCV documentation here :
https://docs.opencv.org/master/dc/dbb/tutorial_py_calibration.html
https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

Image distorsion, Image Credit : OpenCV documentation ↓

Source :https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

The idea of Undistort function is to take series of photos of a chessboard taken by the camera. Then we will find chessboard’s corners in each image and try to figure out distortion coefficients thanks to these corners’ position.

Image distorsion by S7’s rear camera, Image Credit : nguyenrobot ↓

1.3 Image preprocessing

We can get inspired by photography post-processing to have a clean image before any computation :

  • Exposure balance : to correct the average exposure value of a frame
  • White balance : to correct the white balance exposure value of a frame
  • Highlight removing : to remove highlights, inspired by Lightroom
  • Shadow removing : to remove shadows, inspired by Lightroom With these photography’s post-processing inspired technics, we could archieve good results in various light condition and filter some noises.

Then we will apply the color filtering to eliminate noises. A complusory step before color filtering is the color-space transform. Each input frame is in RGB (Red-Green-Blue) color space. However the perception of ‘colour’ of a pixel depends not only on R-G-B chanels value but also on the ratio between them. It’s free to use HSL or HSV color space, in my tutorial I will use HSL color space.

HSL and HSV spaces, Image Credit : wiki ↓

Source : https://en.wikipedia.org/wiki/HSL_and_HSV

If you are intrigued by the mathematics behind, then you can check this very good article :
https://www.niwa.nu/2013/05/math-behind-colorspace-conversions-rgb-hsl

To pick a color of a pixel, ImageGlass is a very versatile tiny software. It’s sometimes to manually pick a pixel to analyze for filtering parameters :

ImageGlass viewer, Image Credit : nguyenrobot ↓

ImageGlass : imageglass.org

1.4 Bird-eye view transform

The bird-eye view transform help us to have a sky-view from vehicle view taken frame.

The bird-eye transform is simply linear stretching and compressing of input pixels. With a good tweaked bird-eye view transform, we should have lane’s lines parallel in the end. This transform is also a source of error for line’s curvature estimation bu we will discard these errors in this tutorial, enhancements should be done in a future tutorial.

Bird-eye transform, Image Credit : nguyenrobot ↓

1.5 Gradient filter with Sobel Operator

In my previous tutorial, we used Canny edge and Gaussian filter to detect edges. However, with Canny Edge we can not know if an edge is ‘horizontal’ or ‘vertical’. In lane’s lines detection, we would like to keep nearly vertical edges and eliminate those nearly horizontal.

Sobel operator is a simple way to filter pixels that could be on a lane’s line. It’s not robust as pattern recognition by convolutional neural-network but it’s simple to implement and to tweak.

The Sobel operator is here to helps us :

  • get partial derivatives of a pixel’s value in x-direction and y-direction
  • get Gaussian filter which can smoothen pixels’ values by comparing with its neighbours With partial derivatives, we can calculate the magnitude and the argument of the gradient vector for each pixel.

Sobel Operator outcomes, Image Credit : nguyenrobot ↓

For further math calculus, opencv documentation explains pretty well here :
https://docs.opencv.org/3.4/d2/d2c/tutorial_sobel_derivatives.html
https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_gradients/py_gradients.html

1.6 Curve fitting with sliding-widows

With Gradient and Color filters, we obtain a binary image of relevant pixels that could lay on a line.

Binary bird-eye frame, Image Credit : nguyenrobot ↓

We make a count of pixels on vertical columns (each column is a vector of pixels) to have a histogram that give us the potential to have a line.

Histogram, Image Credit : nguyenrobot ↓

From each peak on the histogram, we initialize windows and then slide them vertically. Each window is horizontally centered in the end of each iteration by its detected pixels inside.

Sliding-windows, Image Credit : nguyenrobot ↓

Depending on the number of windows whose number of pixels inside requiring a minimum population, we can predict a confidence level of detected ‘line’ to say if it’s a line or not.

1.7 Extrapolation error

3rd degree polynomial is needed for our line equations approximation. However, it gets oscillated easily for extrapolation, interpolation is very good in contrary.

Here is an example of oscillated next-left line :

Oscillation caused by extrapolation of 3rd degree polynomial, Image Credit : nguyenrobot ↓

Source of raw image : Udacity

The oscillation is very annoying because when a line is badly extrapolated it’s usually occurs near to the frame’s bottom. This extrapolation strongly affect our estimation on space-left-in-lane and ego vehicle’s heading angle. These information are crucial for low-level controller acting on the steering wheel.

Oscillation caused by extrapolation, Image Credit : the Mathworks ↓

Source :https://www.mathworks.com/help/matlab/ref/polyfit.html

So, to remediate this extrapolation error, I use a linear extrapolation for missing pixels of a line :

  • Linear extrapolation for missing line’s pixels
  • Polyfit with a 3rd degree polynomial on detected pixels + linearly extrapolated pixels

#Part 2 : Coding

Youtube video with good results ↓

Youtube video with issues caused by over-taking target ↓

2.1 Coordination Systems

Coordination Systems, Image Credit : nguyenrobot ↓

  • When an image frame [1280 x 720 x 3] is read with opencv, pixel (1280,720)’s chanel i value is picked by (720,1280,i) (like is swapped horizontal >> vertical by opencv reading)
  • Frame’s origin is the top-left corner, this coordination is used for curve-fitting.
  • Ego vehicle’s coordination is used to transform curve-fitting equations to has lane equations in vehicle’s coordination system.

#note : the camera offset is due to camera’s lateral distancing from ego vehicle’s origin, when the cameara is not installed on ego vehicle’s center-line

2.2 Information extraction from line equation

From polyfit() for detected pixels on a line, we can obtain a 3-rd polynomial as : x = dy³ + cy² + by + a

*Notion of look-ahead distance We could not-hardly get some basic information from the polynomial :

lane info — space-left, Image Credit : nguyenrobot ↓

lane info — curvature & heading, Image Credit : nguyenrobot ↓

2.3 Camera Calibration

For my road videos, I used the rear camera of a Samsung S7 smartphone mounted behind my Ford Focus 1999 windshield. Astonishingly, distortion correction not really needed with videos filmed by S7’s rear camera, they are well filtered and preprocessed so the video is pretty good that we can directly use without distortion correction. However, it’s better to know about image distortion correction to be able to work with any kind of video input.

Image distorsion by S7’s rear camera, Image Credit : nguyenrobot ↓

2.4 Image pre-processing

I made pre-processing and filtering on both vehicle-view frame and sky-view frame. Sometimes, each fullfills other.

Image pre-processing on sky-view, Image Credit : nguyenrobot ↓

Image pre-processing on vehicle view, Image Credit : nguyenrobot ↓

2.5 Pixel to meter

I have my smartphone S7 fixed by supports behind the windshield without any information of mounting geometry neither optical calibration of my camera.

How can I convert from pixel-dimension to meter-dimension ?

pixel to meter, Image Credit : nguyenrobot ↓

I get disposed information of French highway (I filmed my videos on A5 French highway) :

  • Standard lane-width is 3.5m for French highway
  • By counting the number of barrier supports (lane separator) between two consecutive mile-stone, I found that these barrier supports are 2m distanced from each other

From a well chosen bird-eye view frame, I can estimate pixel-to-meter ratio with acceptable precision

pixel to meter, Image Credit : nguyenrobot ↓

A well chosen frame is a frame that ego vehicle is heading straight ahead and is in the middle of the current lane.
In the end, we can obtain linear coefficients to convert from pixel to m for longitudinal and lateral distances.

2.6 Discriminative power of Sobel gradients’ outcomes

This a summary of discriminative powers of each 1st and 2nd Sobel operations on different channel (H S L) :
1st degree Sobel operations — sobel_resum.pptx

Sobel operations’ discriminative power, an extract, Image Credit : nguyenrobot ↓

Source : https://github.com/nguyenrobot/lane_detection_advanced_sliding_windows/blob/main/images_jupyter/sobel_resum.pptx

It’s fun to have cross-combining of different kinds of Sobel operation to have effective filters.

Wrapping up

There are so many things up to do for our algorithm’s optimization and reliability.

  • Line-type : our algorithm distinguishes well solid and dashed-line. However, for other types as double-line, solid-yellow, barrier, road-edge we will need something more robust and more intelligent. Hence, a layer pattern recognition should be integrated for further release.
  • Sobel vs Pattern : filtering with Sobel gradient has its limits where pattern recognition finds its effectiveness.
  • Sobel at higher degree : I didn’t exploit 2nd degree Sobel information even they are very effective for precision to enhance 1st degree Sobel information.
  • Object detection : road signs, road separator materials, vehicle detection — we did nothing about them in the algorithm. A convolutional neural network is needed for further release.
  • Detection tracking : each line is detected separately from frame-to-frame. A tracking object foreach detected line is needed also.
  • Bird-eye view vs slope vs curve’s radius : bird-eye transform distort and and noises, it’s less effective on hill-road.
  • Algorithm’s precision : some tweakings and calibration on real situations are needed to make better estimations.
  • Algorithm’s performance : frame sectioning, hybrid frame processing — it could be interesting for performance improvement. For the moment, our algorithm process frame-by-frame, which is not powerful and very slow.

Next tutorials : Object Tracking and Driver-behaviour cloning with Neural Networks, stay tuned !

Originally published at https://github.com.

--

--

nguyenrobot

www.nguyenrobot.com. I go far will a small backpack. I live with eagers but not with possessions. an engineer who can fix his car himself.