An Ambient (or bias) lighting system is essentially a set of lights that are placed around the back of a computer monitor or television, so that the lights are facing away from the viewer. The idea is that the lights illuminate the area behind and around the display with colours according to the image that is being displayed. This (apparently) leads to reduced eye strain and increased viewer immersion. If you want to see one in the wild, Philips Ambilight TVs are a great example.
My final year University project was to build one of these using a Digilent Zybo Z7-10. This is a cool development board based on a Xilinx Zynq, which is a System on Chip that contains an ARM CPU as well an FPGA. The FPGA and CPU can communicate to each other over a bus, and allows for hardware-software co-processing. The project is available online as a GitHub repo here.
Physical Setup
As the Zybo contains both a HDMI input and a HDMI output port, the system is designed so that the board can intercept the HDMI connection between a monitor and a video source in order to perform image processing. The board is also connected to a strip of WS2812B addressable RGB LEDs via its PMOD GPIO pins. The LED strip was simply attached to the monitor with Blu Tack. From the image above, you can see that I had to cut the LED strip into 4 smaller strips, and then connected all of the strips together. This was just to allow the strip to bend around the corners of the monitor.
Implementation
The entire system uses the Digilent Zybo Z7 HDMI Demo as a base to begin working from. This took care of all of the intricacies of setting up HDMI input and output, and for the rest of the project, and allowed to me to get working on the core of the project relatively quickly.
Further to that, I also used Marshall Wingerson’s DigiLED WS2812 driver core for controlling the LEDs (although I had to alter the driver file so that it could support the RGB colour model).
The system operates by processing 8 pre-defined sections of a frame (pictured below). This is mainly to increase the execution speed, as less pixels are analysed, but it also allows for different LEDs to have different colours depending on the image. For example, in a sunrise/sunset scene, this would allow the LEDs that are nearest to the Sun to glow yellow or orange.
The basic functional behaviour can described with the following diagram:
We will go over every block in the diagram (apart from the DigiLED core) and explain what they do.
Colour Reducer
This reduces the number of pixels in a frame section, so that the overall number of unique colours in a section is reduced. Doing this speeds up the next step in the process (dominant colour extraction).
Two main methods were developed to do this. The first essentially scales the size of the section down, by moving a sliding window across the sections, and outputting the mean pixel value of the window.
An example of this method being performed on a 16x16 checkerboard can be seen below. The middle image shows the four position that the (8x8) window will be in, and the image on the far right shows the output, which consists of four grey pixels.
The second method is much more simple, it simply samples every nth pixel, where n can be one of 82, 162, 322 or 642.
Only one method of colour reduction can be used. To select which method should be used, one must uncomment the
appropriate flags in the vivado/proj/HDMI.sdk/QLight_1/src/algoFlags.h file. Both methods have their
advantages and disadvantages: the sliding window method produces a more accurate colour for the LEDs, however
the sampling method is faster (so long as you use a sampling frequency that samples more than every
82 pixels).
Dominant Colour Extractor
This is a fancy way of saying ‘mode pixel calculation’. All it does is find the most frequent pixel values in set of reduced pixels in the frame.
DigiLED Core
This is the component supplied by Marshall Wingerson which drives the LEDs.
I did have to make a modification to the driver files (vivado/proj/HDMI.sdk/QLight_1/src/DigiLED.{c, h}) in
order for it to support the RGB colour model, as the drivers supplied only support HSV.
I also had to manually map each LED to each section, which was fairly trivial.
Additional Functionality
If you look at the overall system architecture diagram, you can see that there are some extra features such as:
- Synchronising the video feed’s framerate with the frequency that the LEDs change colour
- Pausing between iterations of the system’s execution to conserve power
- Determining the pause time dynamically, by measuring the dominant colour differences between two consecutive frames
I don’t consider these to be core parts of their functionality, so they won’t be covered in further detail.
Hardware Acceleration
Hardware accelerated components were created using Vivado High-Level Synthesis (HLS). This is a tool that allows you to create hardware using C/C++, instead of VHDL or Verilog. It includes a bunch of macros, that allow you control the hardware synthesis. For example, in the following snippet:
int x = 0;
for (int i=0; i<100; i++) {
#pragma HLS unroll
x++;
}
The #pragma HLS unroll macro will make the for loop execute entirely in parallel, by creating hardware to
make each iteration execute in the same clock cycle.
The part that would definitely benefit the most from hardware acceleration would be the colour reducer. The problem was that I was not able to synthesise hardware that was small enough the fit on the Zybo’s FPGA. This was mainly due to the fact that I had to store each value of pixel data in a 32 bit integer. Although pixel data requires at most 8 bits, the smallest bit width available for an AXI interface between the ARM core and the FPGA is 32 bits. I tried several hacks to try and compress the data but Vivado HLS was not having any of it.
Instead, only dominant colour extraction was hardware accelerated. This was done by pipelining every loop in the algorithm.
End Result
In the end, I was able to create something that could run at 30-40fps, as long as the right flags are selected
in vivado/proj/HDMI.sdk/QLight_1/src/algoFlags.h. I found the best flags to use are HARDWARE_MODE and
SUBSAMPLE_SCALE, with a SUBSAMPLE_SCALE_FACTOR of 32. This strikes a good balance between speed and
accuracy of LED colour.