Sunday, June 12, 2022

Making Opensource USB C industrial camera with Interchangeable C mount lens, Interchangeable MIPI Sensor with Lattice Crosslink NX FPGA Cypress FX3 USB 3.0 controller

 This post is going yet another part in the DIY camera projects which have been doing since quite some time. In this post I will showing you next successful implementation of making C mount high lens mount  USB C camera. This implementation will have absolute modular boards having dedicated Sensor board which can be changed if needed. 

Hardware System
There are three boards, USB, FPGA and Sensor board. 
Sensor board has Sensor itself along with its power and Oscillator, Board has  High Density connector to be able to connect to FPGA/Host board, High Density connector also supply power to the board has I2C, 4Lane MIPI with clock , I2C and also few other control signals. 

FPGA board has in this case Lattice Crosslink NX LIFCL-40 in 256caBGA package with required power regulators, This board also has 2x 16Mybte RAM for applications that need additional memory, FPGA config flash memory is also on this board this board has two high density connector. Board pass I2C signal from Host right threw. 

USB Board in this case has CYUSB3014 USB 3.0 Superspeed controller, along with required power and Memory, As this board has 3.0 USB C connector, so there is USB 3.0 mux is also there to support connector reversal 


Currently I have just one camera PCB, This schematic shows IMX290 IMX327 IMX462 PCB , all these 3 sensors have same resolution and also same PCB footprint.

FPGA board

USB 3.0 board

PCB Boards

All of the boards are 6 Layer, All of them are 27 x 27 mm , while mounting holes are 22 mm appart

Camera sensor Board 


USB 3.0 Board


Hardware and Camera Lens C mount

To be able to mount a C mount lens I designed a mount in Fusion 360. as small threads on FDM printer are little hard to deal with  There are many already made aluminum CS mount to C mount adapters are available online, Raspberry Pi camera also comes with one such ring, I bought these CS to C mount ring and I designed part around my PCB to fit these CS mount adapter ring to give me metal thread. 


FPGA Design

What is MIPI, you can google it to find out but basically it is a interface specification for Displays and Camera sensor to a application processor.

Image blow show block diagram of MIPI.  On one side there is application processor and other side is the peripheral. When peripheral is Camera and CSI apply. though mipi is closed specification which means one has to be member of MIPI consortium to gain access to full specification. And membership of the consortium comes with a big price tag for individuals. Luckily full specification is already available just a right keyword web search away. DCS, CCS, DSI, CSI and DPHY all the specification are available with just few minutes of web search.


The image shows i got from google shows signal level for MIPI , HS driven by differential driver swings -200mV to +200mV at offset of 200mv. while LP signal is a 1.2V lvcmos 

there are two different modes of transmission , HS mode and LP mode, HS mode is for hi speed display data while LP mode is for Low power transmission. 

Receiver must detect when transmitter has gone into HS mode and exited HS mode.
Image blow shows how transmitter enter HS modes. 

Stage 0 : LP-11 state in the shown image is LP state.
Stage 1 : To get into HS mode driver drives LPdp low for Tlpx(minimum 50ns) and stay in LP-01 (HS driver is tristate in LP 01).
Stage 2: Driver drives LPdn low for Ths-prepare (minimum 95ns) stay in LP-00 , Later somewere in the middle of this stage target device will activate it's 100R termination register. 
Stage 3: Now Target is in HS, driver activates HS driver start sending mandatory zeros .
Stage 4: Send mandatory 0xB8 sync byte and then payload.


As explained earlier CSI , describes Packet structure. How exactly bytes are packed on to wire in different lanes configuration.
Image blow shows packet structure. 

There are Two types of Packets
Short Packet: 4 Bytes (Fixed Length)
Long Packet: 6 - 655541 Bytes (Variable Length)

MIPI Short Packet Structure

MIPI Long Packet Structure

Bytes are sent lsbit first and bytes in the packet are sent LSByte first. 


Very important fact with CCS when comparing with DCS , CCS describes command interface to be I2C while with DCS commands are set over same HS line as the data itself. 
But in case of camera as per MIPI spec CCS is implemented over a extra I2C line. 

CSI Single Frame

Single Frame from camera is show in the image blow. 

Camera send a Frame start packet 
Then send embedded line information which tells receiver about the stream 
Then image data line by line. 

Test Video

What make this camera sensor different to camera modules

IMX219 camera is bare bone camera sensor. What do means when i say bare bone camera sensor is , there not much image processing going on on the camera die it self. Camera sensor is Sensor array with Bayer filter on it , 10 bit ADC , clock system , MIPI output driver and I2C controllable system control.

What does this means for us as camera sensor implementer. As my final goal is to interface this camera to USB3.0 UVC with RAW YUV.  This camera does not output YUV, forget about YUV this will not even output RGB. Camera output is absolute RAW 10-bit ADC conversion result from the Bayer filtered sensor array.

So go first get RGB output from bayer raw data, a Debayer or demosaic need to be performed. Once demosaic is done we will have RGB ready to be converted to YUV. And one we have YUV it can be transmitted to USB to be displayed.

What next this camera will not have is any automatic control over exposure. because camera does not have any intelligence to know how dark  or bright scene is.  Solution to this problem what raspberry pi implement is , Raspberry Pi regularly on each frame update analog gain register over I2C to adjust gain according to how bright and dark scene is.

This camera does not have any white balance control as well so host must do correct while balance compensations. To get correct colors out of image.

FPGA module Block Diagram 

FPGA block diagram is show in the image blow. This diagram describe how overall system is implemented and what the key components what this diagram does not describe is control signals and other miscellaneous stuff.

Byte Aligner Received Raw unaligned bits from DDR RX module outputs Aligned bytes, Bytes on MIPI lane does not have any defined byte boundary so this modules Looks for always constant first byte 0xB8 on wire, once 0xB8 is found, byte boundary offset is determined, set output valid to active and start outputting correct bytes stays reset when data lane are in MIPI LP state  

Lane Aligner Receives multiple lane, byte aligned data from mipi rx byte aligner @mipi byte clock  outputs lane aligned data in a multi-lane mipi bus, data on different lane may appear at different offset so this module will wait till of the all lanes have valid output start outputting lane aligned data so byte x from all the lanes outputted at same timescale

MIPI CSI Packet Decoder Basically a packet Stripper, removes header and footer from packet Takes lane aligned data from lane aligner @ mipi byte clock looks for specific packet type, in this case RAW10bit ( 0x2B). Module outputs Stripped bytes in exactly the way they were received. This module also fetch packet length and output_valid is active as long as input data is valid and received number of bytes is still within the limits of packet length.

MIPI CSI RAW10 Depacker  Receives 4 lane raw mipi bytes from packet decoder, rearrange bytes to output 4 pixel 10bit each output is one clock cycle delayed, because the way , MIPI RAW10 is packed output come in group of 5x40bit chunk, output_valid_o remains active only while 20 pixel chunk is outputted

Debayer / demosaic Takes 4x10bit pixel from RAW10 depacker module @mipi byte clock output 4x24bit RGB for each pixel , output is delayed by 2 lines Implement Basic Debayer filter, As debayer need pixel infrom neighboring pixel which may be on next or previous display line, so input data is written onto RAM, only 4 lines are stored in RAM at one time and only three of the readable at any give time , RAM to which data is written to can not be read. As we have enough info in RAM 4 10bit pixel will be coverted to 4x24bit RGB output First line is expected to BGBG , second line GRGR Basically BGGR format  

RGB to YUV Color Space Converter Received 4 pixel 120bit RGB from the Debayer filter output 64bit 4pixel yuv422  Calculation is done based on integer YUV formula from the YUV wiki page 

Output reformatter Takes 64bit 4pixel yuv input from rgb2yuv module @ mipi byte clock outputs 32bit 2pixel yuv output @output_clk_i , output_clk_i must be generated by same way as mipi byte clock, output_clk_i must be exactly double to mipi byteclock This implementation of Output reformatter outputs data which which meant to send out of the system to a 32bit receiver depending on requirement this will be need to be adapted as per the receiver 

Debayer / demosaic  Need little more attention than other modules , IMX219 datasheet incorrectly mention output as to be either GBRG or RGGB. 

But after wasting lots of time it turned out camera output BGGR .  IMX219 Camera only output BGGR as defined by the IMX219 Driver in linux repo MEDIA_BUS_FMT_SBGGR10_1X10,  Camera datasheet incrorrectly defines output as RGGB and GBRG. Data sheet is incorrect in this case.
To test my debayer i was using built in camera test patterns. One key thing about IMX219 is Bayer filter type does affect test pattern as well. It seems like in Test pattern mode it outputs RGGB instead of BGGR. Test pattern will have R and B channel inverted when image have right color.

Update: I have discussed this issue with raspberry pi , It turned out flipping image seems to be the solution, once image flipped bayer output it correct for both data from sensor and test pattern. because flipping image does not affect bayer order of the test pattern.

RAW10 Packet Format

Test image 

Cypress FX3 Firmware

Firmware implementation with FX3 was quite easy. I have put all the resolution and framerate in the USB descriptor , As described earlier this type of camera sensors are quite bare bone all the have sensor element, PLLs and ADC . So this camera sensor does not have any control over exposure, White-balance or even brightness, I have implemented manual control over USB UVC control channel. it possible to completely control camera exposure and brightness.

Few things you keep in mind, cypress fx3 clock frequency need to be set in 400Mhz mode to allow full 100Mhz 32bit GPIF DMA transfer.

One more thing is though Cypress CYUSB3014 has 512KB RAM but only 224 KB and additional 32KB is available for DMA buffer.

Having large buffer chunk is really important because on every DMA chunk cpu intervention is expected to insert UVC header. As this is high performance application less often cpu intervention is needed is better. So i have set DMA chunk / UVC individual packet to 32KB

Scope capture Image blow shows Channel 13 is the individual DMA packet capture and on Channel 12 show CPU DMA finish interrupt.

These Two scope capture show difference between 16KB DMA vs 32KB DMA

16KB DMA Size, CH13 DMA packet , CH12 CPU interrupt
32KB DMA Size, CH13 DMA packet , CH12 CPU interrupt

PCB and Schematic Source is available in the Github Repo

1 comment:

  1. Absolutely interesting! Very good job! Thanks for share!!