Proton Calorimetry/Equipment/FPGAs DDC232

Useful Material

Technical Documents

• DDC232 datasheet. This contains essential information from which the FPGA design was created.
• Nexys Video manual.
• Xilinx Clocking Wizard manual.
• Xilinx Integrated Logic Analyser manual.
• Xilinx FIFO Generator manual.
• FTDI chip datasheet.
• libFTDI. C++ library used to communicate with the FTDI chip.
• FTDI tutorial. Useful example on how to use the Nexys Video FTDI chip with libFTDI.
• libFTDI on Windows.
• NandLand UART modules. This website also contains several useful tutorials for learning FPGA design.
• Zybo Z7 manual.

Presentations

• I/O signals and operation of TI DDC232CK based on datasheet information. Slides available here.
• Results of simulation of FPGA design discussed here.
• UART interface between FPGA and PC discussed in this presentation.
• First operational tests discussed in this presentation.
• DDC232 design features and demo given in this presentation.

Background

DDC232

A Texas Instruments (Dallas, Texas, United States) DDC232CK (DDC) was chosen as the current-input analogue-to-digital converter for its speed, large dynamic range and low power requirements. It is capable of measuring the currents of up to 32 photodiodes with an adjustable integration time (160μs–1s) and full-scale range (FSR, 12pC–350pC). Each of the 32 inputs on the DDC has two integrators, allowing for zero-deadtime measurements: while one integrator digitises and transfers data, the other measures the input current. The DDC is housed on a compact custom circuit board manufactured by CosyLab (Ljubljana, Slovenia), where the charge collected by a photodiode is split across two DDC inputs to give 16 photodiodes per DDC. The role of each digital signal is described below:

• CLK (input): 10 MHz clocking signal that is used to time the internal operations of the DDC, including generation of DVALID.
• CONV (input): signal that controls integration, the time period of which is equal to the integration time. When this signal toggles, the integrator of each input switches.
• DIN_CFG (input): serial data stream of the 12-bit sequence used to set key parameters of the DDC, namely the FSR and measurement precision (16-bit or 20-bit).
• CLK_CFG (input): 20 MHz clocking signal used to time sending and reading of DIN_CFG.
• RESET (input): Asynchronous active-low reset signal for the DDC to revert it to its power-up state.
• DCLK (input): 20 MHz clocking signal used to time sending and reading of DOUT.
• DVALID (output): active-low signal used to indicate that data is ready to be read on DOUT.
• DOUT (output): serial data stream of the 640-bit sequence (when in 20-bit precision mode) containing measurements of the 32 inputs.
• DIN (input): serial data input to the DDC used to daisy-chain other DDCs.

Nexys Video

While the Nexys Video can be controlled entirely from software, the on-board switches, buttons and status LEDs on the Nexys Video have the following assignments:

• LD0: Status for the Clocking Wizard. When lit, a stable 120MHz clock is being generated (converts the on-board master clock from 100MHz).
• LD1: When lit, the FPGA is in the idle state.
• LD2: When lit, the FPGA is in configuration, reading or sending states.
• LD3-5: These three LEDs display the current 3 FSR bits (see below).
• LD6: DDC232 error. When lit, it means that data is not being read fast enough from the DDC232. Typically occurs if the integration time is too short or if data cannot be written to the FIFO.
• LD7: FIFO error. When lit, the FIFO has either underflowed or overflowed.

• SW0: Toggles continuous/triggered acquisition modes. In the former, the FPGA will send data each measurement received from the DDC232 to the PC whereas in the latter, the user must press BTNL to send a measurement. Configuration read-back will occur automatically after a reset in both modes.
• SW1: Toggles DDC232 test mode. In test mode, the DDC232 integrators are disconnected from the inputs and will return dark noise values.
• SW2: Toggles assignment mode of FSR, integration time and test mode.
  • In order to allow FSR bits to be set with on-board switches as well as remotely, a switch was assigned to toggle between the two assignment modes. When in the low position, the FSR bits are set by switches 3-5. When in the high position, the FSR can be set over USB. This switch should be placed in the up position, otherwise commands from the PC are overwritten.

• BTNR: Global reset. When pressed, the FPGA resets itself and the DDC232 and automatically reconfigures with the current values set in the various switches.
• BTNC: Pause button. When pressed, the FPGA and DDC232 are idled. Pressing reset will resume operation.
• BTNL: Acquisition trigger.

Configuration

The DDC must first be configured with a 12-bit sequence of data sent on DIN_CFG, where:

• Bits 11-9 correspond to the 3 FSR bits, allowing for 8 different dynamic ranges; 000 = 12.5 pC, 001 = 50 pC, 010 = 100 pC, 011 = 150 pC, 100 = 200 pC, 101 = 250 pC, 110 = 300 pC, 111 = 350 pC. These correspond to the maximum charge that can be integrated in the photodiodes. 350 pC is typically chosen to minimise risk of saturation of inputs.
• Bit 8 corresponds to the resolution of output data: 0 = 16-bits, 1 = 20-bits. 20-bit resolution is chosen for better precision, at the price of slower readout (due to more bits of data requiring shifting out).
• Bit 7 corresponds to the device version. For the DDC232CK, this bit is set to 1.
• Bit 6 corresponds to a DDC internal divider of the CLK signal. This is set to 0 for no division.
• Bits 5-1 are empty bits set to 0.
• Bit 0 corresponds to the diagnostic test mode setting: 1 = on, 0 = off.

This gives a configuration input for normal operation of: 111110000000. Note that all vectors of data in this design are “little-endian”, in which the bits are labelled from the most-significant bit (MSB) to the least-significant bit (LSB) and are sent to and from devices MSB first. For the configuration input, this means bit 11 is sent first and bit 0 is sent last. After the configuration data is sent, a 640-bit (when in 20-bit mode, which is equal to the number of bits sent during a measurement readout cycle) read-back is sent to confirm configuration settings and test data output. The 640-bits are a 320-bit sequence sent twice, which contains the 12 configuration bits that the DDC received, a 4-bit revision ID (0001), 244 zeros and then a 70-bit test pattern, used as an extra check. In hexadecimal, the test pattern is: 30F066012480F69055.

After the DDC is powered up and power supplies have stabilised, a reset pulse of width t_RST = 1 μs must be sent and after t_WTRST = 2 μs, the configuration data is sent through DIN_CFG on the rising edges of CLK_CFG, to be read by the DDC on the falling edges of CLK_CFG. t_STCF and t_HDCF (both 10 ns) are the minimum times required for DIN_CFG to be valid before and after falling edges of CLK CFG respectively. t_WTWR = 2 μs later, configuration read-back begins on DOUT where the pattern is read on the rising edges of DCLK, after which CONV is strobed (i.e. high for one CLK cycle) to begin integration.

Integration

After CONV is strobed at the end of the configuration read-back, it begins alternating with a time period t_INT, which is the integration time set by the user. Toggles in CONV represent transitions in the dual-integrators, where one side completes integration and begins measurement, reset and auto-zeroing while the other side, currently idling after having finished these three tasks, begins integration. Measurement, reset and auto-zeroing take t_MRAZ = 1612 ± 2 CLK cycles and DVALID is asserted after t_CMDR = 1382 ± 2 CLK cycles, signalling that data ready to be shifted out on DOUT.

Data Transfer

Once DVALID is asserted, photodiode data can be shifted out to the FPGA on the rising edges of DCLK. Each of the 32 DDC inputs is represented by a 20-bit number, which can then be converted to a charge in pC. A key requirement for the DDC to be operated in continuous mode with this method, i.e. with zero dead-time between measurements (not to be confused with the FPGA continuous/triggered acquisition settings), is that the integration time must be sufficient to complete measurement, reset, auto- zero and shifting out of data before the next DVALID. Otherwise, the DDC enters a non-continuous mode where integration stops until the aforementioned processes have finished on both input integrators. Note that data retrieval pauses +-10us around a CONV toggle.

Regardless of whether the FPGA is in continuous or triggered acquisition modes, each measurement sent by the DDC is read, though they are not always sent to the PC (see below). The FPGA has a 640-bit register that is overwritten every time new data arrives. Before being sent to the PC, a measurement is first sent to a FIFO interface on the FPGA. The FIFO was added to the design using the Xilinx FIFO Generator and has the following signals:

• WR_EN: FIFO write enable. When high, a byte is written to the FIFO on clock rising edges.
• RD_EN: FIFO read enable. When high, the oldest byte in the FIFO is read on clock rising edges.
• FULL: High when the FIFO is full.
• EMPTY: High when the FIFO is empty.
• WR ACK: High when a byte has been written successfully.
• VALID: High when a byte has been read successfully.
• OVERFLOW : High if FIFO is written to when full and causes the Zybo RGB LED to turn blue.
• UNDERFLOW: High if FIFO is read when empty and causes the Zybo RGB LED to turn blue.

A 640-bit measurement received by the FPGA is written to the FIFO in byte-sized pieces, such that the FPGA is quickly ready again to read new data. The measurement is then sent from the FIFO to the PC either over the universal asynchronous receiver-transmitter (UART) protocol or using the FTDI chip in DPTI FT245 synchronous FIFO mode. The UART protocol caps out at 1Mb/s whereas the FTDI chip can easily achieve the speeds necessary to send out each measurement before the next arrives. For 1 board, this is about 0.45MB/s and for 5 boards, this is about 2MB/s. While the UART interface is no longer used, each protocol is discussed below.

FTDI DPTI

The DPTI interface is known as a FT245 style synchronous FIFO interface. The chip enters this mode with a software command on the receiver side. Data is written from the FIFO in the FPGA design to the FTDI FIFO, where it is sent to the PC on a software request. The key signals are:

• D[7:0]: 8-bit in/out data bus.
• RXF#: When low, data is available for reading from the FIFO (i.e. from the PC).
• TXE#: When low, data can be written to the FIFO (i.e. to be sent to the PC).
• RD#: Low-pulse triggers data to be read from the FIFO, received from the PC.
• WR#: Low-pulse triggers data to be written to the FIFO, to be sent to the PC.
• SIWU#: Send immediate or wake-up. A low-pulse either wakes up the PC, or causes the data currently in the FIFO to be sent at the next request from the PC.
• OE#: When low, data bus is driven by the USB controller (read transfer). When high, it is driven by the FPGA (write transfer).
• CLKO: 60MHz clock signal, data is read/written on rising edges.

With the DPTI interface, the FIFO in the FPGA design is an independent read/write clock FIFO. Measurements received from the DDC232 are written into the FIFO using the 120MHz master clock, however they are read from the FIFO and written into the DPTI FIFO using the 60 MHz CLKO signal. Once a measurement is read, SIWU is pulsed to ensure the measurement is sent at the next request from the PC. Like with the FPGA FIFO, measurements are written one byte at a time. This entire process should take place before the next measurement is read from the DDC232. On the PC side, all data transfer is controlled by a single C++ script that makes use of the library libFTDI.

Daisy-Chaining DDC232s

DDCs can be easily daisy-chained for applications that require more inputs. A full clinical-ready range detector would require 160 scintillator sheets (5 daisy-chained DDC boards, each with 32 channels) to give a WET of approximately 450 mm – though the detector can offer any multiple of 32 sheets. To daisy-chain DDCs, CLK, CONV, DIN_CFG, CLK_CFG, RESET and DCLK are sent to all boards as usual, however DOUT of boards in the chain are connected to DIN of the board next along the chain. The last board in the chain has DIN connected to ground and DOUT of the first board is connected to the FPGA as normal. DVALID is cascaded through OR logic gates on the circuit boards to indicate that all DDCs are ready to shifted data out to the FPGA. Note that increasing the number of DDC232 boards increases the minimum integration time, as there is more data to shift-out per measurement cycle. With 5 DDC232 boards, the minimum integration time is 181us.

Operation

Setting Up

• The Nexys Video is powered with a 12V 3A power supply while the DDC232 board is powered with 12V 1.25A supply.
• Photodiodes are connected to the DDC232 with the star symbol on the bottom side (closer to the PMOD connections).
• The DDC232 is connected to PMOD header JB on the Nexys Video with jumper cables with a 1:1 connection.
  • Pins 5 and 11 (3V3 pins) on the DDC232 need not be connected, as the DDC232 is powered with a PSU. Pins 6 and 12 on the Nexys Video JB are therefore disconnected.
• A micro-USB to USB-A cable is connected to PROG on the Nexys Video (for JTAG programming and DPTI) to the PC.
  • The DPTI interface and JTAG programming use the same port, however using libFTDI will deregister the USB such that Vivado will no longer be able to see it (for configuration or debugging). As such, if you wish to debug and transfer data at the same time, a modification has to be made with a 6-pin PMOD header soldered onto junction J17 to open a separate channel for JTAG programming. This will then make use of a JTAG-HS2 cable.

Configuring the FPGA

• Opening the Vivado project containing the code to communicate with the DDC232, open up the hardware manager and press auto-connect. If the Nexys Video is connected and powered on, it should automatically be found. Then, press program device.
  • The current debug core will be opened, but this can be ignored (unless you are debugging).
  • Ensure the LD6 remains unlit, indicating correct configuration.
  • Press BTNC to pause operation for the time being.
• If using DPTI, close Vivado and run the FTDI code.
• More conveniently, the .bit file found in the implementation runs folder in the Vivado project can be loaded onto an SD card to automatically program the Nexys Video without Vivado when powered on.
  • JP4 must be in the USB/SD card position and JP3 must be in the SD card position.

Installing libFTDI

MacOS

• On MacOS, libFTDI is simply installed with MacPorts sudo port install libftdi1 and the code can then be compiled, ensuring the correct paths are given to g++, e.g. g++ -L/opt/local/lib -I/opt/local/include/libftdi1 -lftdi1 -std=c++17 FTDI.cpp -o FTDI

Windows

• On Windows, you will need to replace the default USB driver in addition to installing libFTDI. This guide provides information on replacing the driver.
  • For the Nexys Video, you will need to replace the default driver for the first channel listed with libusbK.
• Download libFTDI 1.5 for x86 x64 from picusb sourceforge page and Boost, unpacking the contents in a useful directory (directly in C:\ is a good idea).
• Set up a C++ project on Microsoft Visual Studio and in the project properties:
  • Under Configuration, set All Configurations to Platform x64.
  • Under C/C++, add the additional include directories for libFTDI and boost, e.g. C:\libftdi1-1.5\include\libftdi;C:\boost_1_74_0.
  • Under Linker, add the additional library directories for libFTDI and boost, e.g. C:\libftdi1-1.5\lib64;C:\boost_1_74_0\libs
• You will also need to copy the file libftdi1.dll.a (found in C:\libftdi1-1.5\lib64) into the solution directory for your Visual Studio project.
• In the main editor view, set the Debug platform to x64 (next to the 'Local Windows Debugger' button).
• If you wish to simultaneously use Vivado and run the C++ code, ensure that you run the libFTDI code before opening Vivado (the Nexys Video doesn't need to be programmed, just turned on and connected) so that the default PROG port is deregistered and Vivado only sees the JTAG-HS2 cable connection in the hardware manager, this will prevent any clashes.

Compiling and Running FTDI Code

• The files required for DAQ are tools.h and FTDI.cpp.
• On MacOS, the code is compiled with g++ using: g++ -L/opt/local/lib -I/opt/local/include/libftdi1 -lftdi1 -std=c++17 FTDI.cpp -o FTDI
• On PiOS, the code is compiled with g++ using: g++ -L/opt/local/lib -I/opt/local/include/libftdi1 -lftdi1 -pthread -std=c++17 -Wno-psabi FTDI.cpp -o FTDI
  • Note: adjust library and include paths according to your installation of libFTDI
• To run the code execute: ./FTDI 4 350 170 100000 50 (test)
  • The arguments are the number of DDC232 boards, the FSR (written exactly as in DDC232 datasheet), integration time (units of 5us only), number of measurements to acquire (set to 0 for infinite loop), the refresh rate in Hz of a rolling average written to output file and the optional argument to enable test mode (no brackets).
  • The output of FTDI.cpp is a two-line CSV file with the average charge in each photodiode in the first line and the standard deviation of each in the second line. This file is overwritten at the rate given in the 5th argument of FTDI.cpp.
• Windows will require Visual Studio to compile and run the code.
• To display the live photodiode values or perform fits, see this page for details on analysis scripts.
• Once compiled, useful combination of scripts can be conveniently run simultaneously using a bash script, see the file prog.sh for more details.
• To exit the programs prematurely and cleanly, quit any ROOT plots before using CTRL-C to exit the FTDI script.

Uploading Data to HEP Server

Data can be uploaded to the HEP cluster in order to run a webpage with live photodiode data.

• On your local machine:
  • Move to the directory containing photodiode data being saved by CoolTerm and the C++ script to generate the CSV file: cd /Path/To/Data
  • Generate the photodiode data with: root -l "writeCSV.cpp"
  • In a new terminal, move to the same directory and run the following command in order to copy this data, line by line, to the plus1 web area: tail -F -n 1 output.csv | ssh username@plus2.hep.ucl.ac.uk -T 'cat - > /unix/www/html/pbt/PDdisplay/photodiodeValuesAll.csv'
• On plus2 :
  • In a new terminal, move into the directory where the photodiode data is written: cd /unix/www/html/pbt/PDdisplay/
  • Run the following script in that directory: ./outputSingleLine.sh -v photodiodeValuesAll.csv photodiodeValues.csv 0.02
    • This will output a single line CSV file - photodiodeValues.csv - every 10ms (100Hz). It's a good idea to set the write rate about 2x faster than the desired frame rate.
  • To output this any more or less often, adjust the "waitTime" value at the end (needs to be in seconds).
  • To stop displaying each line as it is written, remove the -v flag.
  • Note that this script contains an infinite loop, so to stop it running you need to kill it by hand with Ctrl-C. You may need to do this a few times to get it to quit...

Legacy Information

The first version of the FPGA interface with the DDC232 used a Zybo Z7-10 and transferred data to a PC over the UART protocol. The board was eventually replaced with a Nexys Video for the larger FPGA and the FTDI chip to allow fast USB communication. Information specific to the Zybo is left here for reference.

Zybo Z7-10

Details of the design-assigned purpose of the on-board buttons, switches and LEDs are provided below:

• Clocking Wizard Status LED (LD0): the 125 MHz FPGA master clock is converted to a 120 MHz clock using the Xilinx Clocking Wizard intellectual property (IP), to allow for easier generation of the 10 and 20 MHz signals required by the DDC. This LED is on when the Clocking Wizard component of the design is generating a stable clock. All FPGA operations are timed using this clock.
• Test-mode LED (LD1): the DDC can be configured into a test diagnostic mode for debugging, in which all the inputs give a zero signal (slightly above zero due to noise and a negative current offset). This LED is on when the user has enabled the test mode, which can be toggled using the Toggle Test-Mode button (BTN3) and becomes active after a Global Reset press.
• State LED 1 (LD2): this LED is on whenever the DDC in the power-up, idle, or configuration state.
• State LED 2 (LD3): this LED is on whenever the DDC is measuring or shifting out data.
• Acquisition mode (SW0): this switch toggles between the continuous and triggered acquisition modes. In continuous mode, data will be output to the PC as fast as possible, the rate of which is dependent on the chosen integration time and the speed of the data transfer. When in triggered mode, the next measurement after the Trigger Acquisition button (BTN2) is pressed is sent to the PC.
• FSR Switches (SW1-3): these 3 switches control the 3-bit FSR code, which allows the user to choose between the 8 different dynamic ranges.
• Global Reset (BTN0): this button resets all aspects of the FPGA and DDC. Any changes since the last reset in the FSR, acquisition mode or test-mode settings are applied. The DDC returns to its power-up state and reconfigures before resuming operation.
• Pause Operation (BTN1): this button idles the FPGA and DDC, all signals are held at their default value apart from the 120 MHz master clock. Pressing Global Reset restarts operation.
• RGB LED (LD6): this LED will be green if the DDC is configured correctly, red if configuration is incorrect or if too short an integration time is chosen, or blue if the first-in first-out (FIFO) interface is read when empty or is written to when full.

UART

The key user settings are:

• Baud rate: the number of bits transferred per second (including start and stop bits). This is set to 921600, which is often the maximum standard baud rate for serial ports.
• Data bits: the number of bits of data sent in a packet. This is set to the maximum number, 8.
• Stop bit: the number of bits at the end of the data bits denoting the end of data. This is set to the minimum, 1.

Key signals for the UART transmitter used to send data from the FPGA to the PC:

• TX_DV: Data valid signal sent to the UART transmitter to state that data is ready for transmission. Functionally, this serves the same purpose as the FIFO VALID signal.
• TX_ACTIVE: High when the UART transmitter is sending data to the PC.
• TX_DONE: High for one clock cycle after a byte is sent.
• TX_SERIAL: Actual data stream sent to the PC.

In triggered mode, the next measurement read by the FPGA is stored into the FIFO and then sent over UART. Subsequent measurements are only read, to be overwritten by the next until the button is pressed again. At the falling edge of DVALID, DCLK begins to shift in data on DOUT, after which the measurement is stored in bytes into the FIFO. The UART transmitter then sends data one byte at time. When the FPGA is in continuous mode, the next read measurement is sent to the FIFO as soon as the UART transmitter has finished sending the last measurement (i.e. when the FIFO becomes empty again). This was chosen so that the FIFO does not fill up over time if the UART transmitter cannot keep up and practically means that the FIFO will only ever contain a maximum of one measurement at any given time. While not the most conventional use of a FIFO, it allows for the DDC integration time to be independent of the data transmission speed: the shortest integration times are still available as the time taken for a measurement to be stored into the FIFO is negligible compared to the UART transmission rate. Note that a 32-bit custom end-of-line sequence is added to the end of each measurement, used to separate measurements when saved to a text file on a PC. Since each byte sent over UART contains a start and stop bit, there are actually 640 + 32 + 84(2) = 840 bits to be sent over UART for the (640+32)/8 = 84 bytes of data saved in the FIFO per measurement.

Data sent over UART is saved to a text file using the CoolTerm serial terminal emulator. This program was chosen for its ability to view and save incoming data as hexadecimal (instead of ASCII), accept the 921600 baud rate and accept custom end-of-line strings such that each measurement is timestamped and saved to a new line in the text file, discarding the end-of-line string. While the application saves data, a C++ script is run that reads the latest measurement, converts it to charge values in pC and then plots the result live in a graph using ROOT. ASCII commands can also be sent from the PC to a UART receiver in the design, which is similar to the transmitter, to reset, pause and trigger acquisitions.

Setting Up

• The Zybo Z7 is powered with a 5V 2.5A power supply – ensure the power supply jumper on the Zybo is set to WALL.
• The DDC232 is connected to PMOD header JC on the Zybo with jumper cables with a 1:1 connection on the 2nd DDC232 prototype. On the first DDC232 prototype pins 6 and 12 (GND pins) are connected to pins 5 and 11 on JC.
  • Pins 5 and 11 (3V3 pins) on the DDC232 need not be connected, as the DDC232 is powered with a PSU. Pins 6 and 12 on JC are therefore disconnected.
• A Digilent USB-UART PMOD adaptor is connected to pins 1-6 of PMOD header JD on the Zybo.
• Micro-USB to USB-A cables are connected from the USB-UART adaptor (for data transfer) and PROG UART on the Zybo (for configuration) to the PC.

CoolTerm

• Apply the following setting in CoolTerm options:
  • Under Serial Port:
    • Port: Select the port that the USB_UART adaptor is connected to.
    • Baudrate: 921600
    • Data bits: 8
    • Parity: none
    • Stop bits: 1
    • Flow control: Leave boxes unchecked
  • Under Receive:
    • Receive buffer size: 84 (equal to the number of bytes per measurement including termination string)
    • Capture format: Hexadecimal
    • Leave "Format Hexadecimal Data" unchecked
    • Check "Add timestamped to received data" with Type "Absolute time" and check "Wait for termination string", which is set to "0D 0A 0D 0A". Uncheck "retain termination string"
    • All other boxes should be unchecked.
• Open the connection to the port.
• To begin capture, press CMD+R and save data as "capture.txt" to the same directory containing the writeCSV.cpp macro.
• Ensuring SW1 is set to 0 (continuous mode), press BTN0 to reset the FPGA and resume operation.
  • Capture can be stopped by pressing SHIFT+CMD+R.
• To view incoming data on a local machine, you can use the ROOT macro plotLive.cpp, executed in the same directory that data is being saved: $ root -l "plotLive.cpp"
  • Make the canvas smaller to increase the frame rate.
• Close the connection to the port before quitting the application.