Multiple Devices on SPI Bus not working (TMC Steppers and Display)

[jjarosz]

Just a recap:

• 5 TMC2130’s in SPI mode
  using a MISO, MOSI, SCK, and each TMC a unique Cs
• 1 MKS Mini on same SPI bus
  using a MOSI, SCK and LCD_Cs along with LCD_A0 and LCD_RST
  LCD is not using MISO.

I’ve tested on this prototype board along with a RAMPS on MEGA with the same results:

• LCD by itself works.
• TMC by themselves work.
• Combined LCD and TMC’s; only the display operates.

Investigated some other board schematics and noticed that multiple SPI buses/pins are available.
Some board implement between 2 hardware SPI’s and implement 2 hardware and a single software. Sometimes the spare pins are available at either the step stick or the display ports for software SPI.

Some thoughts:
Klipper SPI issue - as @koconnor stated provide an option for testing. Thank you! As the CAN Bus suggestion is inexpensive, I’m willing to obtain one and test in order to provide results.

MKS Mini Display issue - Display does not provide any communication on MISO back to the SPI bus, so that could be the root cause? One way to test is to obtain a different SPI display that provides 2-way communication. We are also looking at moving some pins around to allow for Software SPI at either Step Stick or Display.

SPI Issue - Either of the two above testing may also answer this. The fix would be to allow for multiple SPI bus options.

I have also one more MCU on hand to test: Re-ARM, but I am almost expecting the same results.

Thanks for the input and suggestions. More to come in the near future.
-James

[mykepredko]

Could you provide a drawing (with wire lengths) showing how the SPI lines are wired? Is there any chance you can get oscilloscope pictures of the signals?

If you have six loads (5x '5160s and LCD) on the SCK, MISO and MOSI lines along with different lines of different impedances, I think you’re running into a signals integrity issue.

As somebody else pointed out in the thread here, SPI is meant for a PCB with standard 50 Ohm characteristic impedance. If you’re adding wires off the board, you’re going to have problems - especially if you have some fairly long wires. Looking at the datasheet, each SPI pin has a typical 3.5pF pin capacitance which means that just for the '5160s you are getting up to 20pF total capacitance on each line which is not insignificant.

Can you split the LCD into another SPI bus (using bit-banging)? That might be your simplest way to get things working.

[jjarosz]

@mykepredko

If you have six loads (5x '5160s and LCD) on the SCK, MISO and MOSI lines along with different lines of different impedances, I think you’re running into a signals integrity issue.

Even with 1 TMC and display, the same behavior exists.

@mykepredko

Could you provide a drawing (with wire lengths) showing how the SPI lines are wired? Is there any chance you can get oscilloscope pictures of the signals?

The SPI wires from MCU to step stick are baked into the shield. I’ll inquire on the max length of the longest leg. As for the display, it’s using a standard 10-conductor ribbon cable at 30cm long.
I don’t have an oscilloscope, however a logic analyzer is on the way and I’ll provide feedback as soon as I can.

@mykepredko

Can you split the LCD into another SPI bus (using bit-banging)? That might be your simplest way to get things working.

That is our thought as well. We will be looking into this as we are limited to the number of pins exposed / available that a MEGA, DUE, ReArm and Grand Central boards offer.

[mykepredko]

@jjarosz

The SPI wires from MCU to step stick are baked into the shield. I’ll inquire on the max length of the longest leg. As for the display, it’s using a standard 10-conductor ribbon cable at 30cm long.
I don’t have an oscilloscope, however a logic analyzer is on the way and I’ll provide feedback as soon as I can.

A “standard ribbon” cable is significantly different from a PCB in terms of characteristic impedance and line capacitance.

https://multimedia.3m.com/mws/media/22052O/3mtm-round-conductor-flat-cable-050in-3365-series-ts0080.pdf

A logic analyzer is not the right tool for the job as you can’t see what the actual signal looks like on the line and chances are the probes will not have the high impedance at their ends, like an oscilloscope probe’s does, which means they will become part of the circuit - although you will probably have your problem identified if you connect the logic analyzer to your SPI bus without the LCD and you have communications problems.

Honestly, SPI (such as it is defined) is not meant for an application like this one as it’s meant for on board communications over short distances. I understand that you’re I/O constrained but maybe there’s some other way to connect up a user interface.

Good luck, I know you’ve spent a lot of time on this but I don’t think you have a viable approach.

[koconnor]

Separately, as a quick test, you could try purposely misconfiguring the cs_pin in the [display] section. If the display continues to work with an incorrect cs_pin it indicates something is forcing the actual cs wire low on the uc1701 chip - which would also explain your issues.

-Kevin

[jjarosz]

SPI-lengths.zip (4.4 MB)

Attached are some images of the board trace layout.
Each image is of a single trace MISO, MOSI, and SCK
The ECAD software does not have a measure tool for the traces, so the lengths have been estimated:

• MOSI 190mm
• MISO 240mm
• SCK 190mm

@koconnor

Separately, as a quick test, you could try purposely misconfiguring the cs_pin in the [display] section. If the display continues to work with an incorrect cs_pin it indicates something is forcing the actual cs wire low on the uc1701 chip - which would also explain your issues.

I have done this with the same results; display works, but the TMC drivers do not. Do you think that forcing the pin to trigger HIGH in the make menuconfig for the MCU make difference?

I have a SH1106 SPI display on order with the Logic Analyzer and will further test this weekend.

We are also looking at adding a 4050 buffer chip, although if the issue is the uc1701 display, that will not be fruitful. We have some other pins nearby both the display and step sticks that we can provide those for Software SPI as well.

Thanks for the continued suggestions and I’m confident our testing will provide the needed answers.

Cheers!
James

[mykepredko]

NOTE: Edited to put in some lists to make things a bit more readable.

Is this a custom board? Looking at the images, it’s a two sided board - personally for something like this I would recommend a four layer board, with one of the two inside planes being ground and the other being stepper/logic power.

Regardless, from a signals integrity perspective, it isn’t great. Not having a continuous ground plane, you don’t have any impedance control of your signal traces and there are a number of T joints (which are never recommended for communications).

Looking at the schematic and images you sent, I believe the following connections are involved:

• AUX3/J32 - Arduino connection
• J7 - Jumper block for ReArm programming - they look like the jumpers are installed in IMG_6722 provided earlier and lines are going to D50/D51/D52 but I can’t find the connector on the schematic you provided
• IC2 - 4050 CMOS buffer - leads to “EXP2” and I’m assuming this is the LCD connection?
• ICSP - No reference designator other than “ICSP”
• X, Y, E0, E1 & Z - ‘5160 connections/No reference designator other than “X”, "Y’, etc.
• J8-D63/D67-J10 for MISO when running NTC?

I must say it’s really hard to reconcile the labels on the schematic you provided with the PCB images, so despite spending about an hour trying to figure everything out, I’m sure I’ve made mistakes.

For MOSI, your trace path is:

• J32/AUX3-J7 (Arduino to ReArm Jumper block)
• J7-IC2 (ReArm Jumper block to LCD driver)
• IC2-Z (LCD connection to Z 5160 stepper connection)
• Z-ICSP (5160 Stepper connection to programming header)
• ICSP-E1 (5160 stepper connections)
• E1-E0-X-Y (5160 stepper connections with two T connections

So, in this case, you have your MOSI line with three stubs running off it (going to ReArm Pins via jumper, E0 and Y). The MISO trace also has three stubs and SCK has two.

First off, can you run things without the J7 jumpers? That takes out one stub for each of the signals as well as the pin load on it. You might see some differences in operation with this.

If you have a few hours and somebody who’s good with reworking PCBs, I would recommend that you cut the MISO, MOSI & SCK lines at the vias where there are T connections (cuts ideally less than 2mm away from the via on all traces going to it) and run fly wires with the route:

J32/AUX32-Y-X-E0-E1-ICSP-Z-IC2-J7

For MOSI and SCK, you have to cut the traces from J32/AUX to J7, T connection at X/Y and T connection at E0/E1 and run fly wires from J32/AUX to Y, X to E0. In the image below, white “X” indicates where to cut and the white lines are the fly wires:

MISO will need to more time than I’m willing to spend now.

Anyway, first thing is to pull the jumpers in J7 (if possible) and see if that fixes anything. Before you start cutting and running fly wires, I highly suggest that you get an oscilloscope (at least 100Msa/s) and look at the signals to see what they look like (post pictures from the 'scope here).

[mykepredko]

@jjarosz

I just realized that you never answered my question about IC2 - are you connecting the LCD to it?

If so, it’s driving the MISO line in a state that the '5160s can’t overcome. You have a bus contention problem, not a signals integrity issue (although that could be part of it).

[jjarosz]

I have a SSD1306 on i2c, which is on sercom3
The uc1701 and TMC are on sercom7

I’ll comment out the SSD1306 and test without it.

[jjarosz]

Disabled SSD display and sercom3 with no difference.

I have a different OLED SPI display I’ll try next. That should tell me if the issue SPI based or MKS mini display based.

[mykepredko]

Hi @jjarosz

Just to make sure I understand the situation on the SPI bus:

1. You are using EXP-2 as the connector to the SPI user interface
2. When just TMC5160s plugged in, they communicate with the Arduino okay
3. When just the SPI user interface is attached, it communicates with the Arduino okay
4. When you have one or more TMC5160s plugged in AND the SPI user interface attached, the SPI user interface works okay and the TMC5160s do not
5. If you disable the SPI user interface (I’m guessing in printer.cfg) then the TMC5160s work okay

Is this correct?

If it is, I’m very sure you have a bus contention issue with the SPI user interface MISO output idle output of IC2 (Pin 2) overpowering/affecting the TMC5160s’ SDO (same as MISO) output.

I can’t find the IoL/IoH values for the TMC5160 or the pin equivalent circuit so I can’t say what’s happening there. The 4050’s IoL/IoH values are fairly modest (around 5mA) AND you’re going through a 10k resistor which should eliminate this problem…

Could you check and see that the 10k resistor (R2) is actually there and is 10k?

Barring that, you’ll need to get an oscilloscope and observe the MISO (and other SPI) signals as I suggested.

You could try something like using a voltage meter and following these steps:

1. Populating the TMC5160 sockets, removing the SPI user interface, powering up and observing the signals on MISO.
2. Remove the TMC5160 modules, connect the SPI user interface, powering up and measuring the voltage on MISO.
3. Populate the TMC5160 sockets, connect the SPI user interface, powering up and measuring the voltage on MISO.

I’m not sure what you’ll see but maybe there will be something useful.

[jjarosz]

@mykepredko

Looking at the schematic and images you sent, I believe the following connections are involved:

• AUX3/J32 - Arduino connection
• J7 - Jumper block for ReArm programming - they look like the jumpers are installed in IMG_6722 provided earlier and lines are going to D50/D51/D52 but I can’t find the connector on the schematic you provided
• IC2 - 4050 CMOS buffer - leads to “EXP2” and I’m assuming this is the LCD connection?
• ICSP - No reference designator other than “ICSP”
• X, Y, E0, E1 & Z - ‘5160 connections/No reference designator other than “X”, "Y’, etc.
• J8-D63/D67-J10 for MISO when running NTC?

For reference, here is the Adafruit Grand Central Pin Layout: https://github.com/adafruit/Adafruit-Grand-Central-PCB/blob/master/Adafruit%20Grand%20Central%20M4%20Express%20Pinout.pdf

On Arduino Mega/DUE and Grand Central, there is an “ICSP” header (6 pin) in the middle of the board which can supply the MISO, MOSI and SCK. The thought was to use the ICSP header as it is centrally located for the Steppers.

On a Re-ARM board, there is no ICSP header, so Pins 50, 51, and 52 are needed for SPI.
Arduino DUE, pins 50-52 are not SPI, but other pins.

The Arduino MEGA, Re-ARM, and Grand Central can use MISO is on pin50, MOSI is on PIn51, and SCK is on Pin52.

J7 is not installed.

IC2 buffer would go to EXP-2 for the display, it is not installed at the moment, the pins are jumped.

Schematic Page 9 has stepper driver info for pins.

J8 jumper is to adjust for Arduino DUE pin differences, related to the Hot End, Heater Bed, and Chamber NTC Thermistors.

[mykepredko]

Rather than me asking more and more questions can you concisely explain how the SPI lines are wired and how the different devices are connected?

All I’ve had to work with is a very poorly laid out schematic (sorry, but it’s really hard to follow, doesn’t seem to be complete in terms of the Arduino/ReArm pins, pin labels are in different formats and you can’t do searches on text strings), a couple of pictures that don’t really show what’s going on and three of each of the different SPI lines in the CAD system. Now you’re saying that a buffer that connects the different SPI lines to the user interface isn’t installed which can affect things a lot and I have no idea what is going on with J8 and why it’s significant. I, along with other people here, would like to help you but you’ve got to do some work to make it easier for us.

Please, simply explain the connections, taking a few minutes to draw them out would be helpful, and try to look at things from the perspective of people who aren’t as intimately familiar as you are with the design and the current state of the wiring.

[jjarosz]

Spent some time investigating with other devices today.

Configured a Pi Hat SH1106 4wire SPI device via Software SPI and found a stable config.
Moved pins around to be Hardware SPI and the display and TMC did not function.

Besides the MKS Mini display being a uc1701 chip and the Pi Hat being SH1106, the MKS mini display has a 4050 buffer ship on the in bound SPI to the Display and SD lines.

I started digging into SPI lengths, etc, and only discovered that, in theory, this “should” work.

Did more research and finally found this: TMC2130 & UC1701 issue · Issue #1208 · Klipper3d/klipper · GitHub

Now I did check github, but did not run across this specific case. I was looking for “st7567” and “MKS Mini” in my previous searches. This time I did a internet search for: uc1701 spi speed and found the above Klipper issue.

This issue noted that adding spi_speed: 4000000 to each SPI location (TMC and Display), all devices on he bus work together.

Made the change and indeed, all 5 TMC’s and the display are functioning on the same Hardware SPI bus.

The SH1106 display also needs spi_speed: 4000000 in the .cfg.

Not sure if here are any con’s yet, or if the issue was the display was setting one speed that the TMC’s didn’t like?

[jjarosz]

Just some reference material that I’ve found:

uc1701 on Raspberry Pi Project: uc1701/uc1701.c at master · bitbank2/uc1701 · GitHub
Project initializes the SPI bus to 4Mhz

sh1106 - Waveshare Raspberry Pi Hat: https://www.waveshare.com/product/displays/oled/oled-3/1.3inch-oled-hat.htm
Project initializes SPI bus to 10Mhz. I’ve configured this under Klipper with 4Mhz and it works.

This project initializes SPI bus at 1Mhz.

GitHub - 8TN/Raspberry-Pi-SH1106-oled-display: Basic python script to display...

TMC Drivers - Work using the internal clock speed of 4Mhz. I have only tested/used 2130’s.
2130: https://www.trinamic.com/fileadmin/assets/Products/ICs_Documents/TMC2130_datasheet.pdf
2660: https://trinamic.com/fileadmin/assets/Products/ICs_Documents/TMC2660_datasheet_Rev1.07.pdf
5160: https://www.trinamic.com/fileadmin/assets/Products/ICs_Documents/TMC5160A_Datasheet_Rev1.14.pdf

2240: https://www.trinamic.com/fileadmin/assets/Products/ICs_Documents/TMC2240_datasheet_rev1.pdf
Has a max of 10Mhz:

I have also tested setting the spi_speed in klipper just on the display, leaving the TMC settings without and there was no problems running.

Question: In what order does Klipper start configured components?

Thanks!

[jjarosz]

Additional Discovery, following @koconnor suggestion of the CAN Bus logic decoder process noted above. Much appreciation for that article as the low-cost device provided some needed insight.

Here are my findings:

sh1106 display - Klipper does not initialize the display UNLESS an spi_speed is specified in the display section.

u1701 display (MKS Mini) - Klipper will initialize this display with no spi_speed in the display config. The u1701 chip set appears to set a 12Mhz SPI CLK speed.

TMC2130 - Klipper will initialize the TMC drivers with no spi_speed in the tmc_xxxx config and at 4Mhz SPI CLK speed.

From what we can see, all devices on the same SPI Bus need to sync to the same CLK speed, and at the lowest common denominator; 4Mhz in the case of using TMC2130.

This also explains why we only need to set the spi_speed in the display as the TMC are the slower device and will initialize at their highest CLK speed.

Conclusion:
Setting spi_speed: 4000000 in the display config will allow the u1701 MKS Mini display to work on the same SPI bus at TMC step sticks in SPI mode.

I can provide some input for TMC, Display, and SPI sections of Klipper Doc’s as well as a SPI document similar to the CAN Bus doc in the near future.

Note: The topic was: Multiple devices on Adafruit (samd) SPI bus (TMC Steppers and Display). Updated to be more generic towards SPI Bus.

[koconnor]

@jjarosz

all devices on the same SPI Bus need to sync to the same CLK speed, and at the lowest common denominator

This is not required in the general case. It is possible that the Klipper samd mcu code is not correctly changing the speed as it communicates with each chip. I can’t tell from your logic analyzer snap screenshots if that is the case though. Can you check with the logic analyzer that communication with the tmc driver is incorrectly occurring at 12Mhz (going back to the standard non-working printer.cfg)? A simple way to do this would be to restart the printer, wait a few seconds for everything to normalize, set the logic analyzer to trigger on falling CLK, start the logic analyzer, and then issue a DUMP_TMC command in Klipper.

Thanks for troubleshooting this issue.
-Kevin

[jjarosz]

@koconnor,
I used PulseView to capture and I’ve saved the .sr file and attached it. The very last bit of data was the dump_tmc. klippy.log attached for review as well

klippy.log (995.3 KB)

12Mhz_MKSdisplay_TMC2130.zip (21.3 KB)

It looks like the Display Cs is always low, so there is data always traveling to it. The display only has 1-way communications on the SPI Bus: CS, RST, A0, SCK, and MOSI. Either that, or the SPI code doesn’t raise the Display Cs when lowering other devices Cs to allow for a clock frequency change?

I have not grabbed data from the Arduino MCU, but the same issue existed there as well.

[mykepredko]

I’m guessing “X Cs” is the chip select line for the X axis TMC5160. Did you record what the MISO bit values are when the LCD is not connected?

I’d be interested in seeing if you can detect a difference with the logic analyzer.

[jjarosz]

The X Cs is for the TMC driver
Display Cs is for the Display
The two SPI graphs at the bottom are the TMC and Display respectively.

The top image shows the SPI communications during the DUMP_TMC.

The bottom image is zoomed in to one clock cycle showing the 12Mhz SPI clock rate.