nodes failed to create tap [multiple networks]

[Omarabdul3ziz]

it was reported that node 11 on dev net couldn't provision a deployment due to a limit on tap devices. it is not happening now. but worth checking.

[0oM4R]

threefoldtech/tfgrid-sdk-ts#3855

[sameh-farouk]

I noticed that behavior on node 1 too (main net)

2025-02-11 09:30:19	
[+] provisiond: 2025-02-11T08:30:19Z error failed to deploy workload error="could not set up tap device: could not set tap master: exchange full" deployment=837572 name=vm_pulumi twin=10972 type=zmachine

[AhmedHanafy725]

also, it happens on qanet node 2

[scottyeager]

I saw the same on mainnet node 1 today as well.

[Omarabdul3ziz]

during VM deprovisioning either ygg or myc was being deleted, as both were previously considered "Planetry" this led to the br-ygg bridge on some nodes accumulating a large number of unused interfaces, eventually reaching the limit of 1024 interfaces.

For example on node dev:11

~ # brctl show br-ygg | wc -l
1024

found articles referring to linux bridges have a hard limit of 2^10 = 1024 interfaces, as referenced in br_private.h. when this limit is reached, it results in exchange full and the value is not configurable

as a fix:

• modified the deprovisioning code to delete both ygg/myc when either is found
• run a method at networkd startup to clean unused tap devices

[Omarabdul3ziz]

verified on dev net nodes 11,14

• by deploying vm with both nodes with ygg/myc
• catching the update on both nodes
• all other interfaces besides the test vms was deleted
• the vm interfaces still there and accessible

Now node dev:11

~ # brctl show br-ygg | wc -l
4

and

this shows

• which network id node have for my contract
• all the bridges/interfaces related to this network id (before i cancel my deployment)
• after canceling all are deleted