Import very slow and not adding interfaces correctly to all device templates

[KPTheProf]

I previously imported all the device templates using this tool last year and it ran within a few minutes.
I've just tried to update the device templates and it ran so slowly, I had to terminate after many hours.

Looking at netbox logs, at the time, it appears to be querying for every interface_template instead of just for that device type (we currently have over 68,000), and it does this for every device it's importing!

2024-09-18 13:30:41 172.18.0.1 - - [18/Sep/2024:12:30:41 +0000] "GET /api/dcim/interface-templates/?devicetype_id=3201&limit=1000&offset=68000 HTTP/1.1" 200 573608 "-" "python-requests/2.32.3"

I have also found not all devices are having their interfaces added correctly.
As an example, I ran the following command to import a single device from T-Bao.

This is the yaml file:

---
manufacturer: T-bao
model: MN58U
slug: t-bao-mn58u
u_height: 0
is_full_depth: false
airflow: passive
interfaces:
  - name: eth0
    type: 1000base-t
  - name: eth1
    type: 2.5gbase-t
  - name: wlan0
    type: ieee802.11ax
power-ports:
  - name: PSU
    type: usb-c

This is the command:

docker run --rm --env "NETBOX_URL=http://host.docker.internal:8000/" --env "NETBOX_TOKEN=b6996cbab753b10a9eb392e6acc43505eb8167da" netbox-device-type-library-import python3 -u nb-dt-import.py --vendors t-bao --verbose

This is the result:

Package Installed https://github.com/netbox-community/devicetype-library.git
1 Vendors Found
1 Device-Types Found
Manufacturer Exists: T-bao - 139
Device Type Exists: T-bao - MN58U - 3201
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
Script took 0:01:08.028536 to run
0 devices created
0 images uploaded
0 interfaces/ports updated
0 manufacturers created

I've attached the output when run with the -v to give more detail.
Device-Type-Library-Import.txt

Running Netbox version 4.1.1 in a Docker container.

[KPTheProf]

I have also tried first deleting the device type before running the import, but it had the same results and the interface-templates were not created.

docker run --rm --env "NETBOX_URL=http://host.docker.internal:8000/" --env "NETBOX_TOKEN=b6996cbab753b10a9eb392e6acc43505eb8167da" netbox-device-type-library-import python3 -u nb-dt-import.py --vendors t-bao --verbose

Package Installed https://github.com/netbox-community/devicetype-library.git
1 Vendors Found
1 Device-Types Found
Manufacturer Exists: T-bao - 139
Device Type Created: T-bao - MN58U - 3213
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
Script took 0:01:03.742596 to run
1 devices created
0 images uploaded
0 interfaces/ports updated
0 manufacturers created

[KPTheProf]

I have also attempted after first deleting the device and manufacturer but again the interfaces are not created.

docker run --rm --env "NETBOX_URL=http://host.docker.internal:8000/" --env "NETBOX_TOKEN=b6996cbab753b10a9eb392e6acc43505eb8167da" netbox-device-type-library-import python3 -u nb-dt-import.py --vendors t-bao --verbose

Package Installed https://github.com/netbox-community/devicetype-library.git
1 Vendors Found
1 Device-Types Found
Manufacturer queued for addition: T-bao
Manufacturer Created: T-bao - 157
Device Type Created: T-bao - MN58U - 3214
Modules Enabled. Creating Modules...
0 Module Vendors Found
0 Module-Types Found
---
Script took 0:01:03.394234 to run
1 devices created
0 images uploaded
0 interfaces/ports updated
1 manufacturers created

[parmiig]

I'm facing the same issue.

I started the script after upgrading to 4.1.1. The last time it ran was around 6 months ago with v3.7 and it took minutes. It's currently running since 24 hours and still going.

[chatasos]

I'm also facing the same issue with interfaces not being added for many device/module types.
Running 4.1.1, but i believe i had the same issue in 4.1.0 too.

[gurubert]

The issue comes from the import script as it tries to update existing entries.
It first fetches all interface and other components for a device type and then compares this with the yaml definition.

Does the import tool really need to update existing entries?

[KPTheProf]

The issue comes from the import script as it tries to update existing entries. It first fetches all interface and other components for a device type and then compares this with the yaml definition.

Does the import tool really need to update existing entries?

The issue that I'm seeing is that it is fetching all interfaces for all device types, not just the one it is working on at the time.
It would seem that there is an issue with the filtering being used.

[yanecisco]

it is related to this netbox-community/netbox#15410
devicetype_id -> device_type_id
moduletype_id -> module_type_id

[chatasos]

Thas was indeed the cause for my issue. I did the changes and it parsed correctly the manufacturer i gave as input.

[hikhvar]

I tried the fix @chatasos implemeted and it worked for me as well. Do you mind to create a pull request @chatasos? I think this will help others as well.

[hikhvar]

Never mind, I just found #156 which is roughly the same.

[chatasos]

I have already created #155.

I don't think the developers are actively looking into the project any more.