Opnsense suricata rule update for ET Telemetry

[planetf1]

I’m running opnsense 24.7.3 & have been using ET Telemetry Pro for about 4 months.

I’m seeing various errors like

2024-09-09T06:00:02 Error send_heartbeat.py unexpected result from https://opnsense.emergingthreats.net/api/v1/telemetry (http_code 502)

2024-09-09T00:00:02 Error rule-updater.py download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/engine/suricata/5 (http_code: 502)

2024-09-09T00:00:01 Error rule-updater.py download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 502)

These have been happening since 2024-09-04 and occur daily.

No network issues I"m aware of
I have an et_telemetry.token

Could the token be expired? Any tips as to what to check?

[ishaughnessy]

Hey @planetf1 - thanks for joining The Community!

Which log file are you seeing the errors in? I just got a new install setup today and not seeing anything similar to what you’ve shared.

I think the first thing to check is the validity of your token. Going off the docs it looks like the easiest way to check the health of your subscription is to add a widget to the dashboard.

Here are the instructions on how to get the widget added:

1. Go to the dashboard Lobby ‣ Dashboard
2. Click on “Add widget” in the top right corner, click “Telemetry status” in the list
3. Close dialog and click “Save settings” on the right top of the dashboard
4. Open Lobby ‣ Dashboard again to refresh the content

Here is a screenshot of what my status looks like. Let me know what your telemetry status is and we can go from there

[planetf1]

Thanks for the reply!

Looks fairly good, yet in System->Log Files->General (Warning) I still see entries like:

These repeat each night, hence the uncertainty about whether it’s actually updating properly…

[planetf1]

The last rule download was prior to the error, but isn’t too hold, which suggests it might be intermittent. Here’s a lot with more detail:

[Date](javascript:void(0)	[Severity](javascript:void(0)	[Process](javascript:void(0)	[Line](javascript:void(0)
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-worm.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-web_specific_apps.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-web_server.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-web_client.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-voip.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-user_agents.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-tftp.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-telnet.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-sql.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-snmp.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-smtp.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-shellcode.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-scan.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-scada.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-rpc.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-pop3.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-policy.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-phishing.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-p2p.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-netbios.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-mobile_malware.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-misc.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-malware.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-ja3.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-info.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-inappropriate.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-imap.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-icmp_info.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-icmp.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-hunting.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-games.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-ftp.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-exploit_kit.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-exploit.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-dos.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-dns.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-deleted.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-current_events.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-coinminer.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-chat.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-botcc_portgrouped.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-attack_response.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-adware_pup.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	download skipped emerging-activex.rules, same version
2024-09-10T00:00:11	Notice	rule-updater.py	version response for https://opnsense.emergingthreats.net/api/v1/ruleset/version : {“ruleset”: “opnsense-rules.tar.gz”, “version”: “10683”}
2024-09-10T00:00:07	Notice	rule-updater.py	download completed for https://opnsense.emergingthreats.net/api/v1/ruleset/engine/suricata/5
2024-09-10T00:00:01	Error	rule-updater.py	download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 502)
2024-09-09T06:00:02	Error	send_heartbeat.py	unexpected result from https://opnsense.emergingthreats.net/api/v1/telemetry (http_code 502)
2024-09-09T00:00:06	Notice	rule-updater.py	download skipped emerging-worm. Rules, same version

[ishaughnessy]

Thanks for sharing the detailed information! I initially installed OPNSense on a local VM which wasn’t running all night so I haven’t seen any errors on my end.

I installed a new instance in a cloud provider which checks for updates hourly so I’ll let that run for the next day and report back what I’m seeing in my logs.

[Mika4D]

Hello,
I have the same issue, and the message appears very frequently and at irregular intervals since September 9 at 9:00 AM.
My FW is working perfectly, my IPS as well, I have no network issues, and my token is still valid.

[ishaughnessy]

thanks for sharing @Mika4D - we’re looking at it now

[robby]

I’m getting this same error every hour for the past 3 days :
Error send_heartbeat.py unexpected result from https://opnsense.emergingthreats.net/api/v1/telemetry (http_code 500).
The ET telemetry ruleset is stuck on date 2024/10/19, due to :
rule-updater.py: download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/version (http_code: 500)
rule-updater.py: download failed for https://opnsense.emergingthreats.net/api/v1/ruleset/engine/suricata/5 (http_code: 500)
Can you check your api endpoint as my token was still valid ?

[planetf1]

Similar here. The issues have returned.

[Kornelius777]

Same here:

<13>1 2024-10-22T17:53:03+02:00 OPNsense.local send_telemetry.py 13912 - [meta sequenceId="60"] telemetry data collected 936 records in 1.18 seconds @2024-10-22 15:50:04.403484
<11>1 2024-10-22T17:53:14+02:00 OPNsense.local send_telemetry.py 13912 - [meta sequenceId="61"] unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 500)
<13>1 2024-10-22T17:54:02+02:00 OPNsense.local send_telemetry.py 75914 - [meta sequenceId="62"] telemetry data collected 923 records in 1.24 seconds @2024-10-22 15:50:04.403484
<11>1 2024-10-22T17:54:43+02:00 OPNsense.local send_telemetry.py 75914 - [meta sequenceId="63"] unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 500)
<13>1 2024-10-22T17:55:02+02:00 OPNsense.local send_telemetry.py 6082 - [meta sequenceId="64"] telemetry data collected 923 records in 0.84 seconds @2024-10-22 15:50:04.403484
<11>1 2024-10-22T17:55:25+02:00 OPNsense.local send_telemetry.py 6082 - [meta sequenceId="65"] unexpected result from https://opnsense.emergingthreats.net/api/v1/event (http_code 500)
<13>1 2024-10-22T17:56:01+02:00 OPNsense.local send_telemetry.py 56105 - [meta sequenceId="66"] telemetry data collected 925 records in 0.60 seconds @2024-10-22 15:55:04.406770

…and so on and on…

Kind regards,

[robby]

Problem looks to be fixed now : heartbeat working again and ruleset just updated to v10724

[ishaughnessy]

Thanks for letting us know, this should be resolved now! Please let me know if you start seeing issues again.