A simple Python package to Query Json Data.
- Supports pure json files
- Supports multiple json objects in a file, delimited by newlines (/n)
- Supports gzipped files
- Supports customizabile filters
- Supports pure datetime range filters
The filters could be extended easily, adopting Python3 stdlib operator.
See pyjq.PyJQ.filter to extend ops mapping.
pip install pyjq-ng
See example/alerts.json.
pyjq works on lines by lines (splitted by \n).
It have been used for Wazuh alert json files and Django dumps.
pyjq -j examples/django_dump.json -limit 2 -filter 'fields__original_url == https://google.com'
pyjq -j examples/django_dump.json -limit 2 -filter 'model == urlshortener.urlshortener'
‘agent__name’ it’s an example of the namespace used by pyjq to access to nested childs. It other word it means json['agent']['name'].
It haven’t limits on number of nested elements.
Apply some custom filters with AND and OR operators on Wazuh Alert file
pyjq -j ../Scaricati/alerts.json -filter 'agent__ip == 172.16.16.102 and agent__name == telegram-gw or agent__ip == 172.16.16.108'
Contains operator
pyjq -j ../Scaricati/alerts.json -filter 'rule__description in iptables and agent__name == dev-bastion'
Convert a specified filed to a pure datetime object and filter in a specified range
pyjq -j ../Scaricati/alerts.json -start_datetime 2020-04-06T10:22:00 -end_datetime 2020-04-06T13:22:00 -datetime_field timestamp
Realtime reading, it will only takes the latter entries, delimited by newline \n
pyjq -j /var/ossec/logs/alerts/alerts.json -datetime_field timestamp -realtime
Use a gzipped json file directly
pyjq -j ../Scaricati/alerts.json.gzip
Limit results to 2
pyjq -j ../Scaricati/alerts.json -limit 2
Realtime monitoring of a specific entity
pyjq -j /var/ossec/logs/alerts/alerts.json -realtime -filter 'agent__name == tinyurl and rule__level == 3'
Custom callback, usefull for bot integration and other pub/sub APIs
python3 pyjq -j examples/alerts.json -realtime -filter 'agent__name == tinyurl and rule__description in ssh' -callback 'examples.callback.things'
Reading from stdin
cat examples/alerts.json | python3 ./pyjq -filter 'rule__level > 3'
# continous processing
tail -f /tmp/alerts.json | python3 ./pyjq -filter 'location != osquery'
Giuseppe De Marco giuseppe.demarco@unical.it
Wazuh SIEM group @GarrLab
https://github.com/peppelinux/pyjq
Leave a Reply