EDR: Understanding and Deploying Yara Components

[#1]

Environment

• EDR Server: 7.7.x
• Yara Connector: 2.2.0
• Yara Manager: 2.2.0

Objective

To install and configure Yara in both EDR standalone and cluster environments.

Resolution

Yara, a rules engine from VirusTotal, alerts on binaries executing in the environment.  The yml style rules are created with text or binary patterns.  The Yara Connector processes, analyzes and stores (Solr, Yara DB) the analysis for EDR Console to obtain.  The Yara Manager, optional, provides access to the Yara-connector via the EDR Console.
The instructions to install Yara for standalone and cluster EDR environments are the same.   Yara only needs to run on the EDR Primary server utilizing EDR's built-in communication service cb-redis.
 

A.  Install Yara Components (derived from developer.carbonblack.com)

  1. Create the Carbon Black open source repo

cd /etc/yum.repos.d
curl -O https://opensource.carbonblack.com/release/x86_64/CbOpenSource.repo

  2. Install the cb-yara-connector 

yum install python-cb-yara-connector

  3. Install the cb-yara-manager (optional)

yum install python-cb-yara-manager

 

B. Configure and enable Yara Connector (all environments)

1. Obtain an admin user token.

psql cb -p 5002 -c “select username, auth_token from cb_user where global_admin=true;”

2. Create the yara-configuration file

cd /etc/cb/integrations/cb-yara-connector
mv connector.conf.example yaraconnector.conf
vi /etc/cb/integrations/cb-yara-connector/yaraconnector.conf

3.  Modify yaraconnector.conf

mode=standalone   (for EDR cluster and EDR standalone environments distribution comms now use EDR redis)
cb_server_url=https://127.0.0.1
cb_server_token=< API TOKEN from step B1 GOES HERE >
broker_url=redis://127.0.0.1:6379 
num_days_binaries=365 (default is best; initial scan span)
database-scanning_interval=900 (default is best; cannot be less than 360)

4. Use the default rule (already included) to get cb-yara-connector working, then add rules later.
5. Start Yara-connector.

systemctl stop cb-yara-connector
systemctl status cb-yara-connector
systemctl start cb-yara-connector

6. Enable the Threat Intelligence Yara Feed tile.

EDR Console > Threat Intelligence > Yara tile
Click "Enabled".
Notifications > "Create Alert"

 

C.  Adding Yara Rules

• Rules are contained within *.yar files.
• The .yar files are placed in /etc/cb/integrations/cb-yara-connector/yara_rules directory.
• Yara Manager can add rules by uploading the .yar files into Yara Manager browser.  EDR Yara Manager > Yara Rules tab > Choose File > Upload Rule.
• Yara-connector monitors the directory for new rules.  No need to restart cb-yara-connector to ingest new rules.
• Tips: 
• Get yara-connector working with the default rule first. (/etc/cb/integrations/cb-yara-connector/yara_rules/sample.yar)
• Only add a few rules at a time. 
• Cut-n-paste can add extra characters and cause troubleshooting issues.  If cut-n-paste, copy from a text editor.
• Validate they are working – Currently by reviewing the logs. Validate tool should be fixed in next release
• Review yara.readthedocs.io (see link below) for additional information.

D.  Configure Yara Manager - Optional (derived from User Guide)

1. Create the configuration file. 

cd /etc/cb/integrations/cb-yara-manager 
cp config.py.example config.py

2. Create the authentication file. 

vi /etc/cb/integrations/cb-yara-manager/auth.conf
[auth]
api_token=< create a unique adequately_long_and_complex_password >

     (where adequately_long_and_complex_password_or_token is any passphrase.)
3. Add to /etc/cb/cb.conf 

YaraManagerEnabled=true
YaraManagerToken=< insert the unique adequately_long_and_complex_password >

4. To invoke the new cb.conf changes run

/usr/share/cb/cbservice cb-coreservices restart

5. Start the service. 

systemctl start cb-yara-manager

6. Confirm that it is running. 

ps -ef | grep -i manager (there should be 2 instances running)

7. View Yara Manager in the browser after authenticating to the EDR server.

https://<EDR server IP>/connector/yara

 

E.  Yara Components:

This is a article attached image

1. Yara rules .yar files are read, compressed and loaded into memory.  New .yar files are included without restarting cb-yara-connector service. 
Component:  /etc/cb/integrations/cb-yara-connector/yara_rules/.YARA_RULES_xxxx

2. Read the Postgres database binary_status table every 15 minutes to identify new binary entries. 
Component:  Postgres DB table binary_status.

3. New binaries are stored on the EDR servers.  Yara Connector pulls a temporary copy of the new binary from the EDR servers.  
Component:  /var/cb/data/modulestore/

4. The binaries are analyzed against the Yara rules and recorded in Yara Connector’s Sqlite database.   
Component:  /var/cb/data/cb-yara-connector/feed_db/binary.db.   The binary.db-wal and binary.db-shm are helper files for binary.db.

5. If a binary matches a Yara rule, the information is stored in the Yara Connector’s JSON file.
Component: /var/cb/data/cb-yara-connector/feed.json

6. Every hour, Yara Connector performs an incremental synchronization between feed.json and Solr data.  A complete synchronization occurs daily.  Incremental and complete synchronizations can occur on demand from the Yara Feed tile.
Component: /var/cb/data/solr/   

7. The EDR Console uses API calls against the Solr data to extract the Yara alerts in Notifications, Process Searches and Yara Threat Reports.
 

F.  Troubleshooting:

a) Yara Connector logs 

/var/log/cb/integrations/cb-yara-connector/yaraconnector.log 
journalctl -fexu cb-yara-connector

b) Yara Manager logs 

/var/log/cb/integrations/cb-yara-manager/cb-yara-manager.log

c) Monitor binary.db. Does it grow in size as new binaries arrive to EDR?  First command presents the total binaries scanned (should increase).  The second command identifies any missing binaries, perhaps aged out or uploaded to Alliance (should be 0).

sqlite3 /var/cb/data/cb-yara-connector/feed_db/binary.db "select count(*) from binarydetonationresult;"
sqlite3 /var/cb/data/cb-yara-connector/feed_db/binary.db "select count(*) from binarydetonationresult where ‘binary_not_available’ and ‘score’>0;”

d) Run a Process Search in EDR console and expand the time range to "All available".

alliance_score_yara:*

e) Check for new binaries in Postgres.  Modify the timestamp as needed.

psql -p 5002 cb -c “select md5hash,node_id from storefiles where present_locally=true and timestamp>=’2023-05-19 00:00:00’ order by timestamp desc;”

f) Feed tile does not appear on Threat Intelligence page.

• Redis could be overwhelmed.  Remove the client output buffer limit in /etc/cb/redis.conf.template.  Change to 

client-output-buffer-limit pubsub 0 0 0

• OR using EDR Console > Threat Intelligence > Add New Feed

URL: file://var/cb/data/cb-yara-connector/feed.json

g) By design, due to performance reasons, the connector does not retroactively go back to rescan binaries for matches when a rule is uploaded. The db would need to be cleared and trigger a new scan from the beginning.

h) Yara-connector does not automatically start after reboot.
     Check symlink from /etc/systemd/system/multi-user.target.wants/cb-yara-connector.service to etc/system/system/cb-yara-connector.sevice

i) Reset and Restart.  

systemctl stop cb-yara-manager
systemctl stop cb-yara-connector
cp /etc/cb/integrations/cb-yara-connector/yara_rules/.YARA_RULES* /etc/cb/integrations/cb-yara-connector/yara_rules/.YARA_RULES-bkup
rm /etc/cb/integrations/cb-yara-connector/yara_rules/.YARA_RULES*
rm /var/cb/data/cb-yara-connector/feed.json
rm /var/cb/data/cb-yara-connector/feed_db/binary*
systemctl start cb-yara-connector
systemctl start cb-yara-manager

j) Debug mode.   Add to /etc/cb/integrations/cb-yara-connector/yaraconnector.conf then restart cb-yara-connector.

log_level=DEBUG

k) Verify the Threat Report is populated from the Yara Feed tile or run the following command on the EDR Primary server.

curl 'http://localhost:8080/solr/cbfeeds/select?q=id:"binary_<md5 hash lower case here>"'

l) The binary scan does not start and feed.json remains empty.

* Confirm the mode is set to 'standalone' (even in an EDR cluster on primary server)
* Confirm the token in the yara configuration has adequate permissions to access the binaries and the account is active.

j) The binary.db and feed.json are populated but alerts do not appear in the console.  The connection with Solr may be broken.  

* Check the Solr logs and yara-connector logs for errors.
* Consider resetting the yara-connector noted in step h.

Additional Notes

 

• Install cb-yara-connector based on YARA Connector and Manager User Guide for EDR - Carbon Black Developer Network
• Install cb-yara-manager based on Yara Manager
• Yara rules https://yara.readthedocs.io/en/latest/writingrules.html

Related Content

 

• https://community.carbonblack.com/t5/Knowledge-Base/EDR-Yara-Components-Explained-Installed-Configur...
• EDR: Adding and Validating Yara Rules
• EDR: Yara Feed Not Visible on Threat Intelligence Feed Page
• Hosted EDR: How to Enable the Yara Connector
• EDR: No threat report for yara feed
• EDR: Yara is Not Alerting On Expected Binary
• EDR Yara-Connector: New Yara Rule Does Not Run on Binaries
• CB Response: No threat reports in the Yara connector in the web console
• EDR: How to Setup 2.1.2 CB-Yara-Manager
• EDR CB-Yara-Connector: Enable Debug Logging
• CB Response: How to purge the Yara 1.x database