seperate collection of user reports and sending of incident reports for rdlist users into seperate functions.

README.md

@@ -118,7 +118,77 @@ This script can automatically load and block/purge abusive rooms from rdlist, ma
 
 If you are running a public server, please dm me at [@michael:perthchat.org](https://matrix.to/#/@michael:perthchat.org) and I can invite you to the 'Legion of Janitors' room.
 
-Once you have read access to the [rdlist repository](https://code.glowers.club/loj/rdlist), simply run this moderation script like so:
+Once you have read access to the [rdlist repository](https://code.glowers.club/loj/rdlist), this script can be used for multiple rdlist related functions.
+
+
+***
+## Collect User Reports on local users in rdlist rooms
+
+This script can automatically generate 'User Reports' for each one of your local users in rdlist rooms that have the 'recommended tags'.
+
+These user reports can be given to law enforcement or shared in [#janitor-dumps](https://matrix.to/#/#janitor-dumps:glowers.club) to help us locate more abusive users/rooms.
+
+```
+130
+
+rdlist repo already cloned...
+Fetching origin
+Pulling latest changes from rdlist repo...
+
+WARNING! The following local users are current members of rooms tagged in rdlist: ['@fatweeb23838:perthchat.org', '@somecreep29330:perthchat.org']
+
+Do you want to generate a user report file for each of these users? y/n? y
+
+Generating user report for fatweeb23838...
+Report generated successfully on user: "fatweeb23838"
+
+You can send this .zip file when reporting a user to law enforcement.
+.zip file location: /home/pcadmin/PerthchatVault/Perthchat_LIVE/matrix-moderation-tool/reports/fatweeb23838_2023-08-01_23-19-24.zip
+.zip file size: 0.00966 MB
+
+
+Generating user report for somecreep29330...
+Report generated successfully on user: "somecreep29330"
+
+You can send this .zip file when reporting a user to law enforcement.
+.zip file location: /home/pcadmin/PerthchatVault/Perthchat_LIVE/matrix-moderation-tool/reports/somecreep29330_2023-08-01_23-19-27.zip
+.zip file size: 0.29578 MB
+```
+
+
+***
+## Send Incident Reports for remote users in rdlist rooms
+
+This script can automatically generate 'Incident Reports' for every remote homeserver admin with users in rdlist rooms that have the 'recommended tags'.
+
+It examines the homeserver involved to find a admin contact method via [MSC1929](https://github.com/matrix-org/matrix-spec-proposals/pull/1929). If an MXID is returned it will attempt to send the Incident Report over Matrix. If an email is provided it will send the Incident Report over email. If neither is found a whois lookup is performed and the Incident Report are sent to the domain registrar via email.
+
+```
+131
+
+rdlist repo already cloned...
+Fetching origin
+Pulling latest changes from rdlist repo...
+
+WARNING! The following remote users are current members of rooms tagged in rdlist: ['@pedobear847:matrix.org']
+
+Do you want to send out incident reports for these users to every homeserver admin involved? y/n? y
+
+Sending Incident Report for users from matrix.org to abuse@matrix.org
+
+Sending Incident Report for users from perthchat.org to @michael:perthchat.org
+
+```
+
+[Preview of Incident Report.]
+
+
+## rdlist Block/Purge all rooms with recommended rdlist tags
+
+Finally this script can be used to shutdown rooms with the recommended rdlist tags.
+
+This function is much larger and will ask you if you also want to create user/incident reports before the shutdowns. (Recommended) It'll also ask you if you want to shadowban the users in these rooms to prevent them from alerting others. (Recommended) Finally it'll ask if you want to shutdown the local accounts located in these rooms.
+
 ```
 $ python3 moderation_tool.py 
 
@@ -126,7 +196,7 @@ Please select one of the following options:
 ...
 Please enter a number from the above menu, or enter 'q' or 'e' to exit.
 
-121
+132
 
 @mod_team:perthchat.org account already exists. Resetting account password.
 
@@ -145,6 +215,12 @@ Do you want to generate a user report file for each of these users? y/n? n
 
 Skipping user report generation...
 
+WARNING! The following remote users are current members of rooms tagged in rdlist: ['@PC-Admin:matrix.org']
+
+Do you want to send out incident reports for these users to every homeserver admin involved? y/n? n
+
+Skipping incident report generation...
+
 
 Number of rdlist rooms being shutdown: 346
 
@@ -202,8 +278,6 @@ Do you want to also deactivate all these accounts that were kicked from rdlist r
 ...
 ```
 
-Note that this script before shutting these rooms down will save the state events to the "./state_events" folder, please keep this data as it's important for law enforcement.
-
 
 ***
 ## One-touch Reporting
@@ -227,7 +301,6 @@ Report generated successfully on user: "michael"
 
 You can send this .zip file and password when reporting a user to law enforcement.
 
-Password: RwiFrw9zouhVO7Dy9kW7
+.zip file location: ./reports/michael_2023-07-23_02-21-56.zip.aes
-Encrypted .zip file location: ./reports/michael_2023-07-23_02-21-56.zip.aes
+.zip file size: 0.503927 MB
-Encrypted .zip file size: 0.503927 MB
 ```

moderation_tool.py

@@ -61,10 +61,13 @@ while pass_token == False:
 	print("101) Purge remote media repository up to a certain date.\t151) Lookup homeserver admin contact details.")
 	print("102) Prepare database for copying events of multiple rooms.\t152) Send a test email (to yourself).")
 	print("\t\t\t\t\t\t\t\t153) Sent a test Matrix message (to yourself).")
-	print("#### rdlist ####\t\t\t\t\t\t154) Send test incident reports (to yourself).")
+	print("#### rdlist - General ####\t\t\t\t\t154) Send test incident reports (to yourself).")
 	print("120) Block all rooms with specific rdlist tags.")
-	print("121) Block all rooms with recommended rdlist tags.")
+	print("121) Get rdlist tags for a room.")
-	print("122) Get rdlist tags for a room.")
+	print("\n#### rdlist - Recommended Tags ####\nFor rdlist rooms with recommended tags, the following actions are available:")
+	print("130) Collect User Reports on local accounts in rdlist rooms.")
+	print("131) Send Incident Reports on remote accounts in rdlist rooms.")
+	print("132) Block/Purge all rdlist rooms.")
 	print("\n#### ipinfo.io ####")
 	print("140) Analyse a users country of origin.")
 	print("141) Analyse multiple users country of origin.")
@@ -182,10 +185,14 @@ while pass_token == False:
 	elif menu_input == "120":
 		rdlist_commands.block_all_rooms_with_rdlist_tags(False,'','','')
 	elif menu_input == "121":
-		rdlist_commands.block_recommended_rdlist_tags()
-	elif menu_input == "122":
 		rdlist_tags = rdlist_commands.get_rdlist_tags('')
 		print(json.dumps(rdlist_tags, indent=4, sort_keys=True))
+	elif menu_input == "130":
+		rdlist_commands.collect_user_reports_on_rdlist_accounts()
+	elif menu_input == "131":
+		rdlist_commands.send_incident_reports_on_rdlist_accounts()
+	elif menu_input == "132":
+		rdlist_commands.block_recommended_rdlist_tags()
 	elif menu_input == "140":
 		ipinfo_commands.analyse_account_ip('')
 	elif menu_input == "141":

rdlist_commands.py

@@ -3,11 +3,15 @@ import os
 import subprocess
 import json
 import time
+import asyncio
 import user_commands
 import room_commands
 import report_commands
 import hardcoded_variables
 
+def testing_mode_warning():
+	print("\nWARNING! Testing mode is enabled, this will:\n\n- Reduce the amount of data collected in user reports.\n- Slow down rdlist blocking/purging.\n- Prevent the deactivation of accounts.\n- Send incident reports to yourself instead of other homeserver admins.\n")
+
 def sync_rdlist():
 	rdlist_dir = "./rdlist"
 	os.makedirs(rdlist_dir, exist_ok=True)
@@ -58,30 +62,7 @@ def get_rdlist_tags(preset_internal_ID):
 
 	return None
 
-def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,preset_new_room_name,preset_message):
+def get_key_rdlist_info(rdlist_tags):
-	# Git clone the rdlist repo to ./rdlist/
-	sync_rdlist()
-
-	if rdlist_use_recommended == True:
-		# Use the hardcoded recommended tags
-		blocked_tags = hardcoded_variables.rdlist_recommended_tags
-		print(f"\nUsing recommended rdlist tags. Rooms matching the following tags will be purged and/or blocked:\n{hardcoded_variables.rdlist_recommended_tags}")
-
-	elif rdlist_use_recommended == False:
-		# After the git repo has been cloned/pulled, open the file and read it into a string
-		with open(os.path.join("rdlist", "lib", "docs", "tags.md"), 'r') as file:
-			data = file.readlines()
-
-		# Print ./rdlist/lib/docs/tags.md README file for the user
-		print("\nPrinting details about the current tags in rdlist:\n")
-		for line in data:
-			print(line, end='')  # Print the contents of the file
-
-		# Take input from the user and convert it to a list
-		print("\nPlease enter a space seperated list of tags you wish to block:\n")
-		blocked_tags = input().split()
-		print('')
-
 	# Load the summaries JSON file
 	summaries_path = os.path.join("rdlist", "dist", "summaries.json")
 	with open(summaries_path, 'r') as file:
@@ -97,8 +78,8 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 	# Create a dictionary to store the tags for each room
 	room_tags = dict()
 
-	# Iterate over blocked_tags
+	# Iterate over the provided rdlist_tags
-	for tag in blocked_tags:
+	for tag in rdlist_tags:
 		# Filter the data to keep only the entries where the tag appears in the "tags" list
 		filtered_data = [item for item in data if 'report_info' in item and 'tags' in item['report_info'] and tag in item['report_info']['tags']]
 
@@ -131,15 +112,25 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 
 	all_room_ids = list(all_room_ids) # convert the set to a list
 
-	#print(f"all_local_users: {all_local_users}")
+	return all_room_ids, all_local_users, all_remote_users
-	#print(f"all_remote_users: {all_remote_users}")
+
-	#print(f"all_room_ids: {all_room_ids}")
+def collect_user_reports_on_rdlist_accounts(all_local_users=None, skip_input=False):
+	# Print warning if testing mode is enabled
+	if hardcoded_variables.testing_mode == True:
+		testing_mode_warning()
+
+	# If all_local_users is None, then we need to generate it
+	if all_local_users == None:
+		# Git clone the rdlist repo to ./rdlist/
+		sync_rdlist()
+		all_room_ids, all_local_users, all_remote_users = get_key_rdlist_info(hardcoded_variables.rdlist_recommended_tags)
 
 	# If there's at least 1 local user detected, ask the admin if they want to generate a user report for every user found in rdlist rooms
 	if len(all_local_users) > 0:
 		print(f"\nWARNING! The following local users are current members of rooms tagged in rdlist: {list(all_local_users.keys())}")
-		generate_user_report_confirmation = input("\nDo you want to generate a user report file for each of these users? y/n? ")
+		if skip_input == False:
-		if generate_user_report_confirmation.lower() in ['y', 'yes', 'Y', 'Yes']:
+			generate_user_report_confirmation = input("\nDo you want to generate a user report file for each of these users? y/n? ")
+		if generate_user_report_confirmation.lower() in ['y', 'yes', 'Y', 'Yes'] or skip_input == True:
 			for user_id in all_local_users:
 				# Generate report_dict for each user
 				report_content = report_commands.generate_rdlist_report_summary(all_local_users[user_id], user_id)
@@ -149,7 +140,65 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 	elif len(all_local_users) == 0:
 		print(f"\nNo local users were found in rdlist rooms.")
 
-	# Todo: Add Incident Report section
+def send_incident_reports_on_rdlist_accounts(all_remote_users=None, skip_input=False):
+	# Print warning if testing mode is enabled
+	if hardcoded_variables.testing_mode == True:
+		testing_mode_warning()
+
+	# If all_remote_users is None, then we need to generate it
+	if all_remote_users == None:
+		# Git clone the rdlist repo to ./rdlist/
+		sync_rdlist()
+		all_room_ids, all_local_users, all_remote_users = get_key_rdlist_info(hardcoded_variables.rdlist_recommended_tags)
+
+	# If there's at least 1 remote user detected, ask the admin if they want to generate a user report for every user found in rdlist rooms
+	if len(all_remote_users) > 0 or skip_input == False:
+		print(f"\nWARNING! The following remote users are current members of rooms tagged in rdlist: {list(all_remote_users.keys())}")
+		if skip_input == False:
+			send_incident_report_confirmation = input("\nDo you want to send out incident reports for these users to every homeserver admin involved? y/n? ")
+		if send_incident_report_confirmation.lower() in ['y', 'yes', 'Y', 'Yes'] or skip_input == True:
+			loop = asyncio.get_event_loop()
+			loop.run_until_complete(report_commands.send_incident_reports(all_remote_users))
+		elif send_incident_report_confirmation.lower() in ['n', 'no', 'N', 'No']:
+			print("\nSkipping incident report generation...\n")
+
+def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,preset_new_room_name,preset_message):
+	# Git clone the rdlist repo to ./rdlist/
+	sync_rdlist()
+
+	if rdlist_use_recommended == True:
+		# Use the hardcoded recommended tags
+		rdlist_tags = hardcoded_variables.rdlist_recommended_tags
+		print(f"\nUsing recommended rdlist tags. Rooms matching the following tags will be purged and/or blocked:\n{hardcoded_variables.rdlist_recommended_tags}")
+
+	elif rdlist_use_recommended == False:
+		# After the git repo has been cloned/pulled, open the file and read it into a string
+		with open(os.path.join("rdlist", "lib", "docs", "tags.md"), 'r') as file:
+			data = file.readlines()
+
+		# Print ./rdlist/lib/docs/tags.md README file for the user
+		print("\nPrinting details about the current tags in rdlist:\n")
+		for line in data:
+			print(line, end='')  # Print the contents of the file
+
+		# Take input from the user and convert it to a list
+		print("\nPlease enter a space seperated list of tags you wish to block:\n")
+		rdlist_tags = input().split()
+		print('')
+
+	all_room_ids, all_local_users, all_remote_users = get_key_rdlist_info(rdlist_tags)
+
+	#print(f"\nDEBUG all_local_users: {all_local_users}")
+	#print(f"DEBUG all_remote_users: {all_remote_users}")
+	#print(f"DEBUG all_room_ids: {all_room_ids}")
+
+	# If there's at least 1 local user detected, ask the admin if they want to generate a user report for every user found in rdlist rooms
+
+	collect_user_reports_on_rdlist_accounts(all_local_users, False)
+
+	# If there's at least 1 remote user detected, ask the admin if they want to generate a incident report for every user homeserver involved in rdlist rooms
+
+	send_incident_reports_on_rdlist_accounts(all_remote_users, False)
 
 	# Ask the user if they wish to block and purge all these rooms, then collect shutdown parameters
 	if preset_user_ID == '':
@@ -220,7 +269,7 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 def block_recommended_rdlist_tags():
 	# Print warning if testing mode is enabled
 	if hardcoded_variables.testing_mode == True:
-		print("\nWARNING! Testing mode is enabled, this will reduce the amount of data generated in reports and greatly slow down rdlist blocking!\n")
+		testing_mode_warning()
 
 	# Check if user account already exists
 	account_query = user_commands.query_account(hardcoded_variables.rdlist_bot_username)

report_commands.py

@@ -221,7 +221,7 @@ def generate_user_report(preset_username, report_details):
 	zip_file_size = os.path.getsize(zip_file_name) / 1000000
 
 	# Print the password and the encrypted .zip file name
-	print("Report generated successfully on user: \"" + username + "\"\n\nYou can send this .zip file when reporting a user to law enforcement.")
+	print("Report generated successfully on user: \"" + username + "\"\n\nYou can send this .zip file when reporting a user to law enforcement.\n")
 	print(".zip file location: " + zip_file_name)
 	print(".zip file size: " + str(zip_file_size) + " MB\n")
 
@@ -421,7 +421,7 @@ https://{hardcoded_variables.base_url}
 
 	return message_content
 
-async def send_incident_report(incidents_dict):
+async def send_incident_reports(incidents_dict):
 	success = True
 	homeserver_dict = {}
 
@@ -451,7 +451,7 @@ async def send_incident_report(incidents_dict):
 				message_content = prepare_message_content(user_dict, baseurl)
 
 				try:
-					print(f"Sending Incident Report message to {admin['matrix_id']}")
+					print(f"\nSending Incident Report for users from {baseurl} to {admin['matrix_id']}")
 					await bot_commands.send_message(admin["matrix_id"], message_content)
 				except Exception as e:
 					print(f"Failed to send message to {admin['matrix_id']}: {str(e)}")
@@ -464,7 +464,7 @@ async def send_incident_report(incidents_dict):
 					email_content = prepare_email_content(user_dict, from_whois, baseurl)
 
 					email_attachments = []
-					print(f"Sending Incident Report email to {email_address}")
+					print(f"Sending Incident Report for users from {baseurl} to {admin['email_address']}")
 					if not send_email(email_address, email_subject, email_content, email_attachments):
 						print(f"Failed to send email to {email_address}")
 						success = False
@@ -498,7 +498,7 @@ def test_send_incident_reports():
 	try:
 		if hardcoded_variables.testing_mode == True:
 			print("\nNOTE: Testing mode is enabled, sending Incident Reports to you! :)\n")
-		if asyncio.run(send_incident_report(incidents_dict)):
+		if asyncio.run(send_incident_reports(incidents_dict)):
 			print("\nIncident reports successfully sent.")
 		else:
 			print("\nFailed to send the incident reports.")