add functions to examine if room is alrady blocked and to block/unblock rooms. dramatic speedup for applying rdlist by skipping already blocked rooms and just blocking unknown rooms instead of performing a full shutdown. add report_details section to user reporting to collect information from the admin about why the report was generated. add rdlist tag descriptions to allow for automated report_details generation

README.md

@@ -82,18 +82,10 @@ With the popular [matrix-docker-ansible-deploy](https://github.com/spantaleev/ma
 
 To do:
 1) Add the following functions:
-- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#account-data - DONE
-- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#list-all-pushers - DONE
-- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#override-ratelimiting-for-users - DONE
-- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#check-username-availability - DONE
 - https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#find-a-user-based-on-their-id-in-an-auth-provider
 - https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#find-a-user-based-on-their-third-party-id-threepid-or-3pid
 - https://github.com/matrix-org/synapse/blob/master/docs/admin_api/delete_group.md
-2) Make the menu prettier! - DONE
+2) Add fully automated (should just return a web link and decryption password) reporting functions for users:
-3) Modularise the functions into multiple files - DONE
-4) Use URI module for all API calls instead of curl - DONE
-5) Add more automated rdlist function with sane defaults - DONE
-6) Add fully automated (should just return a web link and decryption password) reporting functions for users:
 - Description of why the report was made (what happened)
 - User's ID - DONE
 - Whois Data - DONE
@@ -106,14 +98,14 @@ To do:
 - Timestamp for when illegal material was accessed
 - Description of report format and contents (to guide the reader)
 - Summary of key information
-7) Have recommended rdlist function return a list of offending accounts and the tags they accessed
+3) Have recommended rdlist function return a list of offending accounts and the tags they accessed
-8) Only email reportID in incident report?
+4) Only email reportID in incident report?
-9) Add a room report function to create a properly formatted report for rdlist
+5) Add a room report function to create a properly formatted report for rdlist
-10) Skip already shutdown rooms for speeding up rdlist blocking
+6) Skip already shutdown rooms for speeding up rdlist blocking
-11) Add function for probing the support email of another server automatically
+7) Add function for probing the support email of another server automatically
-12) Automated incident report email to other server owners who has users in rdlist rooms for more scalable coordination
+8) Automated incident report email to other server owners who has users in rdlist rooms for more scalable coordination
-13) Automated public room joining and reminder if reporting email is not available?
+9) Automated public room joining and reminder if reporting email is not available?
-14) Refine ipinfo module to also return region/state of IP
+10) Refine ipinfo module to also return region/state of IP
 
 
 ***
@@ -146,32 +138,66 @@ rdlist repo is up-to-date, no need to pull changes.
 Using recommended rdlist tags. Rooms matching the following tags will be purged and/or blocked:
 ['hub_room_links', 'hub_room_trade', 'preban', 'degen_misc', 'beastiality', 'degen_porn', 'gore', 'snuff', 'degen_larp', 'hub_room_sussy', 'bot_spam', 'cfm', 'jailbait', 'bot_porn', 'toddlercon', 'loli', 'csam', 'tfm', 'degen_meet', 'stylized_3d_loli', '3d_loli']
 
-WARNING! The following local users are current members of rooms tagged in rdlist: ['***REDACTED***:perthchat.org']
+WARNING! The following local users are current members of rooms tagged in rdlist: ['@***REDACTED***:perthchat.org']
 
 Do you want to generate a user report file for each of these users? y/n? n
 
 Skipping user report generation...
 
 
-Number of rdlist rooms being shutdown: 337
+Number of rdlist rooms being shutdown: 346
 
-Are you sure you want to shutdown these rooms? y/n? y
+Are you sure you want to block/shutdown these rooms? y/n? y
 
 
-Shutting down room: !***REDACTED***:matrix.org
+Skipping already blocked room: !***REDACTED***:matrix.org
-!***REDACTED***:matrix.org has been successfully shutdown!
 
 
-Shutting down room: !***REDACTED***:matrix.org
+Skipping already blocked room: !***REDACTED***:matrix.org
-!***REDACTED***:matrix.org has been successfully shutdown!
 
 
-Shutting down room: !***REDACTED***:anontier.nl
+Skipping already blocked room: !***REDACTED***:matrix.org
-!***REDACTED***:anontier.nl has been successfully shutdown!
 
 
-Shutting down room: !***REDACTED***:anontier.nl
+Blocking unknown room: !***REDACTED***:matrix.org
-!***REDACTED***:anontier.nl has been successfully shutdown!
+Successfully blocked room !***REDACTED***:matrix.org
+
+
+Blocking unknown room: !***REDACTED***:matrix.org
+Successfully blocked room !***REDACTED***:matrix.org
+
+
+Skipping already blocked room: !***REDACTED***:matrix.org
+
+
+Shutting down known room: !***REDACTED***:sibnsk.net
+Sleeping for 2 seconds...
+Sleeping for 4 seconds...
+Sleeping for 8 seconds...
+!***REDACTED***:sibnsk.net has been successfully shutdown!
+List of kicked users:
+@***REDACTED***:perthchat.org
+
+
+Skipping already blocked room: !***REDACTED***:anontier.nl
+
+
+Room shutdowns completed!
+
+User login details for your moderator account:
+
+Username: mod_team
+Password: ***REDACTED***
+
+Print rdlist statistics:
+
+Number of rooms blocked: 4
+Number of rooms purged: 2
+Number of local users located in rdlist rooms and kicked: 1
+
+The following users were current members of rooms tagged in rdlist: ['@***REDACTED***:perthchat.org']
+
+Do you want to also deactivate all these accounts that were kicked from rdlist rooms? y/n?
 ...
 ```
 

moderation_tool.py

@@ -51,9 +51,9 @@ while pass_token == False:
 	print("14) Collect account data.\t\t\t33) Delete multiple rooms.")
 	print("15) List account pushers.\t\t\t34) Purge the event history of a room to a specific timestamp.")
 	print("16) Get rate limit of a user account.\t\t35) Purge the event history of multiple rooms to a specific timestamp.")
-	print("17) Set rate limit of a user account.")
+	print("17) Set rate limit of a user account.\t\t36) Get blocked status for room.")
-	print("18) Delete rate limit of a user account.")
+	print("18) Delete rate limit of a user account.\t37) Block a room.")
-	print("19) Check if user account exists.")
+	print("19) Check if user account exists.\t\t38) Unblock a room.")
 	print("\n#### Server Commands ####\t\t\t\t\t#### Report Generation ####")
 	print("40) Delete and block a specific media.\t\t\t\t70) Generate user report.")
 	print("41) Purge remote media repository up to a certain date.\t\t71) Decrypt user report .zip file.")
@@ -116,7 +116,7 @@ while pass_token == False:
 		elif user_account_exists == False:
 			print("\nUser account does not exist.\n")
 	elif menu_input == "20":
-		room_details_dict = room_commands.list_room_details('')
+		room_details_dict = room_commands.get_room_details('')
 		print(json.dumps(room_details_dict, indent=4, sort_keys=True))
 	elif menu_input == "21":
 		room_members_dict = room_commands.get_room_members('',False)
@@ -149,6 +149,17 @@ while pass_token == False:
 		room_commands.purge_room_to_timestamp('','')
 	elif menu_input == "35":
 		room_commands.purge_multiple_rooms_to_timestamp()
+	elif menu_input == "36":
+		blocked_status = room_commands.get_block_status('')
+		if blocked_status == True:
+			print("\nRoom is blocked.\n")
+		elif blocked_status == False:
+			print("\nRoom is not blocked.\n")
+		print(json.dumps(blocked_status, indent=4, sort_keys=True))
+	elif menu_input == "37":
+		room_commands.set_block_status('', True)
+	elif menu_input == "38":
+		room_commands.set_block_status('', False)
 	elif menu_input == "40":
 		server_commands.delete_block_media()
 	elif menu_input == "41":
@@ -167,7 +178,7 @@ while pass_token == False:
 	elif menu_input == "61":
 		ipinfo_commands.analyse_multiple_account_ips()
 	elif menu_input == "70":
-		report_commands.generate_user_report('')
+		report_commands.generate_user_report('','')
 	elif menu_input == "71":
 		report_commands.decrypt_zip_file()
 	elif menu_input == "72":

rdlist_commands.py

@@ -10,7 +10,32 @@ import room_commands
 import report_commands
 import hardcoded_variables
 
-#rdlist_bot_username = hardcoded_variables.rdlist_bot_username
+rdlist_tag_descriptions = {
+    "csam": "Child Sexual Abuse Material",
+    "cfm": "An abundance of content which would directly appeal to those seeking csam.",
+    "jailbait": "Photos which contain underage individuals in questionable or suggestive situations.",
+    "tfm": "An abundance of content which would directly appeal to those seeking jailbait.",
+    "beastiality": "Self explanatory.",
+    "3d_loli": "Pornography which depicts photorealistic underage characters.",
+    "stylized_3d_loli": "Pornography which depicts underage characters that are not depicted in a realistic style.",
+    "gore": "Self explanatory.",
+    "snuff": "Self explanatory.",
+    "degen_misc": "Other types of coomers rooms.",
+    "degen_larp": "Coomer larp rooms.",
+    "degen_meet": "Coomer socializing rooms.",
+    "degen_porn": "Rooms dedicated to pornography, excluding types which have dedicated tags.",
+    "bot_porn": "Rooms which contain bots that spam pornographic content.",
+    "bot_spam": "Rooms which contain bots that spam content. Primarily for malvertising and cryptospam",
+    "preban": "Rooms which may not contain tagged content, however have clear intent. i.e: Rooms with names like 'CP Room', 'Child Porn', etc",
+    "hub_room_trade": "Rooms which exist solely to trade illegal or questionable content. i.e: csam, jailbait",
+    "hub_room_sussy": "A room which is sussy. This tag does not have a solid definition, see existing tagged rooms",
+    "abandoned": "Similar to 'anarchy', primarily for rooms which have automated spam bots.",
+    "anarchy": "Unmoderated rooms.",
+    "hub_room_underage": "Rooms which contain a disproportionate amount of underage users.",
+    "hub_room_links": "Rooms which exist to share links to other rooms.",
+    "toddlercon": "Lolicon but younger.",
+    "loli": "Rooms which exist to host lolicon.",
+}
 
 def sync_rdlist():
 	rdlist_dir = "./rdlist"
@@ -148,7 +173,7 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 	# Deduplicate the list of all room_ids
 	all_room_ids = list(set(all_room_ids))
 
-	# Examine these room_ids for local users
+	# Examine these room_ids for local and remote users
 	all_local_users = []
 	all_remote_users = []
 	for room_id in all_room_ids:
@@ -203,27 +228,42 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 	elif preset_message != '':
 		message = preset_message
 
+	#print(f"all_room_ids: {all_room_ids}")
+
 	# Ask the user if they wish to block and purge all these rooms
-	shutdown_confirmation = input("\nNumber of rdlist rooms being shutdown: " + str(len(all_room_ids)) + "\n\nAre you sure you want to shutdown these rooms? y/n? ")
+	shutdown_confirmation = input("\nNumber of rdlist rooms being shutdown: " + str(len(all_room_ids)) + "\n\nAre you sure you want to block/shutdown these rooms? y/n? ")
 
 	total_list_kicked_users = []
 	num_rooms_blocked = 0
+	num_rooms_purged = 0
 
-	#print(f"all_room_ids: {all_room_ids}")
 	if shutdown_confirmation.lower() in ['y', 'yes', 'Y', 'Yes']:
 		for room_id in all_room_ids:
-			print(f"\n\nShutting down room: {room_id}")
+			blocked_status = room_commands.get_block_status(room_id)
-			room_state_dict = room_commands.export_room_state(room_id, "", False)
+			#print(f"\nroom_details_dict: {room_details_dict}")
-			#print(f"\nroom_state_dict: {room_state_dict}")
+			#print(f"\nblock_status: {blocked_status}")
-			if "Room not found" in room_state_dict.get('error', ''):
+			# If room is already blocked, skip it
-				list_kicked_users = room_commands.shutdown_room(room_id, user_ID, new_room_name, message, False, True)
+			if blocked_status == False:
-			else:
+				# Examine if unblocked room is known, if not block it
-				list_kicked_users = room_commands.shutdown_room(room_id, user_ID, new_room_name, message, True, True)
+				room_details_dict = room_commands.get_room_details(room_id)
-			num_rooms_blocked += 1
+				if "Room not found" in room_details_dict.get('error', ''):
-			total_list_kicked_users.extend(list_kicked_users)
+					print(f"\n\nBlocking unknown room: {room_id}")
-			time.sleep(5)
+					room_commands.set_block_status(room_id, True)
+					num_rooms_blocked += 1
+				# If unblocked room is known, perform a shutdown of the room
+				else:
+					print(f"\n\nShutting down known room: {room_id}")
+					list_kicked_users = room_commands.shutdown_room(room_id, user_ID, new_room_name, message, True, True)
+					num_rooms_purged += 1
+					total_list_kicked_users.extend(list_kicked_users)
+				if hardcoded_variables.testing_mode == True:
+					time.sleep(5)
+			elif blocked_status == True:
+				print(f"\n\nSkipping already blocked room: {room_id}")
+				if hardcoded_variables.testing_mode == True:
+					time.sleep(5)
 	elif shutdown_confirmation.lower() in ['n', 'no', 'N', 'No']:
-		print("\nSkipping these files...\n")
+		print("\nSkipping blocking/shutdown of rooms...\n")
 		return
 	else:
 		print("\nInvalid input, skipping these files...\n")
@@ -236,22 +276,26 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
 	print(f"\n\nList of all kicked users: {total_list_kicked_users}\n")
 	
 	# Return the list of all kicked users
-	return num_rooms_blocked, total_list_kicked_users
+	return num_rooms_blocked, num_rooms_purged, total_list_kicked_users
 
 def block_recommended_rdlist_tags():
+	# Print warning if testing mode is enabled
+	if hardcoded_variables.testing_mode == True:
+		print("\nWARNING! Testing mode is enabled, this will reduce the amount of data generated in reports and greatly slow down rdlist blocking!\n")
+
 	# Check if user account already exists
 	account_query = user_commands.query_account(hardcoded_variables.rdlist_bot_username)
 
 	# Generate random password
-	preset_password = ''.join(random.choice(string.ascii_letters + string.digits) for i in range(20))
+	rdlist_bot_password = ''.join(random.choice(string.ascii_letters + string.digits) for i in range(20))
 
 	# If user is not found, create it
 	if 'User not found' in account_query:
 		# Create user account
-		user_commands.create_account(hardcoded_variables.rdlist_bot_username, preset_password)
+		user_commands.create_account(hardcoded_variables.rdlist_bot_username, rdlist_bot_password)
 	else:
 		print(f"\n@{hardcoded_variables.rdlist_bot_username}:{hardcoded_variables.base_url} account already exists. Resetting account password.")
-		user_commands.reset_password(hardcoded_variables.rdlist_bot_username, preset_password)
+		user_commands.reset_password(hardcoded_variables.rdlist_bot_username, rdlist_bot_password)
 
 	# Promote bot user to server admin
 	print(f"\nEnsuring @{hardcoded_variables.rdlist_bot_username}:{hardcoded_variables.base_url} account is a server admin.")
@@ -262,16 +306,17 @@ def block_recommended_rdlist_tags():
 	preset_message = 'THIS ROOM VIOLATES SERVER POLICIES'
 
 	# Block all rooms with recommended tag set
-	num_rooms_blocked, total_list_kicked_users = block_all_rooms_with_rdlist_tags(True, hardcoded_variables.rdlist_bot_username, preset_new_room_name, preset_message)
+	num_rooms_blocked, num_rooms_purged, total_list_kicked_users = block_all_rooms_with_rdlist_tags(True, hardcoded_variables.rdlist_bot_username, preset_new_room_name, preset_message)
 
 	# Print user login details
 	print("\n\nRoom shutdowns completed!\n\nUser login details for your moderator account:\n")
 	print("Username: " + hardcoded_variables.rdlist_bot_username)
-	print("Password: " + preset_password)
+	print("Password: " + rdlist_bot_password)
 
 	# Print statistics for the admin
 	print(f"\nPrint rdlist statistics:")
-	print(f"\nNumber of rooms blocked/purged: {num_rooms_blocked}")
+	print(f"\nNumber of rooms blocked: {num_rooms_blocked}")
+	print(f"Number of rooms purged: {num_rooms_purged}")
 	print(f"Number of local users located in rdlist rooms and kicked: {len(total_list_kicked_users)}")
 	print(f"\nThe following users were current members of rooms tagged in rdlist: {total_list_kicked_users}")
 

report_commands.py

@@ -62,7 +62,7 @@ def encrypt_user_folder(user_report_folder, username):
 	# You can return the password if you need to use it later, or you can directly print it here
 	return strong_password, zip_file_name + ".aes"
 
-def generate_user_report(preset_username):
+def generate_user_report(preset_username, report_details):
 	if len(preset_username) == 0:
 		username = input("\nPlease enter the username to automatically generate a report: ")
 		username = user_commands.parse_username(username)
@@ -87,6 +87,13 @@ def generate_user_report(preset_username):
 	if os.path.exists(user_report_folder) == False:
 		os.mkdir(user_report_folder)
 
+	# Collect and write the report details to ./report/username/report_details.txt
+	if report_details == '':
+		report_details = input("\nPlease enter the details for this report. Include as much detail as possible, including:\n A description of what happened.\n Timestamps of events. \n Whether this user was a repeat offender, if so include details about previous incidents. \n Other user or rooms involved. \n Other evidence you've collected against this user. \n Whether the offending users were deactivated. \n Whether the offending rooms were shut down.")
+	report_details_file = open(user_report_folder + "report_details.txt", "w")
+	report_details_file.write(report_details)
+	report_details_file.close()
+
 	# Get user account data and write to ./report/username/account_data.json
 	account_data = user_commands.collect_account_data(username)
 	account_data_file = open(user_report_folder + "account_data.json", "w")
@@ -155,7 +162,7 @@ def generate_user_report(preset_username):
 	for room in room_list:
 		count += 1
 		room = room.split(" ")[0]
-		room_details = room_commands.list_room_details(room)
+		room_details = room_commands.get_room_details(room)
 
 		# Check the room conditions to select the proper output folders
 		if room_details['joined_members'] == 2 and room_details['public'] == False:
@@ -215,7 +222,7 @@ def lookup_homeserver_admin_email(preset_baseurl):
 		response = None
 
 	# If the request was successful, the status code will be 200
-	if response and response.status_code == 200:
+	if response.status_code == 200 and "email_address" in response.text:
 		# Parse the response as JSON
 		data = json.loads(response.text)
 

room_commands.py

@@ -13,7 +13,7 @@ def parse_username(username):
 	username = username.replace(tail_end,'')
 	return username
 
-def list_room_details(preset_internal_ID):
+def get_room_details(preset_internal_ID):
 	if preset_internal_ID == '':
 		internal_ID = input("\nEnter the internal id of the room you wish to query (Example: !OLkDvaYjpNrvmwnwdj:matrix.org): ")
 	elif preset_internal_ID != '':
@@ -345,6 +345,7 @@ def shutdown_room(preset_internal_ID,preset_user_ID,preset_new_room_name,preset_
 	status = "null"
 	count = 0
 	sleep_time = 1
+	list_kicked_users = []
 
 	while status != "complete" and count < 8:
 		time.sleep(sleep_time)
@@ -362,7 +363,6 @@ def shutdown_room(preset_internal_ID,preset_user_ID,preset_new_room_name,preset_
 
 	if status == "complete":
 		print(f"{internal_ID} has been successfully shutdown!")
-		list_kicked_users = []
 		if str(output_json["results"][0]["shutdown_room"]["kicked_users"]) != '[]':
 			print("List of kicked users:")
 			for entry in output_json["results"][0]["shutdown_room"]["kicked_users"]:
@@ -370,7 +370,6 @@ def shutdown_room(preset_internal_ID,preset_user_ID,preset_new_room_name,preset_
 				print(entry)
 	else:
 		print(f"Failed to shutdown {internal_ID}!")
-		list_kicked_users = []
 
 	return list_kicked_users
 
@@ -582,3 +581,55 @@ def purge_multiple_rooms_to_timestamp():
 
 # Example:
 # See purge_room_to_timestamp()
+
+def get_block_status(preset_internal_ID):
+	if preset_internal_ID == '':
+		internal_ID = input("\nEnter the internal id of a room to examine if it's blocked (Example: !OLkDvaYjpNrvmwnwdj:matrix.org): ")
+	else:
+		internal_ID = preset_internal_ID
+
+	url = f"https://{hardcoded_variables.homeserver_url}/_synapse/admin/v1/rooms/{internal_ID}/block"
+	headers = {"Authorization": f"Bearer {hardcoded_variables.access_token}"}
+
+	response = requests.get(url, headers=headers, verify=True)
+
+	# Ensure the request was successful
+	if response.status_code == 200:
+		block_status = json.loads(response.text)['block']
+	else:
+		print(f"Error: Unable to fetch block status for room {internal_ID}")
+		block_status = None
+
+	return block_status
+
+# Example:
+# $ curl -X GET -H 'Authorization: Bearer ACCESS_TOKEN' 'https://matrix.perthchat.org/_synapse/admin/v1/rooms/!IdieserRBwPdaCGYuKk:matrix.org/block'
+# {"block":false}
+
+def set_block_status(preset_internal_ID, block):
+	if preset_internal_ID == '':
+		internal_ID = input("\nEnter the internal id of a room to block/unblock (Example: !OLkDvaYjpNrvmwnwdj:matrix.org): ")
+	else:
+		internal_ID = preset_internal_ID
+
+	url = f"https://{hardcoded_variables.homeserver_url}/_synapse/admin/v1/rooms/{internal_ID}/block"
+	headers = {"Authorization": f"Bearer {hardcoded_variables.access_token}", "Content-Type": "application/json"}
+	data = {"block": block}
+
+	response = requests.put(url, headers=headers, json=data, verify=True)
+
+	# Ensure the request was successful
+	if response.status_code == 200:
+		block_status = json.loads(response.text)['block']
+		if block_status == block and block == True:
+			print(f"Successfully blocked room {internal_ID}")
+		elif block_status == block and block == False:
+			print(f"Successfully unblocked room {internal_ID}")
+		else:
+			print(f"Failed to set block status for room {internal_ID} to {block}")
+	else:
+		print(f"Error: Unable to set block status for room {internal_ID}")
+
+# Example:
+#$ curl -X PUT -H 'Authorization: Bearer ACCESS_TOKEN' -H 'Content-Type: application/json' -d '{"block": true}' 'https://matrix.perthchat.org/_synapse/admin/v1/rooms/!UQEvAyhSqHxohkIyvm:perthchat.org/block'
+#{"block":true}