add functions to examine if room is alrady blocked and to block/unblock rooms. dramatic speedup for applying rdlist by skipping already blocked rooms and just blocking unknown rooms instead of performing a full shutdown. add report_details section to user reporting to collect information from the admin about why the report was generated. add rdlist tag descriptions to allow for automated report_details generation

This commit is contained in:
PC-Admin 2023-07-27 03:39:48 +08:00
parent ad87420546
commit acf4bffc1a
5 changed files with 201 additions and 61 deletions

View File

@ -82,18 +82,10 @@ With the popular [matrix-docker-ansible-deploy](https://github.com/spantaleev/ma
To do:
1) Add the following functions:
- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#account-data - DONE
- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#list-all-pushers - DONE
- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#override-ratelimiting-for-users - DONE
- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#check-username-availability - DONE
- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#find-a-user-based-on-their-id-in-an-auth-provider
- https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#find-a-user-based-on-their-third-party-id-threepid-or-3pid
- https://github.com/matrix-org/synapse/blob/master/docs/admin_api/delete_group.md
2) Make the menu prettier! - DONE
3) Modularise the functions into multiple files - DONE
4) Use URI module for all API calls instead of curl - DONE
5) Add more automated rdlist function with sane defaults - DONE
6) Add fully automated (should just return a web link and decryption password) reporting functions for users:
2) Add fully automated (should just return a web link and decryption password) reporting functions for users:
- Description of why the report was made (what happened)
- User's ID - DONE
- Whois Data - DONE
@ -106,14 +98,14 @@ To do:
- Timestamp for when illegal material was accessed
- Description of report format and contents (to guide the reader)
- Summary of key information
7) Have recommended rdlist function return a list of offending accounts and the tags they accessed
8) Only email reportID in incident report?
9) Add a room report function to create a properly formatted report for rdlist
10) Skip already shutdown rooms for speeding up rdlist blocking
11) Add function for probing the support email of another server automatically
12) Automated incident report email to other server owners who has users in rdlist rooms for more scalable coordination
13) Automated public room joining and reminder if reporting email is not available?
14) Refine ipinfo module to also return region/state of IP
3) Have recommended rdlist function return a list of offending accounts and the tags they accessed
4) Only email reportID in incident report?
5) Add a room report function to create a properly formatted report for rdlist
6) Skip already shutdown rooms for speeding up rdlist blocking
7) Add function for probing the support email of another server automatically
8) Automated incident report email to other server owners who has users in rdlist rooms for more scalable coordination
9) Automated public room joining and reminder if reporting email is not available?
10) Refine ipinfo module to also return region/state of IP
***
@ -146,32 +138,66 @@ rdlist repo is up-to-date, no need to pull changes.
Using recommended rdlist tags. Rooms matching the following tags will be purged and/or blocked:
['hub_room_links', 'hub_room_trade', 'preban', 'degen_misc', 'beastiality', 'degen_porn', 'gore', 'snuff', 'degen_larp', 'hub_room_sussy', 'bot_spam', 'cfm', 'jailbait', 'bot_porn', 'toddlercon', 'loli', 'csam', 'tfm', 'degen_meet', 'stylized_3d_loli', '3d_loli']
WARNING! The following local users are current members of rooms tagged in rdlist: ['***REDACTED***:perthchat.org']
WARNING! The following local users are current members of rooms tagged in rdlist: ['@***REDACTED***:perthchat.org']
Do you want to generate a user report file for each of these users? y/n? n
Skipping user report generation...
Number of rdlist rooms being shutdown: 337
Number of rdlist rooms being shutdown: 346
Are you sure you want to shutdown these rooms? y/n? y
Are you sure you want to block/shutdown these rooms? y/n? y
Shutting down room: !***REDACTED***:matrix.org
!***REDACTED***:matrix.org has been successfully shutdown!
Skipping already blocked room: !***REDACTED***:matrix.org
Shutting down room: !***REDACTED***:matrix.org
!***REDACTED***:matrix.org has been successfully shutdown!
Skipping already blocked room: !***REDACTED***:matrix.org
Shutting down room: !***REDACTED***:anontier.nl
!***REDACTED***:anontier.nl has been successfully shutdown!
Skipping already blocked room: !***REDACTED***:matrix.org
Shutting down room: !***REDACTED***:anontier.nl
!***REDACTED***:anontier.nl has been successfully shutdown!
Blocking unknown room: !***REDACTED***:matrix.org
Successfully blocked room !***REDACTED***:matrix.org
Blocking unknown room: !***REDACTED***:matrix.org
Successfully blocked room !***REDACTED***:matrix.org
Skipping already blocked room: !***REDACTED***:matrix.org
Shutting down known room: !***REDACTED***:sibnsk.net
Sleeping for 2 seconds...
Sleeping for 4 seconds...
Sleeping for 8 seconds...
!***REDACTED***:sibnsk.net has been successfully shutdown!
List of kicked users:
@***REDACTED***:perthchat.org
Skipping already blocked room: !***REDACTED***:anontier.nl
Room shutdowns completed!
User login details for your moderator account:
Username: mod_team
Password: ***REDACTED***
Print rdlist statistics:
Number of rooms blocked: 4
Number of rooms purged: 2
Number of local users located in rdlist rooms and kicked: 1
The following users were current members of rooms tagged in rdlist: ['@***REDACTED***:perthchat.org']
Do you want to also deactivate all these accounts that were kicked from rdlist rooms? y/n?
...
```

View File

@ -51,9 +51,9 @@ while pass_token == False:
print("14) Collect account data.\t\t\t33) Delete multiple rooms.")
print("15) List account pushers.\t\t\t34) Purge the event history of a room to a specific timestamp.")
print("16) Get rate limit of a user account.\t\t35) Purge the event history of multiple rooms to a specific timestamp.")
print("17) Set rate limit of a user account.")
print("18) Delete rate limit of a user account.")
print("19) Check if user account exists.")
print("17) Set rate limit of a user account.\t\t36) Get blocked status for room.")
print("18) Delete rate limit of a user account.\t37) Block a room.")
print("19) Check if user account exists.\t\t38) Unblock a room.")
print("\n#### Server Commands ####\t\t\t\t\t#### Report Generation ####")
print("40) Delete and block a specific media.\t\t\t\t70) Generate user report.")
print("41) Purge remote media repository up to a certain date.\t\t71) Decrypt user report .zip file.")
@ -116,7 +116,7 @@ while pass_token == False:
elif user_account_exists == False:
print("\nUser account does not exist.\n")
elif menu_input == "20":
room_details_dict = room_commands.list_room_details('')
room_details_dict = room_commands.get_room_details('')
print(json.dumps(room_details_dict, indent=4, sort_keys=True))
elif menu_input == "21":
room_members_dict = room_commands.get_room_members('',False)
@ -149,6 +149,17 @@ while pass_token == False:
room_commands.purge_room_to_timestamp('','')
elif menu_input == "35":
room_commands.purge_multiple_rooms_to_timestamp()
elif menu_input == "36":
blocked_status = room_commands.get_block_status('')
if blocked_status == True:
print("\nRoom is blocked.\n")
elif blocked_status == False:
print("\nRoom is not blocked.\n")
print(json.dumps(blocked_status, indent=4, sort_keys=True))
elif menu_input == "37":
room_commands.set_block_status('', True)
elif menu_input == "38":
room_commands.set_block_status('', False)
elif menu_input == "40":
server_commands.delete_block_media()
elif menu_input == "41":
@ -167,7 +178,7 @@ while pass_token == False:
elif menu_input == "61":
ipinfo_commands.analyse_multiple_account_ips()
elif menu_input == "70":
report_commands.generate_user_report('')
report_commands.generate_user_report('','')
elif menu_input == "71":
report_commands.decrypt_zip_file()
elif menu_input == "72":

View File

@ -10,7 +10,32 @@ import room_commands
import report_commands
import hardcoded_variables
#rdlist_bot_username = hardcoded_variables.rdlist_bot_username
rdlist_tag_descriptions = {
"csam": "Child Sexual Abuse Material",
"cfm": "An abundance of content which would directly appeal to those seeking csam.",
"jailbait": "Photos which contain underage individuals in questionable or suggestive situations.",
"tfm": "An abundance of content which would directly appeal to those seeking jailbait.",
"beastiality": "Self explanatory.",
"3d_loli": "Pornography which depicts photorealistic underage characters.",
"stylized_3d_loli": "Pornography which depicts underage characters that are not depicted in a realistic style.",
"gore": "Self explanatory.",
"snuff": "Self explanatory.",
"degen_misc": "Other types of coomers rooms.",
"degen_larp": "Coomer larp rooms.",
"degen_meet": "Coomer socializing rooms.",
"degen_porn": "Rooms dedicated to pornography, excluding types which have dedicated tags.",
"bot_porn": "Rooms which contain bots that spam pornographic content.",
"bot_spam": "Rooms which contain bots that spam content. Primarily for malvertising and cryptospam",
"preban": "Rooms which may not contain tagged content, however have clear intent. i.e: Rooms with names like 'CP Room', 'Child Porn', etc",
"hub_room_trade": "Rooms which exist solely to trade illegal or questionable content. i.e: csam, jailbait",
"hub_room_sussy": "A room which is sussy. This tag does not have a solid definition, see existing tagged rooms",
"abandoned": "Similar to 'anarchy', primarily for rooms which have automated spam bots.",
"anarchy": "Unmoderated rooms.",
"hub_room_underage": "Rooms which contain a disproportionate amount of underage users.",
"hub_room_links": "Rooms which exist to share links to other rooms.",
"toddlercon": "Lolicon but younger.",
"loli": "Rooms which exist to host lolicon.",
}
def sync_rdlist():
rdlist_dir = "./rdlist"
@ -148,7 +173,7 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
# Deduplicate the list of all room_ids
all_room_ids = list(set(all_room_ids))
# Examine these room_ids for local users
# Examine these room_ids for local and remote users
all_local_users = []
all_remote_users = []
for room_id in all_room_ids:
@ -203,27 +228,42 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
elif preset_message != '':
message = preset_message
#print(f"all_room_ids: {all_room_ids}")
# Ask the user if they wish to block and purge all these rooms
shutdown_confirmation = input("\nNumber of rdlist rooms being shutdown: " + str(len(all_room_ids)) + "\n\nAre you sure you want to shutdown these rooms? y/n? ")
shutdown_confirmation = input("\nNumber of rdlist rooms being shutdown: " + str(len(all_room_ids)) + "\n\nAre you sure you want to block/shutdown these rooms? y/n? ")
total_list_kicked_users = []
num_rooms_blocked = 0
num_rooms_purged = 0
#print(f"all_room_ids: {all_room_ids}")
if shutdown_confirmation.lower() in ['y', 'yes', 'Y', 'Yes']:
for room_id in all_room_ids:
print(f"\n\nShutting down room: {room_id}")
room_state_dict = room_commands.export_room_state(room_id, "", False)
#print(f"\nroom_state_dict: {room_state_dict}")
if "Room not found" in room_state_dict.get('error', ''):
list_kicked_users = room_commands.shutdown_room(room_id, user_ID, new_room_name, message, False, True)
else:
list_kicked_users = room_commands.shutdown_room(room_id, user_ID, new_room_name, message, True, True)
blocked_status = room_commands.get_block_status(room_id)
#print(f"\nroom_details_dict: {room_details_dict}")
#print(f"\nblock_status: {blocked_status}")
# If room is already blocked, skip it
if blocked_status == False:
# Examine if unblocked room is known, if not block it
room_details_dict = room_commands.get_room_details(room_id)
if "Room not found" in room_details_dict.get('error', ''):
print(f"\n\nBlocking unknown room: {room_id}")
room_commands.set_block_status(room_id, True)
num_rooms_blocked += 1
# If unblocked room is known, perform a shutdown of the room
else:
print(f"\n\nShutting down known room: {room_id}")
list_kicked_users = room_commands.shutdown_room(room_id, user_ID, new_room_name, message, True, True)
num_rooms_purged += 1
total_list_kicked_users.extend(list_kicked_users)
if hardcoded_variables.testing_mode == True:
time.sleep(5)
elif blocked_status == True:
print(f"\n\nSkipping already blocked room: {room_id}")
if hardcoded_variables.testing_mode == True:
time.sleep(5)
elif shutdown_confirmation.lower() in ['n', 'no', 'N', 'No']:
print("\nSkipping these files...\n")
print("\nSkipping blocking/shutdown of rooms...\n")
return
else:
print("\nInvalid input, skipping these files...\n")
@ -236,22 +276,26 @@ def block_all_rooms_with_rdlist_tags(rdlist_use_recommended,preset_user_ID,prese
print(f"\n\nList of all kicked users: {total_list_kicked_users}\n")
# Return the list of all kicked users
return num_rooms_blocked, total_list_kicked_users
return num_rooms_blocked, num_rooms_purged, total_list_kicked_users
def block_recommended_rdlist_tags():
# Print warning if testing mode is enabled
if hardcoded_variables.testing_mode == True:
print("\nWARNING! Testing mode is enabled, this will reduce the amount of data generated in reports and greatly slow down rdlist blocking!\n")
# Check if user account already exists
account_query = user_commands.query_account(hardcoded_variables.rdlist_bot_username)
# Generate random password
preset_password = ''.join(random.choice(string.ascii_letters + string.digits) for i in range(20))
rdlist_bot_password = ''.join(random.choice(string.ascii_letters + string.digits) for i in range(20))
# If user is not found, create it
if 'User not found' in account_query:
# Create user account
user_commands.create_account(hardcoded_variables.rdlist_bot_username, preset_password)
user_commands.create_account(hardcoded_variables.rdlist_bot_username, rdlist_bot_password)
else:
print(f"\n@{hardcoded_variables.rdlist_bot_username}:{hardcoded_variables.base_url} account already exists. Resetting account password.")
user_commands.reset_password(hardcoded_variables.rdlist_bot_username, preset_password)
user_commands.reset_password(hardcoded_variables.rdlist_bot_username, rdlist_bot_password)
# Promote bot user to server admin
print(f"\nEnsuring @{hardcoded_variables.rdlist_bot_username}:{hardcoded_variables.base_url} account is a server admin.")
@ -262,16 +306,17 @@ def block_recommended_rdlist_tags():
preset_message = 'THIS ROOM VIOLATES SERVER POLICIES'
# Block all rooms with recommended tag set
num_rooms_blocked, total_list_kicked_users = block_all_rooms_with_rdlist_tags(True, hardcoded_variables.rdlist_bot_username, preset_new_room_name, preset_message)
num_rooms_blocked, num_rooms_purged, total_list_kicked_users = block_all_rooms_with_rdlist_tags(True, hardcoded_variables.rdlist_bot_username, preset_new_room_name, preset_message)
# Print user login details
print("\n\nRoom shutdowns completed!\n\nUser login details for your moderator account:\n")
print("Username: " + hardcoded_variables.rdlist_bot_username)
print("Password: " + preset_password)
print("Password: " + rdlist_bot_password)
# Print statistics for the admin
print(f"\nPrint rdlist statistics:")
print(f"\nNumber of rooms blocked/purged: {num_rooms_blocked}")
print(f"\nNumber of rooms blocked: {num_rooms_blocked}")
print(f"Number of rooms purged: {num_rooms_purged}")
print(f"Number of local users located in rdlist rooms and kicked: {len(total_list_kicked_users)}")
print(f"\nThe following users were current members of rooms tagged in rdlist: {total_list_kicked_users}")

View File

@ -62,7 +62,7 @@ def encrypt_user_folder(user_report_folder, username):
# You can return the password if you need to use it later, or you can directly print it here
return strong_password, zip_file_name + ".aes"
def generate_user_report(preset_username):
def generate_user_report(preset_username, report_details):
if len(preset_username) == 0:
username = input("\nPlease enter the username to automatically generate a report: ")
username = user_commands.parse_username(username)
@ -87,6 +87,13 @@ def generate_user_report(preset_username):
if os.path.exists(user_report_folder) == False:
os.mkdir(user_report_folder)
# Collect and write the report details to ./report/username/report_details.txt
if report_details == '':
report_details = input("\nPlease enter the details for this report. Include as much detail as possible, including:\n A description of what happened.\n Timestamps of events. \n Whether this user was a repeat offender, if so include details about previous incidents. \n Other user or rooms involved. \n Other evidence you've collected against this user. \n Whether the offending users were deactivated. \n Whether the offending rooms were shut down.")
report_details_file = open(user_report_folder + "report_details.txt", "w")
report_details_file.write(report_details)
report_details_file.close()
# Get user account data and write to ./report/username/account_data.json
account_data = user_commands.collect_account_data(username)
account_data_file = open(user_report_folder + "account_data.json", "w")
@ -155,7 +162,7 @@ def generate_user_report(preset_username):
for room in room_list:
count += 1
room = room.split(" ")[0]
room_details = room_commands.list_room_details(room)
room_details = room_commands.get_room_details(room)
# Check the room conditions to select the proper output folders
if room_details['joined_members'] == 2 and room_details['public'] == False:
@ -215,7 +222,7 @@ def lookup_homeserver_admin_email(preset_baseurl):
response = None
# If the request was successful, the status code will be 200
if response and response.status_code == 200:
if response.status_code == 200 and "email_address" in response.text:
# Parse the response as JSON
data = json.loads(response.text)

View File

@ -13,7 +13,7 @@ def parse_username(username):
username = username.replace(tail_end,'')
return username
def list_room_details(preset_internal_ID):
def get_room_details(preset_internal_ID):
if preset_internal_ID == '':
internal_ID = input("\nEnter the internal id of the room you wish to query (Example: !OLkDvaYjpNrvmwnwdj:matrix.org): ")
elif preset_internal_ID != '':
@ -345,6 +345,7 @@ def shutdown_room(preset_internal_ID,preset_user_ID,preset_new_room_name,preset_
status = "null"
count = 0
sleep_time = 1
list_kicked_users = []
while status != "complete" and count < 8:
time.sleep(sleep_time)
@ -362,7 +363,6 @@ def shutdown_room(preset_internal_ID,preset_user_ID,preset_new_room_name,preset_
if status == "complete":
print(f"{internal_ID} has been successfully shutdown!")
list_kicked_users = []
if str(output_json["results"][0]["shutdown_room"]["kicked_users"]) != '[]':
print("List of kicked users:")
for entry in output_json["results"][0]["shutdown_room"]["kicked_users"]:
@ -370,7 +370,6 @@ def shutdown_room(preset_internal_ID,preset_user_ID,preset_new_room_name,preset_
print(entry)
else:
print(f"Failed to shutdown {internal_ID}!")
list_kicked_users = []
return list_kicked_users
@ -582,3 +581,55 @@ def purge_multiple_rooms_to_timestamp():
# Example:
# See purge_room_to_timestamp()
def get_block_status(preset_internal_ID):
if preset_internal_ID == '':
internal_ID = input("\nEnter the internal id of a room to examine if it's blocked (Example: !OLkDvaYjpNrvmwnwdj:matrix.org): ")
else:
internal_ID = preset_internal_ID
url = f"https://{hardcoded_variables.homeserver_url}/_synapse/admin/v1/rooms/{internal_ID}/block"
headers = {"Authorization": f"Bearer {hardcoded_variables.access_token}"}
response = requests.get(url, headers=headers, verify=True)
# Ensure the request was successful
if response.status_code == 200:
block_status = json.loads(response.text)['block']
else:
print(f"Error: Unable to fetch block status for room {internal_ID}")
block_status = None
return block_status
# Example:
# $ curl -X GET -H 'Authorization: Bearer ACCESS_TOKEN' 'https://matrix.perthchat.org/_synapse/admin/v1/rooms/!IdieserRBwPdaCGYuKk:matrix.org/block'
# {"block":false}
def set_block_status(preset_internal_ID, block):
if preset_internal_ID == '':
internal_ID = input("\nEnter the internal id of a room to block/unblock (Example: !OLkDvaYjpNrvmwnwdj:matrix.org): ")
else:
internal_ID = preset_internal_ID
url = f"https://{hardcoded_variables.homeserver_url}/_synapse/admin/v1/rooms/{internal_ID}/block"
headers = {"Authorization": f"Bearer {hardcoded_variables.access_token}", "Content-Type": "application/json"}
data = {"block": block}
response = requests.put(url, headers=headers, json=data, verify=True)
# Ensure the request was successful
if response.status_code == 200:
block_status = json.loads(response.text)['block']
if block_status == block and block == True:
print(f"Successfully blocked room {internal_ID}")
elif block_status == block and block == False:
print(f"Successfully unblocked room {internal_ID}")
else:
print(f"Failed to set block status for room {internal_ID} to {block}")
else:
print(f"Error: Unable to set block status for room {internal_ID}")
# Example:
#$ curl -X PUT -H 'Authorization: Bearer ACCESS_TOKEN' -H 'Content-Type: application/json' -d '{"block": true}' 'https://matrix.perthchat.org/_synapse/admin/v1/rooms/!UQEvAyhSqHxohkIyvm:perthchat.org/block'
#{"block":true}