peter/copbird_aufarbeitung
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
copbird_aufarbeitung/zusammenfassung.ipynb

1344 lines
56 KiB
Text
Raw Normal View History

init
2023-03-26 16:36:49 +00:00
{
"cells": [
{
"cell_type": "markdown",
"id": "83885e86-1ccb-46ec-bee9-a33f3b541569",
"metadata": {},
"source": [
"# Zusammenfassung der Analysen vom Hackathon für die Webside\n",
"\n",
"- womöglich zur Darstellung auf der Webside\n"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 1,
init
2023-03-26 16:36:49 +00:00
"id": "9bd1686f-9bbc-4c05-a5f5-e0c4ce653fb2",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"import altair as alt"
]
},
{
"cell_type": "markdown",
"id": "81780c9a-7721-438b-9726-ff5a70910ce8",
"metadata": {},
"source": [
"## Daten aufbereitung\n",
"\n",
"Dump der Datenbank vom 25.03.2023. Die verschiedene Tabellen der Datenbank werden einzeln eingelesen. Zusätzlich werden alle direkt zu einem Tweet zugehörige Information in ein Datenobjekt gesammelt. Die Informationen zu den GIS-Daten zu den einzelnen Polizeistadtion (\"police_stations\") sind noch unvollständig und müssen gegebenfalls nocheinmal überprüft werden.\n",
"\n"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 45,
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"id": "fcc48831-7999-4d79-b722-736715b1ced6",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
finished tidying data
2023-03-28 13:35:03 +00:00
"((479991, 3), (151690, 8), (151690, 4), (13327, 3), (163, 7))"
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
]
},
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 45,
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
finished tidying data
2023-03-28 13:35:03 +00:00
"# Merging different table of old (~2021) and new (~2022) scraper\n",
"\n",
"## cols: hashtag, url, mention (same for both)\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"tweets_meta = pd.concat([pd.read_csv(\"data/entity_old.tsv\", sep = \"\\t\"), # data from old scraper\n",
" pd.read_csv(\"data/tweets.csv\")]) # data from new scraper\n",
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"## cols: id, tweet_text, created_at, user_id; only subset from old table (same tsv used in next step)\n",
"tweets_text = pd.concat([pd.read_csv(\"data/tweet_old.tsv\", sep = \"\\t\")[['id','tweet_text', 'created_at', 'user_id']].rename(columns = {\"id\":\"tweet_id\"}),\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" pd.read_csv(\"data/tweets-1679742698645.csv\")])\n",
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"## cols: id, like_count, retweet_count, reply_count, quote_count; only subset from old table\n",
"tweets_statistics = pd.concat([pd.read_csv(\"data/tweet_old.tsv\", sep = \"\\t\")[['id', 'like_count', 'retweet_count', 'reply_count', 'quote_count']].rename(columns = {\"id\":\"tweet_id\"}),\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" pd.read_csv(\"data/tweets-1679742620302.csv\")])\n",
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"## cols: user_id, handle, user_name; colnames do not match betweend old an new data. Even username and handle seem to be mixed up in new data set (inverse order)\n",
"## Info: Only a small amount of user_ids appear in both data sets, but if so username occasionaly have changed an therefore can not easily be merged\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"tweets_user = pd.read_csv(\"data/user_old.tsv\", \n",
finished tidying data
2023-03-28 13:35:03 +00:00
" sep = \"\\t\").rename(columns = {\"id\":\"user_id\",\"name\": \"user_name\"} # uniform names\n",
" ).merge(pd.read_csv(\"data/tweets-1679742702794.csv\" # merge with renamed new data\n",
" ).rename(columns = {\"username\":\"handle\", \"handle\": \"user_name\"}), # reverse col names\n",
" on = \"user_id\", # user_id as matching column\n",
" how = \"outer\", # keep all unique uer_ids\n",
" suffixes = [\"_2021\", \"_2022\"]) # identify column where username and label came from\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"## Some usernames corresponding to one user_id have changed overtime. For easier handling only the latest username and handle is kept.\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"tweets_user = tweets_user.assign(handle = tweets_user.apply(lambda row: row['handle_2021'] if pd.isna(row['handle_2022']) else row['handle_2022'], axis=1),\n",
" user_name = tweets_user.apply(lambda row: row['user_name_2021'] if pd.isna(row['user_name_2022']) else row['user_name_2022'], axis=1)\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" ).drop(['handle_2021', 'handle_2022', 'user_name_2021', 'user_name_2022'], axis =1) # no longer needed\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"## addiditional information concerning the police stations\n",
"## cols: handle, name, typ, bundesland, stadt, lat, long\n",
"police_stations = pd.read_csv(\"data/polizei_accounts_geo.csv\", sep = \"\\t\" \n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" ).rename(columns = {\"Polizei Account\": \"handle\"})\n",
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"tweets_meta.shape, tweets_statistics.shape, tweets_text.shape, tweets_user.shape, police_stations.shape"
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
]
},
{
"cell_type": "markdown",
"id": "0f7b2b95-0a6c-42c6-a308-5f68d4ba94b9",
"metadata": {},
"source": [
"Jetzt können noch alle Tweet bezogenen informationen in einem Data Frame gespeichert werden:"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 24,
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"id": "f30c2799-02c6-4e6a-ae36-9e039545b6b3",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"metadata": {
"tags": []
},
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"outputs": [],
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"source": [
finished tidying data
2023-03-28 13:35:03 +00:00
"# Merge statistics, tweet text and user information in one data frame\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"tweets_combined = pd.merge(tweets_statistics, \n",
" tweets_text,\n",
" on = 'tweet_id').merge(tweets_user, on = 'user_id'\n",
" ).drop(['id'], axis = 1) # drop unascessary id column (redundant to index)\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" "
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 49,
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"id": "bd407aba-eec1-41ed-bff9-4c5fcdf6cb9d",
init
2023-03-26 16:36:49 +00:00
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>tweet_id</th>\n",
" <th>like_count</th>\n",
" <th>retweet_count</th>\n",
" <th>reply_count</th>\n",
" <th>quote_count</th>\n",
" <th>measured_at</th>\n",
" <th>is_deleted</th>\n",
" <th>tweet_text</th>\n",
" <th>created_at</th>\n",
init
2023-03-26 16:36:49 +00:00
" <th>user_id</th>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" <th>handle</th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>user_name</th>\n",
init
2023-03-26 16:36:49 +00:00
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>1321021123463663616</td>\n",
" <td>2</td>\n",
" <td>1</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>NaT</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>&lt;NA&gt;</td>\n",
" <td>@mahanna196 Da die Stadt keine Ausnahme für Radfahrer aufgeführt hat, gilt diese (Stand jetzt) a...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2020-10-27 09:29:13</td>\n",
" <td>778895426007203840</td>\n",
" <td>polizei_ol</td>\n",
" <td>Polizei Oldenburg-Stadt/Ammerland</td>\n",
init
2023-03-26 16:36:49 +00:00
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>1321037834246066181</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaT</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>&lt;NA&gt;</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>@mahanna196 Ja. *sr</td>\n",
" <td>2020-10-27 10:35:38</td>\n",
" <td>778895426007203840</td>\n",
" <td>polizei_ol</td>\n",
" <td>Polizei Oldenburg-Stadt/Ammerland</td>\n",
init
2023-03-26 16:36:49 +00:00
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>1321068234955776000</td>\n",
" <td>19</td>\n",
" <td>3</td>\n",
" <td>3</td>\n",
" <td>0</td>\n",
" <td>NaT</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>&lt;NA&gt;</td>\n",
" <td>#Aktuell Auf dem ehem. Bundeswehrkrankenhausgelände in #Rostrup wurde ein Sprengsatz gefunden. F...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2020-10-27 12:36:26</td>\n",
" <td>778895426007203840</td>\n",
" <td>polizei_ol</td>\n",
" <td>Polizei Oldenburg-Stadt/Ammerland</td>\n",
init
2023-03-26 16:36:49 +00:00
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>1321073940199100416</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaT</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>&lt;NA&gt;</td>\n",
" <td>@Emma36166433 Bitte lesen Sie unseren Tweet 2/2 *sr</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2020-10-27 12:59:06</td>\n",
" <td>778895426007203840</td>\n",
" <td>polizei_ol</td>\n",
" <td>Polizei Oldenburg-Stadt/Ammerland</td>\n",
init
2023-03-26 16:36:49 +00:00
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>1321088646506754049</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>NaT</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>&lt;NA&gt;</td>\n",
" <td>In der vergangenen Woche wurde die Wohnung des Tatverdächtigen durchsucht. Dabei stellten die Be...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2020-10-27 13:57:32</td>\n",
" <td>778895426007203840</td>\n",
" <td>polizei_ol</td>\n",
" <td>Polizei Oldenburg-Stadt/Ammerland</td>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" </tr>\n",
" <tr>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>151685</th>\n",
" <td>1625828803804004354</td>\n",
" <td>5</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>2023-02-19 13:40:36</td>\n",
" <td>False</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>#Sicherheit durch #Sichtbarkeit\\nUnsere #Dir3 hat zu diesem Thema wieder einmal die Puppen tanze...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2023-02-15 12:06:07</td>\n",
" <td>1168873095614160896</td>\n",
" <td>polizeiberlin_p</td>\n",
" <td>Polizei Berlin Prävention</td>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" </tr>\n",
" <tr>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>151686</th>\n",
" <td>1628004105623900167</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2023-02-25 13:14:49</td>\n",
" <td>False</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>Unser Präventionsteam vom #A44 berät heute und morgen tagsüber zum Thema Alkohol &amp;amp; Drogen + ...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2023-02-21 12:10:00</td>\n",
" <td>1168873095614160896</td>\n",
" <td>polizeiberlin_p</td>\n",
" <td>Polizei Berlin Prävention</td>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" </tr>\n",
" <tr>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>151687</th>\n",
" <td>1628004810183016448</td>\n",
" <td>6</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2023-02-25 13:14:49</td>\n",
" <td>False</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>Auch unser #A52 war heute aktiv und hat zum Thema Alkohol &amp;amp; Drogen im Straßenverkehr beraten...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2023-02-21 12:12:48</td>\n",
" <td>1168873095614160896</td>\n",
" <td>polizeiberlin_p</td>\n",
" <td>Polizei Berlin Prävention</td>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" </tr>\n",
" <tr>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>151688</th>\n",
" <td>1628352896352878593</td>\n",
" <td>2</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2023-02-26 13:15:05</td>\n",
" <td>False</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>Gestern führte unser #A13 in einer Wohnsiedlung einen Präventionseinsatz zum Thema „Wohnraumeinb...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2023-02-22 11:15:58</td>\n",
" <td>1168873095614160896</td>\n",
" <td>polizeiberlin_p</td>\n",
" <td>Polizei Berlin Prävention</td>\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
" </tr>\n",
" <tr>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>151689</th>\n",
" <td>1628709531998998529</td>\n",
" <td>10</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2023-02-27 12:17:33</td>\n",
" <td>False</td>\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" <td>Auf dem Gelände der @BUFAStudios (Oberlandstr. 26-35) findet heute die #Seniorenmesse vom Bezirk...</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>2023-02-23 10:53:07</td>\n",
" <td>1168873095614160896</td>\n",
" <td>polizeiberlin_p</td>\n",
" <td>Polizei Berlin Prävention</td>\n",
init
2023-03-26 16:36:49 +00:00
" </tr>\n",
" </tbody>\n",
"</table>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"<p>151690 rows × 12 columns</p>\n",
init
2023-03-26 16:36:49 +00:00
"</div>"
],
"text/plain": [
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" tweet_id like_count retweet_count reply_count \\\n",
"0 1321021123463663616 2 1 2 \n",
"1 1321037834246066181 2 0 0 \n",
"2 1321068234955776000 19 3 3 \n",
"3 1321073940199100416 0 0 0 \n",
"4 1321088646506754049 2 0 0 \n",
"... ... ... ... ... \n",
"151685 1625828803804004354 5 1 1 \n",
"151686 1628004105623900167 2 0 0 \n",
"151687 1628004810183016448 6 0 0 \n",
"151688 1628352896352878593 2 0 0 \n",
"151689 1628709531998998529 10 1 0 \n",
init
2023-03-26 16:36:49 +00:00
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" quote_count measured_at is_deleted \\\n",
"0 0 NaT <NA> \n",
"1 0 NaT <NA> \n",
"2 0 NaT <NA> \n",
"3 0 NaT <NA> \n",
"4 0 NaT <NA> \n",
"... ... ... ... \n",
"151685 0 2023-02-19 13:40:36 False \n",
"151686 0 2023-02-25 13:14:49 False \n",
"151687 0 2023-02-25 13:14:49 False \n",
"151688 0 2023-02-26 13:15:05 False \n",
"151689 0 2023-02-27 12:17:33 False \n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" tweet_text \\\n",
"0 @mahanna196 Da die Stadt keine Ausnahme für Radfahrer aufgeführt hat, gilt diese (Stand jetzt) a... \n",
"1 @mahanna196 Ja. *sr \n",
"2 #Aktuell Auf dem ehem. Bundeswehrkrankenhausgelände in #Rostrup wurde ein Sprengsatz gefunden. F... \n",
"3 @Emma36166433 Bitte lesen Sie unseren Tweet 2/2 *sr \n",
"4 In der vergangenen Woche wurde die Wohnung des Tatverdächtigen durchsucht. Dabei stellten die Be... \n",
"... ... \n",
"151685 #Sicherheit durch #Sichtbarkeit\\nUnsere #Dir3 hat zu diesem Thema wieder einmal die Puppen tanze... \n",
"151686 Unser Präventionsteam vom #A44 berät heute und morgen tagsüber zum Thema Alkohol &amp; Drogen + ... \n",
"151687 Auch unser #A52 war heute aktiv und hat zum Thema Alkohol &amp; Drogen im Straßenverkehr beraten... \n",
"151688 Gestern führte unser #A13 in einer Wohnsiedlung einen Präventionseinsatz zum Thema „Wohnraumeinb... \n",
"151689 Auf dem Gelände der @BUFAStudios (Oberlandstr. 26-35) findet heute die #Seniorenmesse vom Bezirk... \n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" created_at user_id handle \\\n",
"0 2020-10-27 09:29:13 778895426007203840 polizei_ol \n",
"1 2020-10-27 10:35:38 778895426007203840 polizei_ol \n",
"2 2020-10-27 12:36:26 778895426007203840 polizei_ol \n",
"3 2020-10-27 12:59:06 778895426007203840 polizei_ol \n",
"4 2020-10-27 13:57:32 778895426007203840 polizei_ol \n",
"... ... ... ... \n",
"151685 2023-02-15 12:06:07 1168873095614160896 polizeiberlin_p \n",
"151686 2023-02-21 12:10:00 1168873095614160896 polizeiberlin_p \n",
"151687 2023-02-21 12:12:48 1168873095614160896 polizeiberlin_p \n",
"151688 2023-02-22 11:15:58 1168873095614160896 polizeiberlin_p \n",
"151689 2023-02-23 10:53:07 1168873095614160896 polizeiberlin_p \n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"\n",
" user_name \n",
"0 Polizei Oldenburg-Stadt/Ammerland \n",
"1 Polizei Oldenburg-Stadt/Ammerland \n",
"2 Polizei Oldenburg-Stadt/Ammerland \n",
"3 Polizei Oldenburg-Stadt/Ammerland \n",
"4 Polizei Oldenburg-Stadt/Ammerland \n",
"... ... \n",
"151685 Polizei Berlin Prävention \n",
"151686 Polizei Berlin Prävention \n",
"151687 Polizei Berlin Prävention \n",
"151688 Polizei Berlin Prävention \n",
"151689 Polizei Berlin Prävention \n",
"\n",
"[151690 rows x 12 columns]"
init
2023-03-26 16:36:49 +00:00
]
},
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 49,
init
2023-03-26 16:36:49 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
finished tidying data
2023-03-28 13:35:03 +00:00
"# Convert datatypes to appropriate one\n",
"tweets_combined[['like_count', 'retweet_count', 'reply_count', 'quote_count']] = tweets_combined[['like_count', 'retweet_count', 'reply_count', 'quote_count']].fillna(np.NAN).astype(int)\n",
init
2023-03-26 16:36:49 +00:00
"tweets_combined = tweets_combined.assign(measured_at = pd.to_datetime(tweets_combined['measured_at']), # change date to date format\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" created_at = pd.to_datetime(tweets_combined['created_at']),\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" handle = tweets_combined['handle'].str.lower(), # handle to lower case\n",
" is_deleted = tweets_combined['is_deleted'].astype('boolean')) # is deleted column as boolean variable\n",
"tweets_combined#.to_csv(\"data/tweets_all_combined.csv\")"
init
2023-03-26 16:36:49 +00:00
]
},
{
"cell_type": "markdown",
"id": "91dfb8bb-15dc-4b2c-9c5f-3eab18d78ef8",
"metadata": {
"tags": []
},
"source": [
"### Adjazenzmatrix mentions\n",
" \n",
"Information, welche nicht direkt enthalten ist: welche Accounts werden erwähnt. Ist nur im Tweet mit @handle gekennzeichnet."
]
},
{
"cell_type": "code",
"execution_count": 53,
"id": "5d8bf730-3c8f-4143-b405-c95f1914f54b",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"0 Auch wir schließen uns dem Apell an! \\n\\n#Ukra...\n",
"1 @BWeltenbummler Sehr schwer zu sagen. Die Evak...\n",
"2 Halten Sie durch – die Evakuierung ist fast ab...\n",
"3 Halten Sie durch – die Evakuierung ist fast ab...\n",
"4 RT @drkberlin_iuk: 🚨 In enger Abstimmung mit d...\n",
"Name: tweet_text, dtype: object"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# TODO"
]
},
{
"cell_type": "markdown",
"id": "0c242090-0748-488c-b604-f521030f468f",
"metadata": {
"tags": []
},
"source": [
"## Metadaten \n",
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"Welche Daten bilden die Grundlage?\n"
init
2023-03-26 16:36:49 +00:00
]
},
{
"cell_type": "code",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"execution_count": 112,
init
2023-03-26 16:36:49 +00:00
"id": "0e5eb455-6b12-4572-8f5e-f328a94bd797",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"hashtag 267255\n",
"url 141594\n",
"mention 71142\n",
init
2023-03-26 16:36:49 +00:00
"Name: entity_type, dtype: int64"
]
},
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"execution_count": 112,
init
2023-03-26 16:36:49 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tweets_meta[\"entity_type\"].value_counts()\n",
"# tweets_meta[tweets_meta['entity_type'] == \"mention\"]"
]
},
{
"cell_type": "markdown",
"id": "ef440301-cf89-4e80-8801-eb853d636190",
"metadata": {
"tags": []
},
"source": [
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"Insgesamt haben wir 151690 einzigartige Tweets:"
init
2023-03-26 16:36:49 +00:00
]
},
{
"cell_type": "code",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"execution_count": 113,
init
2023-03-26 16:36:49 +00:00
"id": "5a438e7f-8735-40bb-b450-2ce168f0f67a",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"151690"
init
2023-03-26 16:36:49 +00:00
]
},
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"execution_count": 113,
init
2023-03-26 16:36:49 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tweets_combined[\"tweet_id\"].value_counts().shape[0] # Anzahl an Tweets"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 10,
init
2023-03-26 16:36:49 +00:00
"id": "4f1e8c6c-3610-436e-899e-4d0307259230",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
finished tidying data
2023-03-28 13:35:03 +00:00
"Die Tweets wurden vom 2020-10-27 bis zum: 2023-03-16 gesammelt. Also genau insgesamt: 870 Tage. (Mit kleinen Unterbrechungen)\n"
init
2023-03-26 16:36:49 +00:00
]
}
],
"source": [
finished tidying data
2023-03-28 13:35:03 +00:00
"print(\"Die Tweets wurden vom\", tweets_combined['created_at'].min().date(), \"bis zum:\", tweets_combined['created_at'].max().date(), \"gesammelt.\", \"Also genau insgesamt:\", (tweets_combined['created_at'].max() - tweets_combined['created_at'].min()).days, \"Tage. (Mit kleinen Unterbrechungen)\")\n",
init
2023-03-26 16:36:49 +00:00
"# tweets_combined[tweets_combined['created_at'] == tweets_combined['created_at'].max()] # Tweets vom letzten Tag"
]
},
{
"cell_type": "markdown",
"id": "d8b47a60-1535-4d03-913a-73e897bc18df",
"metadata": {
"tags": []
},
"source": [
"Welche Polizei Accounts haben am meisten getweetet?"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 11,
init
2023-03-26 16:36:49 +00:00
"id": "9373552e-6baf-46df-ae16-c63603e20a83",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"metadata": {
"tags": []
},
init
2023-03-26 16:36:49 +00:00
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>handle</th>\n",
" <th>count</th>\n",
" <th>Name</th>\n",
" <th>Typ</th>\n",
" <th>Bundesland</th>\n",
" <th>Stadt</th>\n",
" <th>LAT</th>\n",
" <th>LONG</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>polizei_ffm</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>5512</td>\n",
init
2023-03-26 16:36:49 +00:00
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>polizeisachsen</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>5340</td>\n",
init
2023-03-26 16:36:49 +00:00
" <td>Polizei Sachsen</td>\n",
" <td>Polizei</td>\n",
" <td>Sachsen</td>\n",
" <td>Dresden</td>\n",
" <td>51.0493286</td>\n",
" <td>13.7381437</td>\n",
" </tr>\n",
" <tr>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <th>3</th>\n",
" <td>polizei_nrw_do</td>\n",
" <td>4895</td>\n",
" <td>Polizei NRW DO</td>\n",
" <td>Polizei</td>\n",
" <td>Nordrhein-Westfalen</td>\n",
" <td>Dortmund</td>\n",
" <td>51.5142273</td>\n",
" <td>7.4652789</td>\n",
" </tr>\n",
" <tr>\n",
" <th>92</th>\n",
init
2023-03-26 16:36:49 +00:00
" <td>polizeibb</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>4323</td>\n",
init
2023-03-26 16:36:49 +00:00
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>61</th>\n",
" <td>polizeihamburg</td>\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
" <td>4042</td>\n",
init
2023-03-26 16:36:49 +00:00
" <td>Polizei Hamburg</td>\n",
" <td>Polizei</td>\n",
" <td>Hamburg</td>\n",
" <td>Hamburg</td>\n",
" <td>53.550341</td>\n",
" <td>10.000654</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" handle count Name Typ Bundesland \\\n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"11 polizei_ffm 5512 NaN NaN NaN \n",
"0 polizeisachsen 5340 Polizei Sachsen Polizei Sachsen \n",
"3 polizei_nrw_do 4895 Polizei NRW DO Polizei Nordrhein-Westfalen \n",
"92 polizeibb 4323 NaN NaN NaN \n",
"61 polizeihamburg 4042 Polizei Hamburg Polizei Hamburg \n",
init
2023-03-26 16:36:49 +00:00
"\n",
" Stadt LAT LONG \n",
"11 NaN NaN NaN \n",
"0 Dresden 51.0493286 13.7381437 \n",
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"3 Dortmund 51.5142273 7.4652789 \n",
"92 NaN NaN NaN \n",
init
2023-03-26 16:36:49 +00:00
"61 Hamburg 53.550341 10.000654 "
]
},
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 11,
init
2023-03-26 16:36:49 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"tweets_agg = tweets_combined.groupby(by = [\"user_id\", \"user_name\", \"handle\"]\n",
" )[\"user_id\"].aggregate(['count']\n",
" ).merge(police_stations,\n",
" on = \"handle\",\n",
" how = \"left\"\n",
" ).sort_values(['count'], ascending=False)\n",
init
2023-03-26 16:36:49 +00:00
"tweets_agg.shape\n",
"activy_police_vis = tweets_agg[0:50]\n",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"activy_police_vis.head()"
init
2023-03-26 16:36:49 +00:00
]
},
{
"cell_type": "markdown",
"id": "9cf5f544-706b-41af-b785-7023f04e3ecb",
"metadata": {
"tags": []
},
"source": [
"Visualisierung aktivste Polizeistadtionen:"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 13,
init
2023-03-26 16:36:49 +00:00
"id": "b1c39196-d1cc-4f82-8e01-7529e7b3046f",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"<div id=\"altair-viz-7b78525a62b243eca7b1f4044a328f47\"></div>\n",
init
2023-03-26 16:36:49 +00:00
"<script type=\"text/javascript\">\n",
" var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
" (function(spec, embedOpt){\n",
" let outputDiv = document.currentScript.previousElementSibling;\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" if (outputDiv.id !== \"altair-viz-7b78525a62b243eca7b1f4044a328f47\") {\n",
" outputDiv = document.getElementById(\"altair-viz-7b78525a62b243eca7b1f4044a328f47\");\n",
init
2023-03-26 16:36:49 +00:00
" }\n",
" const paths = {\n",
" \"vega\": \"https://cdn.jsdelivr.net/npm//vega@5?noext\",\n",
" \"vega-lib\": \"https://cdn.jsdelivr.net/npm//vega-lib?noext\",\n",
" \"vega-lite\": \"https://cdn.jsdelivr.net/npm//vega-lite@4.17.0?noext\",\n",
" \"vega-embed\": \"https://cdn.jsdelivr.net/npm//vega-embed@6?noext\",\n",
" };\n",
"\n",
" function maybeLoadScript(lib, version) {\n",
" var key = `${lib.replace(\"-\", \"\")}_version`;\n",
" return (VEGA_DEBUG[key] == version) ?\n",
" Promise.resolve(paths[lib]) :\n",
" new Promise(function(resolve, reject) {\n",
" var s = document.createElement('script');\n",
" document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
" s.async = true;\n",
" s.onload = () => {\n",
" VEGA_DEBUG[key] = version;\n",
" return resolve(paths[lib]);\n",
" };\n",
" s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
" s.src = paths[lib];\n",
" });\n",
" }\n",
"\n",
" function showError(err) {\n",
" outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
" throw err;\n",
" }\n",
"\n",
" function displayChart(vegaEmbed) {\n",
" vegaEmbed(outputDiv, spec, embedOpt)\n",
" .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
" }\n",
"\n",
" if(typeof define === \"function\" && define.amd) {\n",
" requirejs.config({paths});\n",
" require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
" } else {\n",
" maybeLoadScript(\"vega\", \"5\")\n",
" .then(() => maybeLoadScript(\"vega-lite\", \"4.17.0\"))\n",
" .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
" .catch(showError)\n",
" .then(() => displayChart(vegaEmbed));\n",
" }\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" })({\"config\": {\"view\": {\"continuousWidth\": 400, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-59538db49feb940cb722f8834432bfab\"}, \"mark\": \"bar\", \"encoding\": {\"x\": {\"field\": \"count\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"handle\", \"sort\": \"-x\", \"type\": \"ordinal\"}}, \"$schema\": \"https://vega.github.io/schema/vega-lite/v4.17.0.json\", \"datasets\": {\"data-59538db49feb940cb722f8834432bfab\": [{\"handle\": \"polizei_ffm\", \"count\": 5512, \"Name\": null, \"Typ\": null, \"Bundesland\": null, \"Stadt\": null, \"LAT\": null, \"LONG\": null}, {\"handle\": \"polizeisachsen\", \"count\": 5340, \"Name\": \"Polizei Sachsen\", \"Typ\": \"Polizei\", \"Bundesland\": \"Sachsen\", \"Stadt\": \"Dresden\", \"LAT\": \"51.0493286\", \"LONG\": \"13.7381437\"}, {\"handle\": \"polizei_nrw_do\", \"count\": 4895, \"Name\": \"Polizei NRW DO\", \"Typ\": \"Polizei\", \"Bundesland\": \"Nordrhein-Westfalen\", \"Stadt\": \"Dortmund\", \"LAT\": \"51.5142273\", \"LONG\": \"7.4652789\"}, {\"handle\": \"polizeibb\", \"count\": 4323, \"Name\": null, \"Typ\": null, \"Bundesland\": null, \"Stadt\": null, \"LAT\": null, \"LONG\": null}, {\"handle\": \"polizeihamburg\", \"count\": 4042, \"Name\": \"Polizei Hamburg\", \"Typ\": \"Polizei\", \"Bundesland\": \"Hamburg\", \"Stadt\": \"Hamburg\", \"LAT\": \"53.550341\", \"LONG\": \"10.000654\"}, {\"handle\": \"polizeimuenchen\", \"count\": 3951, \"Name\": \"Polizei M\\u00fcnchen\", \"Typ\": \"Polizei\", \"Bundesland\": \"Bayern\", \"Stadt\": \"M\\u00fcnchen\", \"LAT\": \"48.135125\", \"LONG\": \"11.581981\"}, {\"handle\": \"polizeimfr\", \"count\": 3317, \"Name\": \"Polizei Mittelfranken\", \"Typ\": \"Polizei\", \"Bundesland\": \"Bayern\", \"Stadt\": \"N\\u00fcrnberg\", \"LAT\": \"49.453872\", \"LONG\": \"11.077298\"}, {\"handle\": \"polizeimannheim\", \"count\": 3300, \"Name\": \"Polizei Mannheim\", \"Typ\": \"Polizei\", \"Bundesland\": \"Baden-W\\u00fcrttemberg\", \"Stadt\": \"Mannheim\", \"LAT\": \"49.4892913\", \"LONG\": \"8.4673098\"}, {\"handle\": \"bremenpolizei\", \"count\": 2664, \"Name\": null, \"Typ\": null, \"Bundesland\": null, \"Stadt\": null, \"LAT\": null, \"LONG\": null}, {\"handle\": \"polizei_ka\", \"count\": 2568, \"Name\": \"Polizei Karlsruhe\", \"Typ\": \"Polizei\", \"Bundesland\": \"Baden-W\\u00fcrttemberg\", \"Stadt\": \"Karlsruhe\", \"LAT\": \"49.0068705\", \"LONG\": \"8.4034195\"}, {\"handle\": \"polizei_nrw_k\", \"count\": 2544, \"Name\": \"Polizei NRW K\", \"Typ\": \"Polizei\", \"Bundesland\": \"Nordrhein-Westfalen\", \"Stadt\": \"K\\u00f6ln\", \"LAT\": \"50.938361\", \"LONG\": \"6.959974\"}, {\"handle\": \"polizei_nrw_bo\", \"count\": 2367, \"Name\": \"Polizei NRW BO\", \"Typ\": \"Polizei\", \"Bundesland\": \"Nordrhein-Westfalen\", \"Stadt\": \"Bochum\", \"LAT\": \"51.4818111\", \"LONG\": \"7.2196635\"}, {\"handle\": \"polizei_md\", \"count\": 2319, \"Name\": \"Polizei Magdeburg\", \"Typ\": \"Polizei\", \"Bundesland\": \"Sachsen-Anhalt\", \"Stadt\": \"Magdeburg\", \"LAT\": \"52.1315889\", \"LONG\": \"11.6399609\"}, {\"handle\": \"polizei_h\", \"count\": 2302, \"Name\": \"Polizei Hannover\", \"Typ\": \"Polizei\", \"Bundesland\": \"Niedersachsen\", \"Stadt\": \"Hannover\", \"LAT\": \"52.3744779\", \"LONG\": \"9.7385532\"}, {\"handle\": \"polizei_nrw_bi\", \"count\": 2299, \"Name\": \"Polizei NRW BI\", \"Typ\": \"Polizei\", \"Bundesland\": \"Nordrhein-Westfalen\", \"Stadt\": \"Bielefeld\", \"LAT\": \"52.0191005\", \"LONG\": \"8.531007\"}]}}, {\"mode\": \"vega-lite\"});\n",
init
2023-03-26 16:36:49 +00:00
"</script>"
],
"text/plain": [
"alt.Chart(...)"
]
},
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 13,
init
2023-03-26 16:36:49 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"barchart = alt.Chart(activy_police_vis[0:15]).mark_bar().encode(\n",
" x = 'count:Q',\n",
finished tidying data
2023-03-28 13:35:03 +00:00
" y = alt.Y('handle:O', sort = '-x'),\n",
init
2023-03-26 16:36:49 +00:00
")\n",
"barchart "
]
},
{
"cell_type": "markdown",
"id": "90f686ff-93c6-44d9-9761-feb35dfe9d1d",
"metadata": {
"tags": []
},
"source": [
"Welche Tweets ziehen besonders viel Aufmerksamkeit auf sich?"
]
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 14,
init
2023-03-26 16:36:49 +00:00
"id": "d0549250-b11f-4762-8500-1134c53303b4",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
finished tidying data
2023-03-28 13:35:03 +00:00
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>index</th>\n",
" <th>tweet_id</th>\n",
" <th>like_count</th>\n",
" <th>retweet_count</th>\n",
" <th>reply_count</th>\n",
" <th>quote_count</th>\n",
" <th>measured_at</th>\n",
" <th>is_deleted</th>\n",
" <th>tweet_text</th>\n",
" <th>created_at</th>\n",
" <th>user_id</th>\n",
" <th>handle</th>\n",
" <th>user_name</th>\n",
" <th>Name</th>\n",
" <th>Typ</th>\n",
" <th>Bundesland</th>\n",
" <th>Stadt</th>\n",
" <th>LAT</th>\n",
" <th>LONG</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>3053</td>\n",
" <td>1609539240458878979</td>\n",
" <td>21455</td>\n",
" <td>1845</td>\n",
" <td>3643</td>\n",
" <td>341</td>\n",
" <td>2023-01-05 14:44:34</td>\n",
" <td>False</td>\n",
" <td>Die Gewalt, die unsere Kolleginnen &amp;amp; Kollegen in der Silvesternacht erleben mussten, ist une...</td>\n",
" <td>2023-01-01 13:17:13</td>\n",
" <td>2397974054</td>\n",
" <td>polizeiberlin</td>\n",
" <td>Polizei Berlin</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1331</td>\n",
" <td>1355179228396879872</td>\n",
" <td>19186</td>\n",
" <td>3386</td>\n",
" <td>1203</td>\n",
" <td>628</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>An diejenigen, die vergangene Nacht in eine Schule in #Gesundbrunnen eingebrochen sind und 242 T...</td>\n",
" <td>2021-01-29 15:41:20</td>\n",
" <td>2397974054</td>\n",
" <td>polizeiberlin</td>\n",
" <td>Polizei Berlin</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>91693</td>\n",
" <td>1505620459148173316</td>\n",
" <td>15708</td>\n",
" <td>7098</td>\n",
" <td>186</td>\n",
" <td>540</td>\n",
" <td>2022-03-24 20:15:08</td>\n",
" <td>False</td>\n",
" <td>WICHTIGE Info:\\nÜber das Internet wird derzeit ein Video verbreitet, in dem von einem Überfall a...</td>\n",
" <td>2022-03-20 19:01:05</td>\n",
" <td>2389161066</td>\n",
" <td>polizei_nrw_bn</td>\n",
" <td>Polizei NRW BN</td>\n",
" <td>Polizei NRW BN</td>\n",
" <td>Polizei</td>\n",
" <td>Nordrhein-Westfalen</td>\n",
" <td>Bonn</td>\n",
" <td>50.735851</td>\n",
" <td>7.10066</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>91695</td>\n",
" <td>1505620666476896259</td>\n",
" <td>10337</td>\n",
" <td>1539</td>\n",
" <td>59</td>\n",
" <td>35</td>\n",
" <td>2022-03-24 20:15:08</td>\n",
" <td>False</td>\n",
" <td>Die Experten gehen derzeit davon aus, dass es sich um ein absichtliches \"Fake-Video\" handelt, da...</td>\n",
" <td>2022-03-20 19:01:54</td>\n",
" <td>2389161066</td>\n",
" <td>polizei_nrw_bn</td>\n",
" <td>Polizei NRW BN</td>\n",
" <td>Polizei NRW BN</td>\n",
" <td>Polizei</td>\n",
" <td>Nordrhein-Westfalen</td>\n",
" <td>Bonn</td>\n",
" <td>50.735851</td>\n",
" <td>7.10066</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>122631</td>\n",
" <td>1359098196434292739</td>\n",
" <td>9471</td>\n",
" <td>642</td>\n",
" <td>128</td>\n",
" <td>102</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>Weil wir dich schieben! @BVG_Kampagne 😉 https://t.co/N8kdlCxhz2</td>\n",
" <td>2021-02-09 11:13:55</td>\n",
" <td>4876039738</td>\n",
" <td>bpol_b</td>\n",
" <td>Bundespolizei Berlin</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>151685</th>\n",
" <td>7569</td>\n",
" <td>1332625325654757377</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>Sinken die Temperaturen ❄, steigt zeitgleich das Risiko für Verkehrsteilnehmer. Höchste Zeit zu ...</td>\n",
" <td>2020-11-28 10:00:11</td>\n",
" <td>223758384</td>\n",
" <td>polizeisachsen</td>\n",
" <td>Polizei Sachsen</td>\n",
" <td>Polizei Sachsen</td>\n",
" <td>Polizei</td>\n",
" <td>Sachsen</td>\n",
" <td>Dresden</td>\n",
" <td>51.0493286</td>\n",
" <td>13.7381437</td>\n",
" </tr>\n",
" <tr>\n",
" <th>151686</th>\n",
" <td>7572</td>\n",
" <td>1332738525507186692</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>📺Am Sonntag, um 19:50 Uhr, geht es bei #KripoLive im \\n@mdrde\\n auch um die Fahndung nach einem ...</td>\n",
" <td>2020-11-28 17:30:00</td>\n",
" <td>223758384</td>\n",
" <td>polizeisachsen</td>\n",
" <td>Polizei Sachsen</td>\n",
" <td>Polizei Sachsen</td>\n",
" <td>Polizei</td>\n",
" <td>Sachsen</td>\n",
" <td>Dresden</td>\n",
" <td>51.0493286</td>\n",
" <td>13.7381437</td>\n",
" </tr>\n",
" <tr>\n",
" <th>151687</th>\n",
" <td>144702</td>\n",
" <td>1465679768494526467</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>Musik verbindet!\\nUnser #Adventskalender der #Bundespolizei startet morgen ➡ https://t.co/V6CaTV...</td>\n",
" <td>2021-11-30 13:51:02</td>\n",
" <td>4876085224</td>\n",
" <td>bpol_nord</td>\n",
" <td>Bundespolizei Nord</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>151688</th>\n",
" <td>144701</td>\n",
" <td>1464124290605977600</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>@gretchen_hann Hallo, diese Frage kann die Bundespolizei Spezialkräfte besser beantworten. Richt...</td>\n",
" <td>2021-11-26 06:50:07</td>\n",
" <td>4876085224</td>\n",
" <td>bpol_nord</td>\n",
" <td>Bundespolizei Nord</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>151689</th>\n",
" <td>66854</td>\n",
" <td>1376453040283209728</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>-99</td>\n",
" <td>NaT</td>\n",
" <td>NaN</td>\n",
" <td>#Bönen #Holzwickede - Verstöße gegen Coronaschutzverordnung: Polizei löst Gaststättenabend und F...</td>\n",
" <td>2021-03-29 08:35:52</td>\n",
" <td>2389263558</td>\n",
" <td>polizei_nrw_un</td>\n",
" <td>Polizei NRW UN</td>\n",
" <td>Polizei NRW UN</td>\n",
" <td>Polizei</td>\n",
" <td>Nordrhein-Westfalen</td>\n",
" <td>Unna</td>\n",
" <td>51.5348835</td>\n",
" <td>7.689014</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>151690 rows × 19 columns</p>\n",
"</div>"
],
init
2023-03-26 16:36:49 +00:00
"text/plain": [
finished tidying data
2023-03-28 13:35:03 +00:00
" index tweet_id like_count retweet_count reply_count \\\n",
"0 3053 1609539240458878979 21455 1845 3643 \n",
"1 1331 1355179228396879872 19186 3386 1203 \n",
"2 91693 1505620459148173316 15708 7098 186 \n",
"3 91695 1505620666476896259 10337 1539 59 \n",
"4 122631 1359098196434292739 9471 642 128 \n",
"... ... ... ... ... ... \n",
"151685 7569 1332625325654757377 -99 -99 -99 \n",
"151686 7572 1332738525507186692 -99 -99 -99 \n",
"151687 144702 1465679768494526467 -99 -99 -99 \n",
"151688 144701 1464124290605977600 -99 -99 -99 \n",
"151689 66854 1376453040283209728 -99 -99 -99 \n",
"\n",
" quote_count measured_at is_deleted \\\n",
"0 341 2023-01-05 14:44:34 False \n",
"1 628 NaT NaN \n",
"2 540 2022-03-24 20:15:08 False \n",
"3 35 2022-03-24 20:15:08 False \n",
"4 102 NaT NaN \n",
"... ... ... ... \n",
"151685 -99 NaT NaN \n",
"151686 -99 NaT NaN \n",
"151687 -99 NaT NaN \n",
"151688 -99 NaT NaN \n",
"151689 -99 NaT NaN \n",
"\n",
" tweet_text \\\n",
"0 Die Gewalt, die unsere Kolleginnen &amp; Kollegen in der Silvesternacht erleben mussten, ist une... \n",
"1 An diejenigen, die vergangene Nacht in eine Schule in #Gesundbrunnen eingebrochen sind und 242 T... \n",
"2 WICHTIGE Info:\\nÜber das Internet wird derzeit ein Video verbreitet, in dem von einem Überfall a... \n",
"3 Die Experten gehen derzeit davon aus, dass es sich um ein absichtliches \"Fake-Video\" handelt, da... \n",
"4 Weil wir dich schieben! @BVG_Kampagne 😉 https://t.co/N8kdlCxhz2 \n",
"... ... \n",
"151685 Sinken die Temperaturen ❄, steigt zeitgleich das Risiko für Verkehrsteilnehmer. Höchste Zeit zu ... \n",
"151686 📺Am Sonntag, um 19:50 Uhr, geht es bei #KripoLive im \\n@mdrde\\n auch um die Fahndung nach einem ... \n",
"151687 Musik verbindet!\\nUnser #Adventskalender der #Bundespolizei startet morgen ➡ https://t.co/V6CaTV... \n",
"151688 @gretchen_hann Hallo, diese Frage kann die Bundespolizei Spezialkräfte besser beantworten. Richt... \n",
"151689 #Bönen #Holzwickede - Verstöße gegen Coronaschutzverordnung: Polizei löst Gaststättenabend und F... \n",
"\n",
" created_at user_id handle user_name \\\n",
"0 2023-01-01 13:17:13 2397974054 polizeiberlin Polizei Berlin \n",
"1 2021-01-29 15:41:20 2397974054 polizeiberlin Polizei Berlin \n",
"2 2022-03-20 19:01:05 2389161066 polizei_nrw_bn Polizei NRW BN \n",
"3 2022-03-20 19:01:54 2389161066 polizei_nrw_bn Polizei NRW BN \n",
"4 2021-02-09 11:13:55 4876039738 bpol_b Bundespolizei Berlin \n",
"... ... ... ... ... \n",
"151685 2020-11-28 10:00:11 223758384 polizeisachsen Polizei Sachsen \n",
"151686 2020-11-28 17:30:00 223758384 polizeisachsen Polizei Sachsen \n",
"151687 2021-11-30 13:51:02 4876085224 bpol_nord Bundespolizei Nord \n",
"151688 2021-11-26 06:50:07 4876085224 bpol_nord Bundespolizei Nord \n",
"151689 2021-03-29 08:35:52 2389263558 polizei_nrw_un Polizei NRW UN \n",
"\n",
" Name Typ Bundesland Stadt LAT \\\n",
"0 NaN NaN NaN NaN NaN \n",
"1 NaN NaN NaN NaN NaN \n",
"2 Polizei NRW BN Polizei Nordrhein-Westfalen Bonn 50.735851 \n",
"3 Polizei NRW BN Polizei Nordrhein-Westfalen Bonn 50.735851 \n",
"4 NaN NaN NaN NaN NaN \n",
"... ... ... ... ... ... \n",
"151685 Polizei Sachsen Polizei Sachsen Dresden 51.0493286 \n",
"151686 Polizei Sachsen Polizei Sachsen Dresden 51.0493286 \n",
"151687 NaN NaN NaN NaN NaN \n",
"151688 NaN NaN NaN NaN NaN \n",
"151689 Polizei NRW UN Polizei Nordrhein-Westfalen Unna 51.5348835 \n",
"\n",
" LONG \n",
"0 NaN \n",
"1 NaN \n",
"2 7.10066 \n",
"3 7.10066 \n",
"4 NaN \n",
"... ... \n",
"151685 13.7381437 \n",
"151686 13.7381437 \n",
"151687 NaN \n",
"151688 NaN \n",
"151689 7.689014 \n",
"\n",
"[151690 rows x 19 columns]"
init
2023-03-26 16:36:49 +00:00
]
},
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 14,
init
2023-03-26 16:36:49 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
unifying handles and usernames
2023-03-27 19:30:05 +00:00
"tweets_attention = tweets_combined.merge(police_stations,\n",
" on = \"handle\",\n",
" how = \"left\")\n",
init
2023-03-26 16:36:49 +00:00
"pd.options.display.max_colwidth = 100\n",
finished tidying data
2023-03-28 13:35:03 +00:00
"tweets_attention.sort_values('like_count', ascending = False).reset_index()\n",
init
2023-03-26 16:36:49 +00:00
"\n"
]
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
},
{
"cell_type": "code",
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 42,
"id": "621a3b74-e909-435c-8820-b38b63aa4893",
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>user_id</th>\n",
" <th>handle</th>\n",
" <th>user_name</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1000004686156652545</td>\n",
" <td>6jannik9</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>1000043230870867969</td>\n",
" <td>LSollik</td>\n",
" <td>Physiolucy</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>1000405847460151296</td>\n",
" <td>Achim1949Hans</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>1000460805719121921</td>\n",
" <td>WahreW</td>\n",
" <td>WahreWorte</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>1000744009638252544</td>\n",
" <td>derD1ck3</td>\n",
" <td>Ⓓ①ⓒⓚ①③ (🏡)</td>\n",
" </tr>\n",
" <tr>\n",
" <th>...</th>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" <td>...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11554</th>\n",
" <td>99931264</td>\n",
" <td>Havok1975</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11555</th>\n",
" <td>999542638226403328</td>\n",
" <td>Madame_de_Saxe</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11556</th>\n",
" <td>999901133282754560</td>\n",
" <td>tungstendie74</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11557</th>\n",
" <td>999904275080794112</td>\n",
" <td>_danielheim</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11558</th>\n",
" <td>999955376454930432</td>\n",
" <td>amyman6010</td>\n",
" <td>Systemstratege:</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>11559 rows × 3 columns</p>\n",
"</div>"
],
"text/plain": [
" user_id handle user_name\n",
"0 1000004686156652545 6jannik9 Systemstratege: \n",
"1 1000043230870867969 LSollik Physiolucy\n",
"2 1000405847460151296 Achim1949Hans Systemstratege: \n",
"3 1000460805719121921 WahreW WahreWorte\n",
"4 1000744009638252544 derD1ck3 Ⓓ①ⓒⓚ①③ (🏡)\n",
"... ... ... ...\n",
"11554 99931264 Havok1975 Systemstratege: \n",
"11555 999542638226403328 Madame_de_Saxe Systemstratege: \n",
"11556 999901133282754560 tungstendie74 Systemstratege: \n",
"11557 999904275080794112 _danielheim Systemstratege: \n",
"11558 999955376454930432 amyman6010 Systemstratege: \n",
"\n",
"[11559 rows x 3 columns]"
]
},
finished tidying data
2023-03-28 13:35:03 +00:00
"execution_count": 42,
added old data and began writing code for merch
2023-03-26 19:50:55 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
finished tidying data
2023-03-28 13:35:03 +00:00
"source": []
init
2023-03-26 16:36:49 +00:00
}
],
"metadata": {
"kernelspec": {
"display_name": "python-scientific kernel",
"language": "python",
"name": "python-scientific"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in a new issue Copy permalink