It is a WiFi honeypot (HoneyDB), it records ssh sessions of people attached to it (I got a very cute ssh message from another con-goer), and it is readily hackable to do other things as well. It also had an ad-hoc badge-to-badge thing going on, and if you were near BTV the organizers could push updates to it (properly signed and authenticated). It is also set up very well; it's root password randomizes on start-up and you can find it using the badge's built-in screen and buttons. You can plug a USB cable into the data port on the RasPi Zero W and enable USB Ethernet so you can ssh into it on a wired connection.

One of the help files in the badge's menu said to drop down to a shell. When I did, nothing seemed to happen, it was just a shell and I had no keyboard. After a couple of moments it switched back to the badge menu. So, I decided to do what any gamer with a controller would do: I entered the Konami code using the buttons. Something unlocked!

this is the konami code unlockable add-on.  now that you know how the unlock process works, know that there are a few other items to unlock.

So, after putting it off for quite a while I decided to get down to it and try to hack the badge like I was told to. So let's explore what that was like!

Finding the challenge

Following the procedure from the help file, I connected to the USB Ethernet on the device and SSH'd into the badge.

malachite@xps:~$ ssh -p 22336 root@192.168.7.4
root@192.168.7.4's password: 
Linux MalachiteOS 4.14.52+ #1123 Wed Jun 27 17:05:32 BST 2018 armv6l
root@MalachiteOS:~# ls -alh
total 52K
drwx------  5 root root 4.0K Aug  1 04:28 .
drwxr-xr-x 23 root root 4.0K Jul 31 08:22 ..
-rw-------  1 root root 1.5K Aug  1 04:28 .bash_history
-rw-r--r--  1 root root  570 Mar 12  2018 .bashrc
drwx------  3 root root 4.0K Jul 22 21:43 .cache
-rw-------  1 root root 1.6K Aug 23  2018 .fbtermrc
drwx------  2 root root 4.0K Jul 23 14:18 .gnupg
-r--------  1 root root   32 Aug  1 04:30 passwd.txt
-rw-r--r--  1 root root  148 Mar 12  2018 .profile
-rw-------  1 root root 1.0K Jul 23 13:41 .rnd
-rw-r--r--  1 root root   75 Jan  5  2019 .selected_editor
dr-x------  2 root root 4.0K Jul 31 23:58 .ssh
-rw-r--r--  1 root root   37 Dec 11  2018 .vimrc
root@MalachiteOS:~# cd .gnupg/
root@MalachiteOS:~/.gnupg# ls -alh
total 12K
drwx------ 2 root root 4.0K Jul 23 14:18 .
drwx------ 5 root root 4.0K Aug  1 04:28 ..
-rw------- 1 root root   32 Jul 23 14:18 pubring.kbx
root@MalachiteOS:~/.gnupg# 

The only thing that stood out to me was the .gnupg folder, which contained a keyring. This might get brought up later.

root@MalachiteOS:/# ls -alh
total 91K
drwxr-xr-x  23 root root  4.0K Jul 31 08:22 .
drwxr-xr-x  23 root root  4.0K Jul 31 08:22 ..
-rw-r--r--   1 root root     0 Jul 31 10:10 0
# entries cut for space
-rw-r--r--   1 root root     0 Jul 31 08:22 30
drwxr-xr-x   9  501 staff 4.0K Jul 23 08:27 badge
drwxr-xr-x   2 root root  4.0K Aug 17  2018 bin
drwxr-xr-x   3 root root  3.0K Jan  1  1970 boot
drwxr-xr-x   2 root root  4.0K Jun 27  2018 debootstrap
drwxr-xr-x  14 root root  3.4K Aug  1 04:29 dev
lrwxrwxrwx   1 root root    13 May 18 11:12 dialogrc -> /etc/dialogrc
lrwxrwxrwx   1 root root    19 May 18 10:52 dialogrc-green -> /etc/dialogrc-green
lrwxrwxrwx   1 root root    17 May 18 10:59 dialogrc-red -> /etc/dialogrc-red
drwxr-xr-x 100 root root  4.0K Aug  1 04:30 etc
drwxr-xr-x   8 root root  4.0K Jul 14 05:27 home
drwxr-xr-x  16 root root  4.0K May 25 15:12 lib
drwx------   2 root root   16K Jun 27  2018 lost+found
drwxr-xr-x   2 root root  4.0K Jun 27  2018 media
drwxr-xr-x   3 root root  4.0K Jan  5  2019 mnt
drwxr-xr-x   3 root root  4.0K Jun 27  2018 opt
dr-xr-xr-x  72 root root     0 Jan  1  1970 proc
drwx------   5 root root  4.0K Aug  1 04:28 root
drwxr-xr-x  17 root root   520 Aug  1 04:33 run
drwxr-xr-x   2 root root  4.0K Aug 17  2018 sbin
drwxr-xr-x   2 root root  4.0K Jun 27  2018 srv
-rw-r--r--   1 root root     1 Feb 14 12:23 status_tail.txt
dr-xr-xr-x  12 root root     0 Jan  1  1970 sys
drwxrwxrwt   7 root root  4.0K Aug  1 04:38 tmp
drwxr-xr-x  10 root root  4.0K Jun 27  2018 usr
drwxr-xr-x  12 root root  4.0K Jul 23 03:23 var
root@MalachiteOS:/# 

I didn't know what the 0...30 files were until later: They are the buttons on the face of the BTV badge. Other than that, there is a folder named badge. That looks interesting, we'll look at that after we check status_tail.txt.

root@MalachiteOS:/# cat status_tail.txt 

root@MalachiteOS:/# ls -alh badge
total 36K
drwxr-xr-x  9  501 staff 4.0K Jul 23 08:27 .
drwxr-xr-x 23 root root  4.0K Jul 31 08:22 ..
drwxr-xr-x  2  501 staff 4.0K Jul 23 07:03 addons
drwxr-xr-x  3  501 staff 4.0K Jul 30 15:47 admin
drwxr-xr-x  4  501 staff 4.0K Aug  2  2019 art
drwxr-xr-x  3  501 staff 4.0K Jul 31 07:49 bin
drwxr-xr-x  7 root root  4.0K Jul 31 09:07 data
drwxr-xr-x  2  501 staff 4.0K Jul 22 19:25 deploy
drwxr-xr-x  2  501 staff 4.0K Jul 23 09:01 help
root@MalachiteOS:/#

status_tail.txt was empty, and there's a bunch of folders in badge. data is the only one here owned by root, so I want to see what that's about before I go rooting through other people's stuff.

root@MalachiteOS:/badge# ls -alh data
total 76K
drwxr-xr-x 7 root    root    4.0K Jul 31 09:07 .
drwxr-xr-x 9     501 staff   4.0K Jul 23 08:27 ..
-rw-r--r-- 1 root    root       2 Jul 31 07:54 blingrepeat.txt
-rw-r--r-- 1 root    root      42 Jul 31 10:10 friends.txt
-rw-r--r-- 1 root    root      12 Jul 31 08:11 handle.txt
drwxr-xr-x 2 honeydb honeydb 4.0K Jul 31 08:27 honeydb
-rw-r--r-- 1 root    root      14 Jul 31 19:40 honeyssid.txt
-rw-r--r-- 1 root    root    1.1K Aug  1 00:52 log-all-simple.txt
-rw-r--r-- 1 root    root     387 Jul 31 19:41 log-honeydb-simple.txt
-rw-r--r-- 1 root    root       0 Jul 31 07:54 log-honeydb.txt
-rw-r--r-- 1 root    root       0 Jul 31 08:27 log-honeydhcp.txt
-rw-r--r-- 1 root    root      42 Jul 31 09:07 log-honeyssh-simple.txt
-rw-r--r-- 1 root    root       0 Jul 31 07:54 log-honeyssh.txt
-rw-r--r-- 1 root    root     690 Aug  1 00:52 log-honeywap-simple.txt
-rw-r--r-- 1 root    root    1.4K Aug  1 02:01 log-honeywap.txt
drwx------ 2 root    root     16K Jul 31 07:54 lost+found
drwxr-xr-x 2 root    root    4.0K Jul 31 08:11 messages
drwxr-xr-x 2 cowrie  cowrie  4.0K Jul 31 09:08 ssh
drwxr-xr-x 2 root    root    4.0K Aug  1 00:56 unlocks
root@MalachiteOS:/badge# ls -alh data/unlocks/
total 16K
drwxr-xr-x 2 root root 4.0K Aug  1 00:56 .
drwxr-xr-x 7 root root 4.0K Jul 31 09:07 ..
-rwxr-xr-x 1 root root  337 Aug  1 00:57 easy.sh
-rwxr-xr-x 1 root root  258 Jul 31 15:24 konami.sh
root@MalachiteOS:/badge# cat data/unlocks/konami.sh 
#!/bin/bash

# keep this here to read badge variables
source /badge/bin/badge_vars.sh

# put add-on code here
echo "this is the konami code unlockable add-on.  now that you know how the unlock process works, know that there are a few other items to unlock."
root@MalachiteOS:/badge# cat data/unlocks/easy.sh 
#!/bin/bash

clear

echo "congrats, you've unlocked the easy challenge.  there are 3 other unlock challenges that will require skill, luck, hashing or cracking.

prizes await for the first to unlock the hard, medium and expert challenges.

separate from this contest, there are also several games and utilities to unlock on the badge.
"
root@MalachiteOS:/badge# 

Okay. So those files were what I unlocked when I was pushing buttons. easy.sh was just the 'A' button, and konami.sh was... well, the konami code (Up Down Up Down Left Right Left Right A B Start). There was a contest but I have to assume someone else managed to unlock them during Def Con 27 and claimed their prize at the venue. Anyways, there are other things that I need to unlock, but at this point I am not sure they are all button inputs. Maybe there are some CTF things I need to do around the badge, so let's keep looking around.

Data has all of the user-facing variables, and of course the unlocks. The text tiles contain things like the handle, your HoneyDB scores, friends list, etc.. Basically anything that the badge displays to users. These are all probably handled by other scripts, so messing around with them won't do anything here.

So, typically bin is where all the executables and the like are. I reckon that the button scripts are there, and there might be a clue as to what buttons will unlock other challenges.

root@MalachiteOS:/badge# ls -alh bin
total 308K
drwxr-xr-x 3  501 staff 4.0K Jul 31 07:49 .
drwxr-xr-x 9  501 staff 4.0K Jul 23 08:27 ..          
# many, many entries removed for space
-rwxr-xr-x 1  501 staff 2.5K Jul 31 07:43 button_debug.py
-rwxr-xr-x 1  501 staff 4.1K Jul 31 07:43 button_handler_konami.py
-rwxr-xr-x 1  501 staff 3.3K Jul 31 07:43 button_handler_main_menu.py
-rwxr-xr-x 1  501 staff 3.4K Jul 31 07:43 button_handler_moon-buggy.py
-rwxr-xr-x 1  501 staff 3.4K Jul 31 07:43 button_handler_netris.py
-rwxr-xr-x 1  501 staff 2.9K Jul 31 07:43 button_handler_test.py
# many, many more entries removed for space
root@MalachiteOS:/badge#

There was a load of stuff in the bin folder. More than I expected, honestly. I ended up having to go back and pipe it through less to make it easier to look through them. Most of them were python scripts. I truncated the results above so that I could point out the scripts that caught my eye: button_*.py. I looked at all of them, and button_handler_konami.py is the one that has the booty. I am going to go through that right here.

#!/usr/bin/python
import RPi.GPIO as GPIO
import time
import signal
import sys
import subprocess
import hashlib

'''
Bunch of stuff truncated here, setting up the GPIO signaling for the face buttons
'''

global press
press=""

def interrupt_handler(channel):
    global press

    if channel == 19:
        #print("19 - A")
        press+="A"
        time.sleep(.01)
    elif channel == 26:
        #print("26 - START")
        press+="S"
        subprocess.call(['systemctl', 'restart', 'getty@tty1.service'], shell=False)
        time.sleep(.01)
    elif channel == 20:
        #print("20 - SELECT")
        subprocess.call(['/badge/bin/badge_display_pwm.sh'], shell=False)
        time.sleep(.01)
# A bunch more elif commands, setting up the buttons. Followed by the button initialization that reference these inturrupts.

tick = 0
while (tick < 20):
    time.sleep(.25)
    tick = tick + 1

# The rest of the script, continued below.

Alright. So I can tell from opening it that it uses /usr/bin/python and not python3. Then I can see that there is hashlib in there, which tells me there's going to be some authentication or hashing involved. Looking at press and the interrupt_handler(), I can see that whenever you press a button it appends a character to the string press. It's global, so it'll get used everywhere (I don't know if this gives it any other properties).

Now, I truncated the interrupt_handler(), but here's what to take away from it:

• Each button press adds a letter to string press
• The list of possible letters that gets added to press are A,S,U,B,L,D,R
• Start (S) and Select run subprocesses, that may or may not interrupt the rest of the script

From what I can tell about the tick part is that this determines how long you get to start pushing buttons, and if it gets interrupted by a button press, the interrupt sleeps for .01s. Without the button interrupts this gives you about five seconds to input buttons, which is a pretty reasonable amount of time to enter any codes.

What's next?

# welcome.  this should get you started :)
salt = 'XstblibaQNaAWO8dYo:'
codes = {
        '50f9d9597642a0d090d4a613bf81f9d8bc4c7b4ec1db48c9f4abf88225c3cfad' : 'konami',
        'b3a218481d173dcd066a42eb74702714c5bf7f0c77982666c703f0d2b78b4a35' : 'admin',
        'fda91114369fdd643f5ab0c12a808dfbc0d360a1eb2fd3d9a56e66220d000313' : 'debug',
        'df722c019cb312748828b3b547d685448e43092e65d78eae2999d39bd96052b3' : 'moonbuggy',
        '3a868834d0e8d1690020d9fd01793df75496a127d7e82dab8dece423ee0bad5f' : 'netris',
        '90974350053c8ea1fc721da375242d888bde1d5b7c818ab1aa2f5265bcc2eebc' : 'da',
        '1d5420fdff3a2529e89edc307eba788c9b177d8482d49e4f05a3bd71fadf2444' : 'easy', 
        'b62d9205a693b3601ade197944cd39f305c6388dd1b97cff1190b5b7e6801077' : 'medium', 
        '8c30f3cdcf02ec6e0fc8e97f12738be7e2b2f710f5888932da5163a8891f75ea' : 'hard', 
        'f2664042ea3d51d5b94f49c36d5d9892eb9b913fd306b9648e1fd5dcd848932f' : 'expert', 
    }

hashed = hashlib.sha256(salt + press.encode()).hexdigest()

for item in codes:
    if hashed in codes:
        passphrase = salt + press
        addone = '/badge/addons/' + codes[hashed] + '.sh.asc'
        addond = '/badge/data/unlocks/' + codes[hashed] + '.sh'
        
        cmd = '/usr/bin/gpg -q --batch --passphrase ' + passphrase + ' ' + '-d ' + addone + ' > ' + addond
        
        subprocess.Popen([cmd], shell=True)
        
        cmd = 'chmod +x ' + addond
        subprocess.Popen([cmd], shell=True)


        print codes[hashed] + ' unlocked!'
        time.sleep(1)
        break

Aha, here's the booty. Here is where I get to show some of my college education doing authentication and encryption and the like (and also how long it's been). Forgive me any incorrect terms. Let's break down the first half, before the for item in codes bit:

• The salt is XstblibaQNaAWO8dYo:, and this should be added to any hashing we do. This is like a nonce, which helps prevent dictionary attacks by adding additional randomness to the hashes.
• codes is a dictionary of salted hashes, next to what appear to be challenge names. Close analogy: usernames and password hashes.
• The hashes are sha256, and are generated by adding the salt to press and then running it through hashlib.sha256. The hexdigest moves it from a <byte object> in python to a string, which we can actually use later.

So, we happen to know what the value of press will be for two entries, konami and easy. If we add XstblibaQNaAWO8dYo: to A, and then run it through the hashing algorithm, we'll get a hash code that will match easy. For konami, it's XstblibaQNaAWO8dYo: + UUDDLRLRBAS (Up Up Down Down Left Right Left Right B A Start). I am getting a little ahead of myself though, and should probably explain what we'll do with this.

We are going to want to crack the hashes in the table, that is, attempt to find a matching set of words that when put into their hashing function will get us matches. We want to build what's called a rainbow table. We want to do this in an offline attack, so that our brute-forcing attempts are not being run on the device we're trying to break into. So what do we have that I have determined is useful to this goal?

• We've got a seven character alphabet: A,S,U,B,L,D,R
• We have the salt: XstblibaQNaAWO8dYo:
• We have the function used to generate the hashes: hashlib.sha256(salt + press.encode()).hexdigest()
• We have the list of all correct hashes.

A quick rundown of some cryptography and attacker concepts

Alright, quick run-down of terminology from above before we continue (how I remember/use them, and not what they may actually mean):

• A salt is like a nonce, which is a number that is used once in encryption. These exist to add randomness.
• A word is well, a word, that you want to encrypt. This stands in for plaintext which is unencrypted information, or passwords in our case.
• A rainbow table is a pre-computed table used for cracking password hashes. It's a table filled with words and the resulting hashes in different protocols, like sha256. We will be generating this table by brute-forcing all possible button combinations and finding matches. (For those about to be pedantic, brute-forcing and rainbow tables are different things. Actual rainbow tables are much more complex.)
• An offline attack is an attack that is not run on/against the active system; often cracking passwords/hashes, or decrypting files
• An alphabet in encryption is all the possible characters that could be part of a word or plaintext. If we are cracking actual passwords, the alphabet would be a-zA-Z0-9 and maybe a list of special characters.

I guess now that I have explained an alphabet I should take a moment to explain why a seven character is important for us. If our alphabet only contains seven characters, it means each character in a password can only be one of those seven characters. This decreases the amount of time it will take to compute all reasonable possibilities.

An important part about encryption (and hashing is a form of encryption, fight me Scott, pedantic teacher of my security class) is that encryption is meant to be timely, which is to say that it cannot be unbreakable. You want an attacker to spend as much time as possible decrypting/discovering your encrypted/hashed data, so much time that by the time the attacker succeeds the data is irrelevant. Encryption protocols/algorithms must advance as computational power advances to ensure attacks against encrypted data are unreasonable for most attackers.

Below is some math that shows how the size of the alphabet and the width increase computational time by adding an ideally unreasonable amount of possibilities to try.

4 [alphabet] ^ 4 [width] = 256 possible results
4 [alphabet] ^ 7 [width] = 16384 possible results
7 [alphabet] ^ 4 [width] = 2401 possible results
7 [alphabet] ^ 7 [width] = 823543 possible results

Anyways, we need to do the thing. So let's get back to this script, why don't we?

Back to business

The script way above includes how when it finds a matching has it locates the file, decrypts the file, moves it to the proper place on the badge, and then change the permissions so it belongs to root. I could take time to really parse it all out and make it into a script, but I want to push the buttons myself (unless one of the combinations is too long). We're going to be doing a proper attack, but first we should test the functions to make sure our understanding of the inputs is correct.

Let's test something first in REPL. (This fails in python 3 by the way, but succeeds in python 2.7.15+, as noted earlier)

Python 2.7.15+ (default, Nov 27 2018, 23:36:35) 
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> 
>>> salt = 'XstblibaQNaAWO8dYo:'
>>> press=""
>>> press+="A"
>>> codes = {
...         '50f9d9597642a0d090d4a613bf81f9d8bc4c7b4ec1db48c9f4abf88225c3cfad' : 'konami',
...         'b3a218481d173dcd066a42eb74702714c5bf7f0c77982666c703f0d2b78b4a35' : 'admin',
...         'fda91114369fdd643f5ab0c12a808dfbc0d360a1eb2fd3d9a56e66220d000313' : 'debug',
...         'df722c019cb312748828b3b547d685448e43092e65d78eae2999d39bd96052b3' : 'moonbuggy',
...         '3a868834d0e8d1690020d9fd01793df75496a127d7e82dab8dece423ee0bad5f' : 'netris',
...         '90974350053c8ea1fc721da375242d888bde1d5b7c818ab1aa2f5265bcc2eebc' : 'da',
...         '1d5420fdff3a2529e89edc307eba788c9b177d8482d49e4f05a3bd71fadf2444' : 'easy', 
...         'b62d9205a693b3601ade197944cd39f305c6388dd1b97cff1190b5b7e6801077' : 'medium', 
...         '8c30f3cdcf02ec6e0fc8e97f12738be7e2b2f710f5888932da5163a8891f75ea' : 'hard', 
...         'f2664042ea3d51d5b94f49c36d5d9892eb9b913fd306b9648e1fd5dcd848932f' : 'expert', 
...     }
>>> 
>>> hashed = hashlib.sha256(salt + press.encode()).hexdigest()
>>> 
>>> if hashed in codes:
...     print(codes[hashed])
... 
easy
>>> press="UUDDLRLRBAS"
>>> hashed = hashlib.sha256(salt + press.encode()).hexdigest()
>>> if hashed in codes:
...     print(codes[hashed])
...
>>> press="UUDDLRLRBA"
>>> hashed = hashlib.sha256(salt + press.encode()).hexdigest()
>>> if hashed in codes:
...     print(codes[hashed])
... 
konami
>>> 

We understand the inputs. We have also learned that the S in my konami entry earlier was incorrect. That's good to know. This also means we have two known good results that we can use to test against, and one of them is the first possible thing we'll find. Now we just have to build a brute-forcer in python using these bits. We need to make a script that will check every possible combination of letters from our alphabet with an arbitrary width.

I had a lot of difficulty getting it to loop past two characters, but luckily I had help from a friend, @kardonice who helped me out hugely by introducing itertools (which I had tried but didn't understand, and I also tried the wrong thing from itertools), and later multiprocessing to speed this up. Let's see the results of the first pass with the working (single-threaded) version.

Python 2.7.15+ (default, Nov 27 2018, 23:36:35)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> import itertools as it
>>>
>>> salt = 'XstblibaQNaAWO8dYo:'
>>> codes = {
...         '50f9d9597642a0d090d4a613bf81f9d8bc4c7b4ec1db48c9f4abf88225c3cfad' : 'konami',
...         'b3a218481d173dcd066a42eb74702714c5bf7f0c77982666c703f0d2b78b4a35' : 'admin',
...         'fda91114369fdd643f5ab0c12a808dfbc0d360a1eb2fd3d9a56e66220d000313' : 'debug',
...         'df722c019cb312748828b3b547d685448e43092e65d78eae2999d39bd96052b3' : 'moonbuggy',
...         '3a868834d0e8d1690020d9fd01793df75496a127d7e82dab8dece423ee0bad5f' : 'netris',
...         '90974350053c8ea1fc721da375242d888bde1d5b7c818ab1aa2f5265bcc2eebc' : 'da',
...         '1d5420fdff3a2529e89edc307eba788c9b177d8482d49e4f05a3bd71fadf2444' : 'easy',
...         'b62d9205a693b3601ade197944cd39f305c6388dd1b97cff1190b5b7e6801077' : 'medium',
...         '8c30f3cdcf02ec6e0fc8e97f12738be7e2b2f710f5888932da5163a8891f75ea' : 'hard',
...         'f2664042ea3d51d5b94f49c36d5d9892eb9b913fd306b9648e1fd5dcd848932f' : 'expert',
...     }
>>> alphabet = "ABSURDL"
>>>
>>> def check_match(password):
...     hashed = hashlib.sha256((salt + password).encode()).hexdigest()
...     if hashed in codes:
...         print(password, codes[hashed])
...
>>> for i in range(1, 11):
...    for comb in it.product(alphabet, repeat=i):
...        check_match(''.join(comb))
...
('A', 'easy')
# REDACTED
# REDACTED
# REDACTED
# REDACTED
# REDACTED
('UUDDLRLRBA', 'konami')
>>> exit()

Conclusion

The both the single and multithreaded versions of this script work perfectly. I managed to unlock one additional challenge after setting it to try combinations up to twenty and running it for nine hours on my Dell XPS 13, and only on three cores. I am going to start re-running the script on a more powerful machine, and hopefully I'll unlock the extreme one sometime soon! Special thanks to Munin, who got me my BTV badge and to @Kardonice, who helped me write the repeater.

Support the Author

Devon Taylor (They/Them) is a Canadian python developer, network architect, and consultant. Their blog covers technical problems they encounter and design and media analysis. Their twitter You can support their independent work via Patreon (USD), or directly through Ko-Fi (CAD).

Hi there! It has been a while since I made a post on my blog, and I figured what better than to do a wrap-up of my very successful DEFCON 27 trip. It was a blast and I got to meet up with some of my best friends for the first time, and it was as successful at every stage of the trip as it possibly could have.

The Definition of Success

It is really important to start with why my trip was successful, because success at a conference is going to be different for everyone attending. Here are the things that I wanted out of my trip:

• Don't pack anything I didn't need (Pack light, pack efficiently)
• Spend as much time as possible among friends
• Respect my own limits and boundaries
• Eat well, stay hydrated, be presentable, and get plenty of rest
• Bring home cool swag, but don't overspend

So, what did my DEFCON 27 experience look like at the end of it all? Here is it in list form:

• I used (nearly) everything that I packed.
• I did not have any time where I was alone and wondering what to do.
• I was among friends at almost all times.
• I was comfortable dipping out when I was tired or didn't feel comfortable, and had ensured I always had a way out if it was needed.
• My group and I ate meals often, and ensured each other were staying properly hydrated.
• There were not any times when I felt gross or that I wasn't presentable to new people I was meeting (yay frequent showers).
• I actually slept better in Vegas than I have at home all year. I got all my necessary rest and more.
• Brought home a Plunder Bug, Packet Squirrel, and a Proxmark 3 RDV4.0. Also brought back the Blue Team Village badge!
• I started with $1,000 USD. I bought my friends a lot of meals and presents, covered taxi costs, bought my swag and badges, and had exactly $3 USD left over!

Big Important Packing List

• Mec Duffle Bag (35L) (can squish to carry-on size)
  • Three books wrapped tight in a bag, and the bag taped in place. Book cube. They were a gift for a friend, and didn't return with me.
  • Mec Cascade Field Bag (Day bag)
  • Large Eagle Creek Packing Cube
    • 4x 100% Cotton T-shirts (They're very comfortable)
    • 2x Shorts
    • 1x Swimsuit (in a leak-resistant bag)
  • Small Eagle Creek Packing Cube
    • 4x 100% Cotton Boxer Briefs
    • 2x Quick-drying/antimicrobial polyester boxer briefs (for swimming, and generally useful)
    • 2x Crew-length socks (padded for walking)
    • 4x shorter socks (also padded for walking)
  • 1 Quart Leak-resistant Transparent Bag (wtf tsa, really)
    • Small bottle of Shampoo
    • Small bottle of body wash
    • Small aerosol can of shaving gel
    • Disposable Razor
    • Travel-size mouthwash
    • Travel-size toothpaste
    • Toothbrush
    • Floss
    • Anti-Acne stuff
    • Antiperspirant
  • Amazon Packing Cube
    • Small cardboard box for stickers and small swag items
    • Various sizes of anti-static bags
    • 4x Large Ziplock freezer bags
    • Ziplock bags
    • Anchored rubber bands
    • velcro cable ties
    • small zip ties (for like the inside of PC cases)
    • Very Small Eagle Creek Packing Cube
      • Power Block for TS100
      • Solder
      • Pelican Case
        • Small screwdriver
        • Side cutters
        • Disassembled TS100
• GoRuck G1 (21L)
  • Dell XPS 13
  • Kindle Paperwhite
  • Earbuds coiled around a Bluelounge spool thing
  • Portable Bluetooth Speaker
  • Hak5 Wifi Pineapple (Nano Tactical)
  • Bunch of CLIF bars
  • Insulated Water Bottle
  • "Necessities" Kit (LIHIT LAB Compact Pencil Case)
    • Mooltipass and USB-A to Micro-USB Cable
    • USB-A to USB-C Adapter
    • Yubikey FIDO U2F Key (But you should get the updated one here.)
    • 2x 16GB USB 3.0 Flash Drives (They have os installation/recovery media on them)
    • 1x 64GB USB 3.0 Flash Drive (mass storage)
    • Mechanical Pencil, Eraser, Ruler, Pen, Sharpie
    • Field Notes 48-Page Memo Book (They were much cheaper when I bought a set)
  • Charging Kit (Skooba Mini Cable Stable)
    • 10,000mAh RAVPower USB-C Battery Bank
    • Multi-Port USB-C Wall Changer
    • USB-C to USB 3.0 Cable (charges battery bank and phone)
    • Anker Powerline+ USB-C Cable (3ft) (charges laptop)
    • USB-A to Micro-USB cable (Charges battery bank and Kindle)
    • USB-C USB 3.0 4-port hub and Ethernet Adapter
    • Ethernet Cable (6 ft)
    • USB-A to USB-C Adapter
  • GoRuck Padded Field Pouch (3L)
    • Multi-day travel pill container (holding about 6 days worth of medication)
    • Official prescription reciepts for all the medication I had
    • Sheet of Imodium and information packet
    • Sheet of Reactin and information packet
    • Deodorant
    • Travel-sized hand sanitizer
    • Travel-sized moisturizer
    • 3x pairs of disposable earplugs

Loadout debrief

A super important part of traveling light is keeping track of the stuff you used and the stuff you didn't. Learning from your trips, cutting what you didn't need, and adding what you did makes future trips more successful! So, let's do that.

Stuff I packed but didn't use

The list of stuff I didn't actually use is way shorter than the list of stuff I packed. In fact the list of things that I packed but did not use consists of seven items:

• Swimsuit (friend forgot their swimsuit, so pool stuff canceled)
• Wifi Pineapple
• TS100 soldering iron, and solder
• small screwdriver
• rubber bands, velcro ties, freezer bags
• USB-C USB hub and Ethernet adapter, and ethernet cable

Out of those items, I will likely only leave the WiFi Pineapple behind next year.

The TS100 and small screwdriver I will keep in my luggage because they would be invaluable if a badge broke during the conference. The rubber bands, velcro ties, and freezer bags take up such a small ammount of space that I likely won't dispose of them, and they could come in handy in the right circumstances. I will certainly keep the USB-C hub/ethernet adapter and cable, because it fits in my charging kit and can come in useful at the hotel or at the conference in the future if I take part in a CTF or go to the packet hacking village.

Items that I packed and didn't expect to use, but ended up being invaluable

Earplugs. I did not expect to actually use the earplugs by mid-day friday, but I ended up handing out two sets to my friends and using the third set when the DJ at the bar kept turning up the music slowly. Next year I plan on packing an extra pair to share with another friend, since our walking-around crew tended to be about three to four people. The earplugs were also invaluable at the airport and on the flight home where there were many screaming infants shrieking the whole way. Lesson: Pack earplugs, you will be happy you have them.

Things I wish I had packed

Three items: A hairbrush, a USB-C/Thunderbolt to HDMI adapter (and HDMI cable), and a spare (unlocked) phone.

The hairbrush was an item that was actually on my packing list, but I had needed to use it at home the morning of my flight out and I just forgot to toss it in my bag. A friend ended up buying one and bringing it to me at the conference for Saturday. Thanks, by the way!

I wanted to watch a movie on Netflix at the hotel but I had completely forgotten that my laptop does not have a port that any hotel television is going to be able to plug into. I am going to purchase an adapter and cable and maybe put together a "hotel room kit" of stuff I bring that I only need at the hotel. It might just end up being an A/V kit, in which case it'll be useful if I ever do a talk.

A spare, fresh (unlocked) phone. Oh boy. My friend lost their phone on Friday, and we spent all of Friday night and Saturday night trying to find the phone and then to get them access to their accounts, but were unsuccessful at both. MFA doing what it's supposed to, I suppose. Anyways, wisdom is learning from the hardship of others, innit? I am going to start bringing DEF CON's favorite hacker accessory: A spare "burner" phone. I'll get one when my budget allows.

The spare phone will likely go into my "Essentials" kit, from above, and I will move the pens and notebook elsewhere. It will become an emergency "get running at 100% and lockdown old devices in minutes" kit with one of my yubikeys, my dedicated hardware password manager, and that 64GB flash drive with an encrypted backup of important files from my laptop. This is a kit that I've wanted to put in place for a while, so now is as good at time as any.

Other lessons learned

I learned some things just about traveling in general that I will keep in mind for next year.

1. A 100% disassembled soldering iron will not make it past Canadian TSA
2. Since my duffel bag is carry-on sized, I can 'courtesy check' (aka 'gate check') it for free. No baggage fee, but I still get to bring tools and lockpicks both ways in my small bag.

It wasn't a new lesson, but having the packing cubes and other various kits/bags made sure that I knew where everything was and made forgetting anything next to impossible. When I checked a kit, if there was an open space in it, there was something missing (and I knew what it was). When I was repacking my cubes, if I forgot anything it would be obvious by the size/shape of the cube. Organization good.

What did I actually carry around at the con?

My GoRuck GR1 was primarily for getting my laptop and tech to Las Vegas for the conference. Most of the contents stayed back at the hotel room. My main bag was the Mec Cascade Field Bag, which I had packed in my duffel (it squishes up real good). Here's what I actually had in the Field Bag (all from the lists above):

• Insulated Water Bottle
• Kindle Paperwhite (Gives a little structure to the bag and gives it a bit of weight, so it's not all floppy)
• Battery bank, USB-A to USB-C Cable, and USB-C to USB-C Cable
• Some CLIF bars
• Pen, small notebook
• Earplugs
• Hand sanitizer
• Anti-static bag filled with stickers to hand out and trade
• Passport and wallet

The bag actually had loads of space left over for whatever goodies I picked up between trips back to my room. Plenty of space for collecting stickers, badges, and buying stuff at the vendors or more water/snacks. Never wanted for or needed anything that I had left back at the room. For the things that I came to do, I always had what I needed on me at all times without being overburdened, and everything had a place to go back to when I wasn't using it.

Enough of packing lists and load-outs. What did you do?

The last time I went to DEF CON, it was okay, but there were a bunch of issues. The hotel I chose was too far away from the convention, I didn't bring enough money, I didn't know enough people, and I spent most of my time wandering around trying to find someone to hang out with. This year was a full shift.

I stayed at the Flamingo, which happened to be where Blue Team Village was. This ended up being a 15-30 minute walk from the farthest conference floor at Planet Hollywood, with most of that walk being inside. I take this moment now to recognize that I am an able-bodied person who walks quickly, and who can navigate casinos with ease. There was a back exit from Paris (just stick to the left wall when entering from Bally's and hitting the Paris casino area) that let you skip the crosswalk that would get you stuck out in the heat, and put Planet Hollywood only a two or three minute walk away. The longest and easily the worst part of the trip was the crosswalk between the Flamingo and Bally's. I paid for taxis a total of three times, spending a grand total of $100 (To and from airport, and once to meet friends. I over-tipped quite a bit).

What did I do with full access to all the conference areas without worrying about trips or costs? I met up with a friend on Thursday, and then met up with more friends on Friday, and we stuck together every day throughout the con. We're all a bunch of introverts and none of us were particularly great at approaching new people or trying to make new friends, so we stuck to our own and enjoyed our time together. I didn't head out to any parties, or spend too long in areas where the music or sound was too loud.

Our group was very good at checking in on each other and making sure everyone was okay. We ate plenty of meals and made sure each other had plenty of water when we were walking around. At least on my end, I was okay taking off to my room fairly early (typically before or around midnight), and I fell asleep pretty quickly.

Advice and Lessons Learned

I think that I had the best possible experience, for the type of experience I was looking for. Much of my travel and conference advice still stands true, so I am just going to call out some of the more important ones, and update them with new advice. Mostly, just go read that other post, it was much more information.

• Never check a bag if you can avoid it. This lowers the chance of your luggage being stolen, lost, or mishandled. If you do check bags, keep all your valuables in your carry-on. If your bag is carry-on size but you have some non-carry-on items, your airline might allow you to courtesy or gate check your bag for free.
• Print your boarding passes. The rest of this tip used to read "unless you have a battery pack", but now I will just say: Just print your boarding passes. See above regarding losing your phone.
• If you drink lots of water like I do you will have to dump your water into a garbage bin at every security checkpoint. This goes for all beverages. Dump it out before you're at the front of the line.
• Ask for consent before taking people's picture or touching them for any reason. It makes a huge difference, even if you know the person really well.
  • Making eye contact and doing the open-arms-hug? gesture and asking, "Hug?" is easy to execute and isn't awkward. Respect their decision if they say no.
• If you are better known by your online handle than your real name include your handle in your introduction. "Hi, I'm Malachite, also @AwfulyPrideful and @MalachiteOS on Twitter."
  • Also, try to bring a picture you can pin to your badge or something with your avatar on it. A lot of people do much better remember that than names and handles.
• Most important tip of all: Don't push yourself. If you get anxious and can't be on the conference floor for long periods, just duck out and recharge. Under no circumstance should you force yourself to stay at the conference all day. You will exhaust yourself and ruin the experience if you do.
• Stay hydrated, bring snacks. You will have a better experience this way, I promise you. My personal choices are tap water (yuck, but free) and CLIF bars (trail food, keeps energy up sans meal).
  • Staying hydrated means drink small amounts of water every 15-20 minutes. Drinking lots of water all at once will not hydrate you since your body will only absorb a bit of it at once.
• Con Flu is real. Wash your hands often, or carry around hand sanitizer and use it often. The alternative is to not touch anything, but that won't save you.
  • Good strategy is that if you are about to touch your face, eyes, or nose, stop and use hand sanitizer or wash your hands with soap and water.
  • Honestly though, last I heard most people get 'con flu' on the trip home from the conference.
• Wear good walking shoes. You will be on your feet most of the time, either walking around or standing in lines. Get good insoles / inserts for your shoes. Your feet will thank you for it.
• Carry lots of cash for tips. Tipping your service people is really important, and people attending conferences are known not to tip well. Also thank them and be patient because it's really busy and hard for them too.
• If you are at a bar or party, always order your own drink and keep your eyes on it. Stick with friends. Look after each other.

Support the Author

Devon Taylor (They/Them) is a Canadian python developer, network architect, and consultant. Their blog covers technical problems they encounter and design and media analysis. Their twitter You can support their independent work via Patreon (USD), or directly through Ko-Fi (CAD).

Hello! When I was learning system and network administration, building test environments and virtual labs was a vital skill. When working with Windows administration having solid, up-to-date reference images sped up testing by enabling us to tear down and reset environments in minutes.

Creating reference images for virtual environments allow you to skip installing a fresh operating system every time you need a new machine. Windows reference images need to be set up properly or they will cause issues if multiple copies of the same reference image are connected to the same active directory domain.

So, let's learn how to set up proper reference images for Windows 10 Enterprise and Microsoft Server 2016 in Client Hyper-V. At the end of this guide, we will have:

• Official ISO files for Windows 10 Enterprise and Microsoft Server 2016
• Virtual hard disk images of fresh installs of Windows 10 Enterprise and Microsoft Server 2016 that can be repeatedly deployed for testing

Gathering Materials

The first thing that you need is to have the installation files for Windows 10 Enterprise and Windows Server 2016. Windows 10 Enterprise and Windows 10 Enterprise LTSC are available for free here, and multiple versions of Windows Server 2016 are available here for free as well.

Note: The downloads from the Microsoft Evaluation Center ask you to fill out information on your company, and provide contact details. However, none of this information is validated or used, so enter dummy data.

The version of Windows 10 Enterprise that you choose will depend on the environment you want to emulate. You will want to select the ISO for Windows Server 2016, because you will be making your reference image yourself. Once you have the images, you can install Client Hyper-V and get working.

Installing Client Hyper-V

The official requirements for installing Client Hyper-V on Windows 10 are located on Microsoft Docs here.

1. Open the Start Menu and type Turn Windows Features on or off.
2. Select Turn Windows Features on or off from the list.
3. In the list, mark the checkbox for Hyper-V and press okay.
4. You will be prompted to restart to complete the installation. Do so.

Installing Client Hyper-V using PowerShell

Open PowerShell as Administrator and enter:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All
Restart-Computer

Setting up your reference VM in Hyper-V

1. Open Hyper-V Manager.
2. On the right side, under Actions, select New > Virtual Machine. The New Virtual Machine Wizard will open.
3. On the Specify Name and Location page, name your virtual machine. Click Next.
4. On the Specify Generation page, select Generation 1 or 2. Selecting Generation 2 will change some of the menus later on when working with this virtual machine, but otherwise has no impact on our function. Click Next.
5. On the Assign Memory page, give your virtual machine a decent amount of memory. I give my virtual machines 2-4GB and leave Dynamic Memory enabled. Dynamic Memory will let your Windows host reclaim memory not in use by your virtual machines. Click Next.
6. On the Configure Networking page, select Default Switch from the drop-down menu. Click next.
7. On the Connect Virtual Hard Disk page, select Create a virtual hard disk, and name the disk so it is easily identifiable later. Examples, win10ent-reference.vhdx and server2016-reference.vhdx.
8. On the Installation Options page, select Install an operating system from a bootable CD/DVD-ROM and select one of the ISO files you downloaded earlier. Click Finish.

Quick script for creating a virtual machine for a fresh installation

$VMName = ClientGold
$ISOPath = "C:\Hyper-V\ISOs\1534.RS3.Win10Ent.x64.iso

New-VM -Name $VMName -Generation 2 -MemoryStartupBytes 4GB -NewVHDPath "$VMName.vhdx" -NewVHDSize 127GB -SwitchName (Get-VMSwitch).Name
Add-VMDvdDrive -VMName $VMName -Path $ISOPath
Set-VMFirmware $VMName -FirstBootDevice (Get-VMDvdDrive $VMName)

Creating the reference images

To create the reference images, we will simply go through the installation process normally once you start up the virtual machine.

In Windows 10 Enterprise, you will be asked to login with a Microsoft account. Below and to the left there is a button that says Domain Join. Press that button to create a local account. We will be deleting this account before we finish up.

Once you have installed Windows 10 Enterprise or Windows Server 2016, open the  Settings app (Win + i) and navigate to Update & Security. Click Check for updates and let everything download and install. When it's finished, you will have to reboot, so do that. If it does not prompt you to reboot, do it manually.

After the reboot, and you are logged in fully, open File Explorer (Win + e) and navigate to C:\Windows\System32\Sysprep. Run sysprep.exe, and the Sysprep dialogue will appear.

Optionally, in Windows 10 Enterprise we can remove the account created on the VM. Select Enter Audit Mode, optionally select Generalize, Reboot and click okay.

When the device reboots and enters audit mode, open Windows Search (Win) and type Add, edit, or remove other users. The Settings app will open to the users page. Under Other Users, you should see the local administrator account that you made when you initially installed. Remove it.

Once that's done, open the Sysprep tool, select Enter Out of Box Experience (OOBE), Generalize, and Shutdown, and click OK.

If you don't care about leaving behind a user from the previous install, once Windows 10 Enterprise is finished updating you can just run Sysprep with Enter Out Of Box Experience (OOBE), Generalize, and Shutdown selected, and move on.

Windows Server 2016

Windows Server 2016 does not create a new user when you install it. Instead it just assigns the local Administrator account a password. This will be removed when it Sysprep is run. After you finish updating Windows Server run Sysprep with Enter Out Of Box Experience (OOBE), Generalize, and Shutdown selected, and click OK.

Last Steps

The final step is moving our reference disks to a permanent home. Before we can move the files, we need to remove any checkpoints that Hyper-V has made. Open the Hyper-V Manager window, select your reference machine, right-click the top-level Automated Checkpoint, and select Delete Checkpoint.

Deleting the checkpoint rolls the virtual machine's current state into it's previous state, creating a single hard drive file. Once it is all rolled up, we can move it to a dedicated folder for reference images so can find them later. On my machine, I use C:\HyperV\reference to store my reference images.

Using the reference images in Hyper-V

To use the reference images, we need to create a differencing disk. This is a VHDX file that only stores the differences between the current state and the state of the parent image. After the first time we boot it up, it will take up about 4-5GB, but that is significantly smaller than multiple different installations.

Note: Once you create a differencing disk and select the target reference disk, you cannot move the reference disk without breaking the child images.

1. In the Hyper-V Manager window, click New > Hard Disk.... The New Virtual Hard Disk Wizard will open.
2. In the wizard, select Differencing Disk, then click Next.
3. Choose the name of your virtual hard disk, and where it is stored. Click Next.
4. On the Configure Disk page, click Browse and select the reference image you want this disk to be based on. Then click Next.

Now that we have the differencing disk, all that's left is to create a new virtual machine to attach it to. When you are creating your virtual machine, at the hard drive step, select your hard drive. After you create the virtual machine, you will have to go into its settings and edit the boot order to ensure it boots from the drive first.

It may take a little while the first time one of your child disks start up. This is due to the Generalize that we preformed earlier that removed machine-specific information that now needs to be regenerated for this new machine.

Once a child VM has started, the bottom corner of the desktop shows the current status of the VM's license. The evaluation license should be between 90 and 180 days. Because we ran Sysprep.exe, every time we create a new child disk off our reference disk it will have a fresh license of 90 or 180 days.

What next?

I don't know what you needed these for. Create a lab environment. Deploy active directory. Configure group policy. Create a PKI environment. Infect them with malware (you'll need a more robust lab for this). Or do whatever else you would want short-term Windows virtual machines for.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Alright, so I wanted to work on my last crawler that did a couple new things. Here are the goals for this one.

• Use lxml's etree html parser over bs4
• Separate into functions / clean up code a bit
• Cut out duplicate entries

I do not understand xpaths, but that's the idea with this change is to learn about them. So, let's start with some raw guesses and some googling. I initially had some trouble, but thankfully stackoverflow came to the rescue.

import requests
import lxml
from lxml import etree

htmlparser = etree.HTMLParser()

def web(page,WebUrl):
    if(page>0):
        url = WebUrl
        code = requests.get(url)
        plain = code.text
        # Using a new html parser
        tree = etree.fromstring(plain, htmlparser)
        for link in tree.xpath('//a/@href'):
        	print(link)
if __name__ == '__main__':
	web(1,'https://notawful.org')

The result:

malachite@localhost:~/python/scraper3$ python3 scraper3-test.py
https://notawful.org/
https://notawful.org/about/
https://notawful.org/content/images/2018/07/devon_taylor_resume.pdf
https://notawful.org/reading-list/
https://www.patreon.com/notawful
https://ko-fi.com/notawful
https://twitter.com/awfulyprideful
https://feedly.com/i/subscription/feed/https://notawful.org/rss/
/hotkeys-shortcuts-commands/
/author/notawful/
/data-security-while-traveling/
/author/notawful/
/installing-and-troubleshooting-netbox/

Alright, so from here what I want to do is cut out duplicates and keep only the links to pages under my domain. To cut out the duplicates we are going to create a set() which we will dump our list of links into. This will automatically remove any duplicate entries. I don't want my web crawler to wander from my web page, so I am going to need to filter out anything that doesn't start with /.

def web(page,WebUrl):
    if(page>0):
        url = WebUrl
        code = requests.get(url)
        plain = code.text
        tree = etree.fromstring(plain, htmlparser)
        pages = set()
        na_tasks = []
        for link in tree.xpath("//a/@href"):
            if link.startswith('/'):
                na_tasks.append(link)
        pages.update(na_tasks)
        for entry in pages:
            print(WebUrl + entry)

And the results:

malachite@localhost:~/python/scraper3$ python3 scraper3-test.py
https://notawful.org/password-managers/
https://notawful.org/fixing-ghost-on-digitalocean/
https://notawful.org/sr-part2/
https://notawful.org/data-security-while-traveling/
https://notawful.org/lessons-learned-october-2017/
https://notawful.org/gogs-install/
https://notawful.org/college-university-guide/
https://notawful.org/installing-and-troubleshooting-netbox/
https://notawful.org/online-presence-on-a-budget/
...

That is exactly the list that I was looking to get. Now, I might want to structure this script a little differently so that I can reuse the same function again. Let's try that next. Also I am going to reduce another line down a little.

def make_request(url):
    resp = etree.fromstring(requests.get(url).text, htmlparser)
    return resp

def get_images(page,WebUrl):
    if(page>0):
        pages = set()
        na_tasks = []
        tree = make_request(WebUrl)
        for link in tree.xpath("//a/@href"):
            if link.startswith('/'):
                na_tasks.append(link)
        pages.update(na_tasks)
        for entry in pages:
            print(WebUrl + entry)

That looks cleaner. It produces the same result as above. So now I need to start digging for images on other pages. We're going to basically duplicate the xpath search, except instead of "//a/@href" we'll be looking for "//img/@src" instead. To make things a little cleaner in case I wanted to pipe the list of images to a downloader, I am going to separate remote and local images and append the root URL to the local images. Here's what the function ends up looking like:

def get_images(page,WebUrl):
    if(page>0):
        pages = set()
        na_tasks = []
        tree = make_request(WebUrl)
        for link in tree.xpath("//a/@href"):
            if link.startswith('/'):
                na_tasks.append(link)
        pages.update(na_tasks)
        image_local = []
        image_remote = []
        image_set = set()
        for entry in pages:
            req = WebUrl+entry
            resp = make_request(req)
            for img in resp.xpath('//img/@src'):
                if img.startswith('/'):
                    image_local.append(WebUrl+img)
                else:
                    image_remote.append(img)
        image_set.update(image_local)
        image_set.update(image_remote)
        for entry in image_set:
            print(entry)

And the results are exactly what I wanted to see, as well. Here are some of those:

malachite@localhost:~/python/scraper3$ python3 scraper3-test2.py
https://assets.digitalocean.com/articles/pdocs/site/control-panel/networking/domains/no-domains-yet.png
https://notawful.org/content/images/2018/07/role-services-window.png
https://notawful.org/content/images/2018/07/rootCDP.png
https://assets.digitalocean.com/articles/putty_do_keys/new_ssh_key_prompt.png
https://notawful.org/content/images/2018/07/certtemplatecomputer.png
https://notawful.org/content/images/2018/08/webcrawl1-chromeconsole.png
https://pbs.twimg.com/media/DUp9AABW0AA82cI.jpg:large
https://notawful.org/content/images/2018/07/submitarequest.png
https://notawful.org/content/images/2018/07/rootCAProperties.png
https://notawful.org/content/images/2018/07/rootCAName.png
https://notawful.org/content/images/2018/07/revoked.png
https://notawful.org/content/images/2018/07/adcsconfigurationwindow.png
https://imgs.xkcd.com/comics/security.png
...

This script took a little while to run because it had to make all the requests one by one and wait for each result before it could move onto the next step. If I were to continue iterating on this I would have to make this script make asynchronous requests. *wink*.

All in all, changing over the HTML parser and cleaning up the script took about two and a half hours. List of things that I used that I was not aware of when I started making this post:

• Anything related to xpaths
• set()
• .startswith()

Here's the final code for this post:

import requests
import lxml
from lxml import etree

htmlparser = etree.HTMLParser()

def make_request(url):
    resp = etree.fromstring(requests.get(url).text, htmlparser)
    return resp

def get_images(page,WebUrl):
    if(page>0):
        pages = set()
        na_tasks = []
        tree = make_request(WebUrl)
        for link in tree.xpath("//a/@href"):
            if link.startswith('/'):
                na_tasks.append(link)
        pages.update(na_tasks)
        image_local = []
        image_remote = []
        image_set = set()
        for entry in pages:
            req = WebUrl+entry
            resp = make_request(req)
            for img in resp.xpath('//img/@src'):
                if img.startswith('/'):
                    image_local.append(WebUrl+img)
                else:
                    image_remote.append(img)
        image_set.update(image_local)
        image_set.update(image_remote)
        for entry in image_set:
            print(entry)

if __name__ == '__main__':
    get_images(1,'https://notawful.org')

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Just collecting the hotkeys, shortcuts, and commands that I use frequently (and some I use infrequently but often have to go looking for).

Windows

General Use

• Win + L: Lock computer (very useful)
• Win + X: Quick link window (links to a bunch of administrative tools)
• Win + X, U: Shutdown menu, follow with a U for shutdown, R for restart, and I for sign out
• Win + Shift + S: Snipping tool (not the program, select an area of the screen to copy to clipboard as an image)
• Win + Shift + [Right|Left Arrow]: Move focused window to next monitor
• Win + [1-9]: Select window on taskbar, open pinned application if closed
• Win + E: Open Windows Explorer
  Bonus: Win + Spacebar: Switches through keyboard layout, lets you escape the French Canadian keyboard mode.

Administrative Stuff

• Win + X, I: Powershell
• Win + X, A: Powershell (Admin)
• Win + X, G: Computer Management (Task scheduler, event viewer, local users, performance, device manager, disk management)
• Win + I: Settings menu

Chromebook

Crouton/Shell

Ctl + Alt + T: Open Chrome developer terminal
Crtl + Alt + Shift + [Forward|Backward]: Shift back and forth between ChromeOS and graphical chroot

*nix

VI

Modes

• esc: Exit insert/append mode and enter command mode
• i: enter insert mode; insert text before cursor
• shift + i: enter insert mode; insert text at start of line
• a: enter append mode; append text after cursor
• shift + a: enter append mode; append text to end of line

Command Mode

• dd: delete this line
• d10000d: fuck this entire file (as long as file is under 10,000 lines)
• yy: cut (yank) this line
• pp: put buffer here (paste copied line)
• :read ~/secretkey: copy the contents of ~/secretkey to the current line
• ^: place cursor at start of line (start from the top)
• $: place cursor at end of line (collect money at end)
• /word: find next word; can find broadsword or password
• ?word: find previous word; can find broadsword or password
• /\<word\> find next complete instance of word; does not find swords or password
• ?\<word\> find previous complete instance of word; does not find swords or password
• :q: quit
• :q!: I SAID GOOD DAY, SIR. (Quit, but shouting and bypasses save prompts)
• :w: write to file (save)
• :wq: Write and quit

Git

Set-up

Getting git set up initially:

sudo apt install git
git config --global user.name "Name"
git config --global user.email email@example.com
git config --global core.editor vim
git config --global credential.helper cache
git config --global credential.helper 'cache --timeout=3600'

The credential.helper configurations are so that you don't have to re-enter your password repeatedly when cloning over HTTPS.

• git config --list: this will list the current configuration settings git has.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Ohai. I was thinking about how I would secure sensitive data while traveling without inconveniencing myself, of course. I figured it would be an interesting thought to share, to see if anyone wanted to comment on it.

Goals:

• Data is secure in transit
• Cannot be feasibly coerced into providing access to data
• Able to get up and running quickly after border crossings

Tools:

• Chromebook (Intel x86_64, with TPM)
• USB Flash Drive or Portable SSD
• Mooltipass
• VeraCrypt (Or your choice of disk/volume encryption software.)

So, this is how it would work, before you travel:

1. Create an encrypted volume/partition on your mass storage with VeraCrypt and use the Mooltipass to create a long, secure password.
2. Create a copy of your Mooltipass's card and back up the mooltipass's database (binary). Store these safely before you travel. Write the incorrect PIN on the back of both cards.
   • The Mooltipass allows three attempts at entering a pin (each digit is 1-F), and if you can't get in by then it erases the card which makes that card unable to decrypt your database.
   • When you back up your mooltipass's binary, the database is still fully encrypted and needs to be loaded onto a mooltipass and access with the correct card in order to access them again.
3. Back up your crouton chroot to your VeraCrypt protected portable storage. Put in anything else you'll need with you on there at the same time.
4. Powerwash your Chromebook. In the help article, do it through the settings menu.
   • ChromeOS encrypts its drive by default and stores the encryption keys in the Trusted Platform Module (TPM). When you Powerwash, it deletes the keys in the TPM, making it impossible (within reason) to decrypt the drive and recover files from the previous installation.
5. Log into your Chromebook with a management account. It shouldn't be a completely clean account, but an account with no access to any important files or tied to any of your other devices.

So, at this point you have a clean device, a USB drive, and a Mooltipass and its card. If you are stopped at the border, the only way for anyone including you to access the encrypted data you are traveling with is to use the Mooltipass. You can give them the wrong pin or enter it incorrectly, and now nobody can access the encrypted data.

In order to get back up and running once you've crossed the border:

1. Powerwash the chromebook again, just in case.
2. Log into your chromebook with whatever account you want to use.
3. Install crouton and create a temporary chroot.
4. Download/install veracrypt into your chroot, decrypt your portable storage.
5. Restore your previous chroot from your backup.

Now you're back up and running and you can continue on without any hassle. All you have to do now is backup your chroot again before you cross another border, powerwash your chromebook again, and you're ready to go.

All in all, this basically just saves you time when you are setting up your environment at your destination. You can also just backup your chroot to storage online and pull it back down at the destination, but this is definitely faster than that (and can be done offline).

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

There's a story to this. Short version is a friend posted about netbox, an IP address management and data center infrastructure tool. Later I wanted to redo my virutal lab environment from scratch but with better documentation. I figured I'd set up an instance of netbox to document how my lab should be. So here goes.

SECOND ATTEMPT AND TROUBLESHOOTING

After following the documentation once and making notes, this is what I had going into my second attempt at installing netbox. I am not going to break this up, this is just raw text file that I have been working out of. We'll get into a bunch of troubleshooting as we go, and I'll point out my changes along the way. This is pretty close to the documentation, with a few changes to respond to issues I ran into on my first attempt.

sudo apt update ; sudo apt upgrade
sudo apt autoremove

sudo apt-get install -y postgresql libpq-dev
sudo -u postgres psql
CREATE DATABASE netbox;
CREATE USER netbox WITH PASSWORD 'XY,gTr6GP-Xy';
GRANT ALL PRIVILEGES ON DATABASE netbox TO netbox;
\q

sudo apt-get install -y python3 python3-dev python3-setuptools build-essential libxml2-dev libxslt1-dev libffi-dev graphviz libpq-dev libssl-dev zlib1g-dev
sudo apt install -y python3-pip

wget https://github.com/digitalocean/netbox/archive/v2.4.2.tar.gz
sudo tar -xzf v2.4.2.tar.gz -C /opt
cd /opt/
sudo ln -s netbox-2.4.2/ netbox
cd /opt/netbox/

pip3 install -r requirements.txt

cd /opt/netbox/netbox/netbox
sudo cp configuration.example.py configuration.py
python3 /opt/netbox/netbox/generate_secret_key.py > ~/secretkey

sudo vi configuration.py
ALLOWED_HOSTS = ['192.168.0.11']
DATABASE = {
    'NAME': 'netbox',
    'USER': 'netbox',
    'PASSWORD': 'XY,gTr6GP-Xy',
    'HOST': 'localhost',
    'PORT': '',
}

SECRET_KEY = 'J8g4+*aFfpV#2tGEPOSA%NriQx=Yvb_uWzKLy-U)0^w$Tl&5os'

cd /opt/netbox/netbox/
python3 manage.py migrate
python3 manage.py createsuperuser
sudo python3 manage.py collectstatic --no-input

sudo apt install -y nginx
sudo vi /etc/nginx/sites-available/netbox
server {
    listen 80;

    server_name 192.168.0.11;

    client_max_body_size 25m;

    location /static/ {
        alias /opt/netbox/netbox/static/;
    }

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
    }
}

cd /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/netbox

cd ~
sudo pip3 install gunicorn --install-option="--install-scripts=/usr/bin" -t python_modules/
which gunicorn

sudo vi /opt/netbox/gunicorn_config.py

command = '/usr/bin/gunicorn'
pythonpath = '/opt/netbox/netbox'
bind = '127.0.0.1:8001'
workers = 3
user = 'www-data'

sudo apt install -y supervisor

sudo vi /etc/supervisor/conf.d/netbox.conf

[program:netbox]
command = gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi
directory = /opt/netbox/netbox/
user = www-data

[program:netbox-rqworker]
command = python3 /opt/netbox/netbox/manage.py rqworker
directory = /opt/netbox/netbox/
user = www-data

Ends in 502 Bad Gateway. This is an expected error. From the documents, 'If you receive a 502 (bad gateway) error, this indicates that gunicorn is misconfigured or not running.'

Checking the status of supervisor confirms that gunicorn is not running.

malachite@netbox:~$ service supervisor status
● supervisor.service - Supervisor process control system for UNIX
   Loaded: loaded (/lib/systemd/system/supervisor.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 18:10:01 UTC; 9min ago
     Docs: http://supervisord.org
  Process: 51436 ExecStop=/usr/bin/supervisorctl $OPTIONS shutdown (code=exited, status=0/SUCCESS)
 Main PID: 51437 (supervisord)
    Tasks: 1 (limit: 972)
   CGroup: /system.slice/supervisor.service
           └─51437 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf

Aug 09 18:10:06 netbox supervisord[51437]: 2018-08-09 18:10:06,610 INFO spawned: 'netbox' with pid 51468
Aug 09 18:10:06 netbox supervisord[51437]: 2018-08-09 18:10:06,614 INFO spawned: 'netbox-rqworker' with pid 51469
Aug 09 18:10:06 netbox supervisord[51437]: 2018-08-09 18:10:06,881 INFO exited: netbox-rqworker (exit status 1; not expected)
Aug 09 18:10:06 netbox supervisord[51437]: 2018-08-09 18:10:06,989 INFO exited: netbox (exit status 1; not expected)
Aug 09 18:10:09 netbox supervisord[51437]: 2018-08-09 18:10:09,995 INFO spawned: 'netbox' with pid 51475
Aug 09 18:10:09 netbox supervisord[51437]: 2018-08-09 18:10:09,998 INFO spawned: 'netbox-rqworker' with pid 51476
Aug 09 18:10:10 netbox supervisord[51437]: 2018-08-09 18:10:10,259 INFO exited: netbox-rqworker (exit status 1; not expected)
Aug 09 18:10:10 netbox supervisord[51437]: 2018-08-09 18:10:10,351 INFO gave up: netbox-rqworker entered FATAL state, too many start retries too quickly
Aug 09 18:10:10 netbox supervisord[51437]: 2018-08-09 18:10:10,368 INFO exited: netbox (exit status 1; not expected)
Aug 09 18:10:11 netbox supervisord[51437]: 2018-08-09 18:10:11,370 INFO gave up: netbox entered FATAL state, too many start retries too quickly

These workers exist because of the python scripts /etc/supervisor/conf.d/netbox.conf and /opt/netbox/gunicorn_config.py. We know that /etc/supervisor/conf.d/netbox.conf is working because supervisor is attempting to spawn the workers defined in the script.

/etc/supervisor/conf.d/netbox.conf is run first and reads:

[program:netbox]
command = gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi
directory = /opt/netbox/netbox/
user = www-data

[program:netbox-rqworker]
command = python3 /opt/netbox/netbox/manage.py rqworker
directory = /opt/netbox/netbox/
user = www-data

/opt/netbox/gunicorn_config.py is run by the netbox worker defined in the first script.

command = '/usr/bin/gunicorn'
pythonpath = '/opt/netbox/netbox'
bind = '127.0.0.1:8001'
workers = 3
user = 'www-data'

So, looking back at the status, it runs netbox first, and then starts netbox-rqworker. When netboxrqworker fails, it closes and takes netbox with it. Either that or it is starting them both and they are both failing for the same reason. Although it has to do with php, this question at StackOverflow brings up what I've been thinking.

"I more assume that the file is located under your home directory and the root user is not able access and run it. So I would try to change the location of the script."

The first time I had to sudo to put something into /opt/ I figured this was going to become an issue, especially since both /etc/supervisor/conf.d/netbox.conf and /opt/netbox/gunicorn_config.py both specify user = www-data. So there are two potential problems. Either gunicorn isn't installed correctly, as the 502 Bad Gateway implies, or www-data is the user attempting to run the scripts and cannot access anything in /opt/.

Running which gunicorn shows that gunicorn exists at /usr/bin/gunicorn which is what we intended when we ran sudo pip3 install gunicorn --install-option="--install-scripts=/usr/bin" -t python_modules/. So let's make sure that the agent is able to use gunicorn.

malachite@netbox:~$ ls -alh /usr/bin | tail -5
-rwxr-xr-x  1 root   root    1.8K Jun 28  2017 xzless
-rwxr-xr-x  1 root   root    2.2K Jun 28  2017 xzmore
-rwxr-xr-x  1 root   root     31K Jan 18  2018 yes
-rwxr-xr-x  1 root   root     19K Apr 16 20:14 zdump
-rwxr-xr-x  1 root   root     48K Jul 18 14:21 zipdetails
malachite@netbox:~$ ls -alh /usr/bin/gunicorn
-rwxr-xr-x 1 root root 390 Aug  9 17:37 /usr/bin/gunicorn

The permissions are identical to the typical binaries in /usr/bin, so we can check that off. This means gunicorn itself isn't the issue, which makes the official documentation's implication that Gunicorn is 'misconfigured' a little frustrating. This is clearly a permissions issue.

Let's test something. I cannot su - www-data to see if it can see inside /opt/, but I made another user netbox with --logon-disabled to see if I can even see inside of /opt/.

malachite@netbox:~$ sudo su - netbox
[sudo] password for malachite:
netbox@netbox:~$ ls -alh /opt/netbox
lrwxrwxrwx 1 root www-data 13 Aug  9 17:23 /opt/netbox -> netbox-2.4.2/
netbox@netbox:~$ ls -alh /opt/netbox/
total 84K
drwxrwsr-x  6 root www-data 4.0K Aug  9 17:39 .
drwxr-xr-x  3 root root     4.0K Aug  9 17:23 ..
-rw-rwSr--  1 root www-data  492 Aug  8 13:16 base_requirements.txt
-rw-rwSr--  1 root www-data 5.5K Aug  8 13:16 CONTRIBUTING.md
drwxrwsr-x 10 root www-data 4.0K Aug  8 13:16 docs
-rw-rwSr--  1 root www-data   17 Aug  8 13:16 .gitattributes
drwxrwsr-x  3 root www-data 4.0K Aug  8 13:16 .github
-rw-rwSr--  1 root www-data  190 Aug  8 13:16 .gitignore
-rw-r-Sr--  1 root www-data  118 Aug  9 17:39 gunicorn_config.py
-rw-rwSr--  1 root www-data  10K Aug  8 13:16 LICENSE.txt
-rw-rwSr--  1 root www-data 2.3K Aug  8 13:16 mkdocs.yml
drwxrwsr-x 17 root www-data 4.0K Aug  9 17:32 netbox
-rw-rwSr--  1 root www-data   38 Aug  8 13:16 old_requirements.txt
-rw-rwSr--  1 root www-data 2.3K Aug  8 13:16 README.md
-rw-rwSr--  1 root www-data  458 Aug  8 13:16 requirements.txt
drwxrwsr-x  2 root www-data 4.0K Aug  8 13:16 scripts
-rw-rwSr--  1 root www-data  297 Aug  8 13:16 .travis.yml
-rwxrwsr-x  1 root www-data 1.8K Aug  8 13:16 upgrade.sh

The fact that www-data has group permissions was the result of me attempting to fix this permissions issue previously by using this method from StackOverflow.

sudo chown -R root:www-data /opt/netbox
sudo chmod -R g+s /opt/netbox
sudo chown -R root:www-data /opt/netbox-2.4.2/
sudo chmod -R g+s /opt/netbox-2.4.2/

Either way, a non-sudo user can peak inside. Can it read execute anything?

netbox@netbox:~$ ls -alh /opt/netbox/netbox
total 76K
drwxrwsr-x 17 root www-data 4.0K Aug  9 17:32 .
drwxrwsr-x  6 root www-data 4.0K Aug  9 17:39 ..
drwxrwsr-x  7 root www-data 4.0K Aug  9 17:32 circuits
drwxrwsr-x  7 root www-data 4.0K Aug  9 17:32 dcim
drwxrwsr-x  8 root www-data 4.0K Aug  9 17:32 extras
-rwxrwsr-x  1 root www-data  305 Aug  8 13:16 generate_secret_key.py
drwxrwsr-x  7 root www-data 4.0K Aug  9 17:32 ipam
-rwxrwsr-x  1 root www-data  249 Aug  8 13:16 manage.py
drwxrwsr-x  3 root www-data 4.0K Aug  8 13:16 media
drwxrwsr-x  3 root www-data 4.0K Aug  9 17:32 netbox
drwxrwsr-x  8 root www-data 4.0K Aug  8 13:16 project-static
drwxrwsr-x  2 root www-data 4.0K Aug  8 13:16 reports
drwxrwsr-x  9 root www-data 4.0K Aug  9 17:32 secrets
drwxr-sr-x 14 root www-data 4.0K Aug  9 17:32 static
drwxrwsr-x 14 root www-data 4.0K Aug  8 13:16 templates
drwxrwsr-x  6 root www-data 4.0K Aug  9 17:32 tenancy
drwxrwsr-x  5 root www-data 4.0K Aug  9 17:32 users
drwxrwsr-x  6 root www-data 4.0K Aug  9 17:32 utilities
drwxrwsr-x  7 root www-data 4.0K Aug  9 17:32 virtualization
netbox@netbox:~$ python3 /opt/netbox/netbox/manage.py
Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

Oh, that's an issue isn't it? Another issue I suspected, but this confirmed it. ModuleNotFoundError: No module named 'django' means that it wasn't just gunicorn that needed to be installed specially somewhere. Using pip3 install -r requirements.txt only installed the requirements for malachite@netbox rather than system-wide. I had to use sudo pip3 install gunicorn --install-option="--install-scripts=/usr/bin" -t python_modules/ specifically because gunicorn wasn't being installed to /usr/bin/.

This has actually been an issue I have run into generally. Pip and Pip3 do not function the way developers are expecting it to on Ubuntu 18.04.1 LTS. They do not seem to be installed with python3-setuptools as implied by other documentation, and python3-pip has been a pain when it comes to installing modules.

This page from the official documentation says that there is a possibility that pip isn't installed by default in some cases.

Before I blow away my installation and reset this, let's see if running pip3 install -r requirements.txt with sudo will change where the packages are installed...

malachite@netbox:/opt/netbox$ sudo pip3 install -r requirements.txt
The directory '/home/malachite/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/malachite/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: Django<2.1,>=1.11 in /home/malachite/.local/lib/python3.6/site-packages (from -r requirements.txt (line 1))
Requirement already satisfied: django-cors-headers==2.4.0 in /home/malachite/.local/lib/python3.6/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied: django-debug-toolbar==1.9.1 in /home/malachite/.local/lib/python3.6/site-packages (from -r requirements.txt (line 3))
...

Alright, so pip3 is installing everything to /home/malachite/.local/lib/python3.6/site-packages. This is why when www-data attempts to execute the scripts it fails. Also, I know that www-data in particular is attempting to execute these scripts because in the supervisord documentation it states:

user
Instruct supervisord to use this UNIX user account as the account which runs the program. The user can only be switched if supervisord is run as the root user. If supervisord can’t switch to the specified user, the program will not be started.

Continuing to read the python documention, I have come across something rather important here.

Warning Recent Debian/Ubuntu versions have modified pip to use the “User Scheme” by default, which is a significant behavior change that can be surprising to some users.

Doing further digging I decided to look at some comparisons. Checking installed python3 modules on malachite@netbox:

malachite@netbox:~$ python3 -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
attrs (17.4.0)
Automat (0.6.0)
bcrypt (3.1.4)
blinker (1.4)
certifi (2018.4.16)
cffi (1.11.5)
chardet (3.0.4)
click (6.7)
cloud-init (18.3)
colorama (0.3.7)
command-not-found (0.3)
configobj (5.0.6)
constantly (15.1.0)
coreapi (2.3.3)
coreschema (0.0.4)
cryptography (2.3)
distro-info (0.18)
Django (2.0.8)
...

Versus installed python3 modules on netbox@netbox

netbox@netbox:~$ python3 -m pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
attrs (17.4.0)
Automat (0.6.0)
blinker (1.4)
certifi (2018.1.18)
chardet (3.0.4)
click (6.7)
cloud-init (18.3)
colorama (0.3.7)
command-not-found (0.3)
configobj (5.0.6)
constantly (15.1.0)
cryptography (2.1.4)
distro-info (0.18)
httplib2 (0.9.2)
hyperlink (17.3.1)
idna (2.6)
incremental (16.10.1)
Jinja2 (2.10)
jsonpatch (1.16)
...

I decided to go to this specific level because we see some differences. User malachite@netbox in just this top section of modules has the followind additional packages:

bcrypt (3.1.4)
cffi (1.11.5)
coreapi (2.3.3)
coreschema (0.0.4)
Django (2.0.8)
...

And more. This means that python packages are being installed per user, and since www-data doesn't have a shell, let a lone a home directory, it cannot use any non-system packages for python3.

So I am going to try to cheat, here.

malachite@netbox:~$ sudo su - root
root@netbox:~# pip3 install -r /opt/netbox/requirements.txt

Now let's restart supervisor and see what we get...

malachite@netbox:~$ service supervisor status
● supervisor.service - Supervisor process control system for UNIX
   Loaded: loaded (/lib/systemd/system/supervisor.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 19:53:04 UTC; 6s ago
     Docs: http://supervisord.org
  Process: 52035 ExecStop=/usr/bin/supervisorctl $OPTIONS shutdown (code=exited, status=0/SUCCESS)
 Main PID: 52036 (supervisord)
    Tasks: 2 (limit: 972)
   CGroup: /system.slice/supervisor.service
           ├─52036 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
           └─52080 python3 /opt/netbox/netbox/manage.py rqworker

Aug 09 19:53:07 netbox supervisord[52036]: 2018-08-09 19:53:07,220 INFO spawned: 'netbox' with pid 52071
Aug 09 19:53:07 netbox supervisord[52036]: 2018-08-09 19:53:07,221 INFO success: netbox-rqworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Aug 09 19:53:07 netbox supervisord[52036]: 2018-08-09 19:53:07,756 INFO exited: netbox (exit status 1; not expected)
Aug 09 19:53:07 netbox supervisord[52036]: 2018-08-09 19:53:07,772 INFO exited: netbox-rqworker (exit status 1; not expected)
Aug 09 19:53:08 netbox supervisord[52036]: 2018-08-09 19:53:08,775 INFO spawned: 'netbox-rqworker' with pid 52073
Aug 09 19:53:09 netbox supervisord[52036]: 2018-08-09 19:53:09,779 INFO spawned: 'netbox' with pid 52078
Aug 09 19:53:09 netbox supervisord[52036]: 2018-08-09 19:53:09,779 INFO success: netbox-rqworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Aug 09 19:53:10 netbox supervisord[52036]: 2018-08-09 19:53:10,239 INFO exited: netbox-rqworker (exit status 1; not expected)
Aug 09 19:53:10 netbox supervisord[52036]: 2018-08-09 19:53:10,255 INFO spawned: 'netbox-rqworker' with pid 52080
Aug 09 19:53:10 netbox supervisord[52036]: 2018-08-09 19:53:10,292 INFO exited: netbox (exit status 1; not expected)

That is a new error. netbox-rqworker entered running state and stayed up for more than 1 second. Now, on the incredibly small off chance that it works. The log shows that it started to work, but crashed immediately after. The web page wasn't able to load, but we're closer. It's now very unstable unstable rather than dead. This is suprisingly a very good sign. Back under root@netbox, let's run pip3 install gunicorn just in case.

Aug 09 20:02:16 netbox systemd[1]: Stopped Supervisor process control system for UNIX.
Aug 09 20:02:16 netbox systemd[1]: Started Supervisor process control system for UNIX.
Aug 09 20:02:16 netbox supervisord[53810]: 2018-08-09 20:02:16,993 CRIT Supervisor running as root (no user in config file)
Aug 09 20:02:16 netbox supervisord[53810]: 2018-08-09 20:02:16,995 INFO Included extra file "/etc/supervisor/conf.d/netbox.conf" during parsing
Aug 09 20:02:17 netbox supervisord[53810]: 2018-08-09 20:02:17,003 INFO RPC interface 'supervisor' initialized
Aug 09 20:02:17 netbox supervisord[53810]: 2018-08-09 20:02:17,003 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Aug 09 20:02:17 netbox supervisord[53810]: 2018-08-09 20:02:17,003 INFO supervisord started with pid 53810
Aug 09 20:02:18 netbox supervisord[53810]: 2018-08-09 20:02:18,006 INFO spawned: 'netbox' with pid 53835
Aug 09 20:02:18 netbox supervisord[53810]: 2018-08-09 20:02:18,009 INFO spawned: 'netbox-rqworker' with pid 53836
Aug 09 20:02:19 netbox supervisord[53810]: 2018-08-09 20:02:19,559 INFO success: netbox entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Aug 09 20:02:19 netbox supervisord[53810]: 2018-08-09 20:02:19,559 INFO success: netbox-rqworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Aug 09 20:02:22 netbox supervisord[53810]: 2018-08-09 20:02:22,674 INFO exited: netbox-rqworker (exit status 1; not expected)
Aug 09 20:02:23 netbox supervisord[53810]: 2018-08-09 20:02:23,680 INFO spawned: 'netbox-rqworker' with pid 53869

Checking the web site, it was still down. Refreshed again a little after and it's back up. Let's check the status one last time.

Aug 09 20:03:55 netbox supervisord[53810]: 2018-08-09 20:03:55,914 INFO success: netbox-rqworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Aug 09 20:03:56 netbox supervisord[53810]: 2018-08-09 20:03:56,167 INFO exited: netbox-rqworker (exit status 1; not expected)
Aug 09 20:03:57 netbox supervisord[53810]: 2018-08-09 20:03:57,170 INFO spawned: 'netbox-rqworker' with pid 54135
Aug 09 20:03:58 netbox supervisord[53810]: 2018-08-09 20:03:58,171 INFO success: netbox-rqworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

So installing the requirements from pip3 from root@netbox actually works. Which means we can cut out a bunch of the things we tried but didn't succeed with and then we can blow it all away and check again.

WORKING STEPS

Here's the deployment, starting from a fresh install of Ubuntu 18.04.1 LTS as user malachite@netbox (we will have to su - root, but I try to avoid being there as much as possible).

sudo apt update ; sudo apt upgrade
sudo apt autoremove

sudo apt-get install -y postgresql libpq-dev
sudo -u postgres psql
CREATE DATABASE netbox;
CREATE USER netbox WITH PASSWORD 'XY,gTr6GP-Xy';
GRANT ALL PRIVILEGES ON DATABASE netbox TO netbox;
\q

sudo apt-get install -y python3 python3-dev python3-setuptools build-essential libxml2-dev libxslt1-dev libffi-dev graphviz libpq-dev libssl-dev zlib1g-dev python3-pip

wget https://github.com/digitalocean/netbox/archive/v2.4.2.tar.gz
sudo tar -xzf v2.4.2.tar.gz -C /opt/
cd /opt/
sudo ln -s netbox-2.4.2/ netbox

Use sudo su - root to switch to root in order to install the python modules we are going to need.

pip3 install -r /opt/netbox/requirements.txt
pip3 install gunicorn
exit

Back as malachite@netbox, continue:

cd /opt/netbox/netbox/netbox
sudo cp configuration.example.py configuration.py
python3 /opt/netbox/netbox/generate_secret_key.py > ~/secretkey

sudo vi configuration.py

configuration.py changes, using :read ~/secretkey to pull the secret key into the file:

ALLOWED_HOSTS = ['192.168.0.11']
DATABASE = {
    'NAME': 'netbox',
    'USER': 'netbox',
    'PASSWORD': 'XY,gTr6GP-Xy',
    'HOST': 'localhost',
    'PORT': '',
}

SECRET_KEY = 'J8g4+*aFfpV#2tGEPOSA%NriQx=Yvb_uWzKLy-U)0^w$Tl&5os'

rm ~/secretkey

cd /opt/netbox/netbox/
python3 manage.py migrate
python3 manage.py createsuperuser
sudo python3 manage.py collectstatic --no-input

sudo apt install -y nginx
sudo vi /etc/nginx/sites-available/netbox

/etc/nginx/sites-available/netbox contents:

server {
    listen 80;

    server_name 192.168.0.11;

    client_max_body_size 25m;

    location /static/ {
        alias /opt/netbox/netbox/static/;
    }

    location / {
        proxy_pass http://127.0.0.1:8001;
        proxy_set_header X-Forwarded-Host $server_name;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header P3P 'CP="ALL DSP COR PSAa PSDa OUR NOR ONL UNI COM NAV"';
    }
}

As malachite@netbox:

cd /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/netbox

sudo vi /opt/netbox/gunicorn_config.py

/opt/netbox/gunicorn_config.py contents:

command = '/usr/bin/gunicorn'
pythonpath = '/opt/netbox/netbox'
bind = '127.0.0.1:8001'
workers = 3
user = 'www-data'

Install Supervisor, to keep things running, I guess.

sudo apt install -y supervisor

sudo vi /etc/supervisor/conf.d/netbox.conf

/etc/supervisor/conf.d/netbox.conf contents:

[program:netbox]
command = gunicorn -c /opt/netbox/gunicorn_config.py netbox.wsgi
directory = /opt/netbox/netbox/
user = www-data

[program:netbox-rqworker]
command = python3 /opt/netbox/netbox/manage.py rqworker
directory = /opt/netbox/netbox/
user = www-data

Finally restart the service to get everything up and running.

service nginx restart
service supervisor restart

And now netbox should be running and available on the network!

Thoughts

The installation process was frustrating because I never log onto root@localhost unless I absolutely have to. This was the first instance I have encountered where I require things to be installed by root@localhost. So, things I've learned in summary:

• Most documentation seems to assume that you just exist as root@localhost all the time, which is an awful assumption and an awful practice.
• Python modules are installed into a user's home directory.
• Supervisor as a service is run by root@localhost, and the user set in the conf file is only changed to using setuid, meaning that the program still uses all of root@localhost's environment variables (including where python modules are).
• I misattributed the python module error to a permission error in accessing the folders. This was due to me assuming that the python modules were installed for everyone and not per user, and that Supervisor specifically ran programs as root@localhost. I found these things in the documentation.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

I wanted to build a web crawler in python to dive into pages and look for images. So of course the first thing I did was google it. After looking through several pages, I stumbled across this simple article. Well, that seems easy enough let's see if we can't build from it.

Here's the original code, for reference:

import requests
from bs4 import BeautifulSoup
def web(page,WebUrl):
    if(page>0):
        url = WebUrl
        code = requests.get(url)
        plain = code.text
        s = BeautifulSoup(plain, "html.parser")
        for link in s.findAll('a', {'class':'s-access-detail-page'}):
            tet = link.get('title')
            print(tet)
            tet_2 = link.get('href')
            print(tet_2)
web(1,'http://www.amazon.in/s/ref=s9_acss_bw_cts_VodooFS_T4_w?rh=i%3Aelectronics%2Cn%3A976419031%2Cn%3A%21976420031%2Cn%3A1389401031%2Cn%3A1389432031%2Cn%3A1805560031%2Cp_98%3A10440597031%2Cp_36%3A1500000-99999999&bbn=1805560031&rw_html_to_wsrp=1&pf_rd_m=A1K21FY43GMZF8&pf_rd_s=merchandised-search-3&pf_rd_r=2EKZMFFDEXJ5HE8RVV6E&pf_rd_t=101&pf_rd_p=c92c2f88-469b-4b56-936e-0e65f92eebac&pf_rd_i=1389432031')

Naturally, the first thing I did was pop open a file in Vi, pase the code in, and run it with python3. Naturally, it failed, which was the point. Specifically the error it returned cited a failure on the second line, meaning that I already had the module requests. All that I need is to get BeautifulSoup4, and it should work.

$ sudo apt-get install python3-pip
$ pip3 install beautifulsoup4

I changed the website to my own before I ran it this time. Mostly because it's okay if I break my website: web(1,'https://notawful.org'). And when I ran it...

malachite@localhost:~$ vi crawler_fail.py
malachite@localhost:~$ python3 crawler_fail.py
malachite@localhost:~$

Pros: No errors. Cons: No successes. Now is where I start breaking it down and figuring out how this little script works. The best place to start is probably the documentation, but I found it wholly unhelpful because I have not had the programming experience or patience with tiny words to read through and understand it. My strategy here was to fail fast, and itterate quickly. But first to read the code and approximate what it does.

Well, the entirety of the program is a function, which is cool because I understand those. We pass through page, which was set to 1, and WebUrl which was set to https://notawful.org.

def web(page,WebUrl):
    if(page>0):
        url = WebUrl
        code = requests.get(url)
        plain = code.text
        s = BeautifulSoup(plain, "html.parser")
        for link in s.findAll('a', {'class':'s-access-detail-page'}):
            tet = link.get('title')
            print(tet)
            tet_2 = link.get('href')
            print(tet_2)

Stepping through:

1. if page is greater than zero, do the rest. There is no itterator on page, so it will remain as what we set it as when we called web()
2. url contains the value of WebUrl. So this should be a string value of https://notawful.org
3. code contains the object resulting from requests.get(url). I do not know what kind of object this is, but my guess is that it did a get request to url and stored it in code
4. plain contains the result of code after running the object function/method text on it. I assume this converted whatever requests.get(url) to plaintext, probably the plaintext html.
5. s contains BeautifulSoup(plain, "html.parser"). I do know that BeautifulSoup turns HTML into objects that can be searched and operated on. I also know this statement is creating a BeautifulSoup object and storing it in s This however, I don't understand.
6. for each link, something we're just creating right now, in s.findAll('a',{'class':'s-access-detail-page'}) we are going to do a thing.

Alright, I can assume that steps one through five worked as intended, since it is just shifting around object values and if any of those operations were to fail I would get some kind of invalid type error. The next couple of steps operate on link, which will be a new object for each itteration of this link stepping through the values of s.findAll('a',{'class':'s-access-detail-page'}).

The smart thing to do at this point would have been to look at the documentation for what arguments findAll() took, and surely I am a smart person. Well, galaxy brain: I did not.

So, from link we are looking to get('title') and get('href'). These are going to be strings, because we use print() on them. link is an itterated object containing the results of findAll(), which is presumably limiting its search to objects containing both properties 'a' and {'class':'s-access-detail-page'}. Or, perhaps its looking for objects that have property 'a' that contains {'class':'s-access-detail-page'}. Either way, I opened the Chrome Developer console to see if I can spot anything.

<a class="post-card-content-link" href="/conference-travel-advice/">...</a>

Well then, that clears that up. s.findAll('a',{'class':'s-access-detail-page'}) is looking through s for something that is similar to this, except that instead of post-card-content-link it is looking for one that contains s-access-detail-page. I know this is the one that I am looking for because we printed `link.get('href'), which would be the value of href. It doesn't appear that this one contains a title field, so I'll remove that. So let's make some changes to that script.

Now we are running:

import requests
from bs4 import BeautifulSoup
def web(page,WebUrl):
    if(page>0):
        url = WebUrl
        code = requests.get(url)
        plain = code.text
        s = BeautifulSoup(plain, "html.parser")
        for link in s.findAll('a', {'class':'post-card-content-link'}):
            tet = link.get('href')
            print(tet)
web(1,'https://notawful.org')

And this is what we got back:

malachite@localhost:~$ python3 crawler_small.py
/college-university-guide/
/fixing-ghost-on-digitalocean/
/conference-travel-advice/
/gogs-install/
/online-presence-on-a-budget/
/sr-part2/
/sr-part1/
/strava/
/defcon-travel-advice/
/windows-pki/
/engineering-journals/
/why-i-hate-mathematicians/
/lessons-learned-october-2017/
/password-managers/
malachite@localhost:~$

So I hit the nail on the head with my initial instinct. We also know that we are pulling exactly the right tag in all of the html document because we see /conference-travel-advice/ in that list, which was the one I used as a reference.

Something else I noticed is that those hrefs contain a leading /, which means that we can concatenate the href and our WebUrl to get the full url of a child page. This gives us the ability to iterate and run a second loop to pull things from child pages.

To start opening up child pages, I copied the object creation so we could run findAll() on it and just added _low to the end of all the variables. I made sure that I printed the href value so that I could see what page the images I pulled were from. Here's a snippet from that:

for link in s.findAll('a',{'class':'post-card-content-link'}):
    tet = link.get('href')
    print("href " + tet)
    url_low = WebUrl + tet
    code_low = requests.get(url_low)
    plain_low = code_low.text
    s_low = BeautifulSoup(plain_low, "html.parser")

Now, before I ran this I needed to know what the tags for images looked like. The only page that I know has lots of images is /windows-pki/.

<img src="/content/images/2018/07/PKI-DomainMap.jpg" alt="PKI-DomainMap">

This makes it very easy. My s_low.findAll() is only going to have to look for img, and then I will have to print the value of src. So here's what the final code looks like:

import requests
from bs4 import BeautifulSoup
def web(page,WebUrl):
    if(page>0):
        url = WebUrl
        code = requests.get(url)
        plain = code.text
        s = BeautifulSoup(plain, "html.parser")
        for link in s.findAll('a',{'class':'post-card-content-link'}):
            tet = link.get('href')
            print("href " + tet)
            url_low = WebUrl + tet
            code_low = requests.get(url_low)
            plain_low = code_low.text
            s_low = BeautifulSoup(plain_low, "html.parser")
            for img in s_low.findAll('img'):
                tet_low = img.get('src')
                print(tet_low)

web(1,'https://notawful.org')

I ran it, and it worked, but there was a lot of the same image so I piped the results through uniq. Here's what those results looked like:

malachite@localhost:~$ python3 crawler.py | uniq
href /college-university-guide/
/content/images/2018/07/headshot.jpg
href /fixing-ghost-on-digitalocean/
https://assets.digitalocean.com/articles/pdocs/site/control-panel/networking/domains/no-domains-yet.png
https://assets.digitalocean.com/articles/putty_do_keys/new_ssh_key_prompt.png
https://assets.digitalocean.com/articles/pdocs/site/control-panel/droplets/n-choose-image.png
https://pbs.twimg.com/media/DaMLUoGXUAI21V6.jpg:large
/content/images/2018/07/headshot.jpg
href /conference-travel-advice/
/content/images/2018/07/headshot.jpg
...

From no crawler to a one-level image crawler in next to no time. I spent more time on iteration than I showed here. Most of that extra time was spent on getting python to work since I had never had to install pip before, and I installed the wrong pip about three times. I also tried a couple different ways to make web crawlers, but this one I decided was the easiest since I had to build all the moving pieces myself. Building all of tiny pieces myself made me learn how all the pieces fit together, which is how I learn best.

This is what failing fast means to me, particularly. I made one or two changes here and there, test it, watch where it fails, and look into what those failures mean. Repeat until it works as intended. All in all, even with the issues with pip and watnot this took me about an hour and a half to figure out without knowing anything about web page design or having seen any of these packages before.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Since I know several young people heading to college or university soon, I thought that I'd write a quick guide for things to do to manage things. This is based on my own experiences.

Personal Brand

I hate the phrase "personal brand", and I am only using it ironically here. You are at the point where you will want to start caring about how you present yourself online and what kind of online footprint you have. You are also going to want to start collecting stuff and curating a CV/Resume. So, here are some tips:

• Buy a professional-enough domain name, a cheap blog or web page, and email hosting under your domain name. This combo costs about $15-20/year for the domain name, and about $12-15 a month.
  • Do not use your professional email address to sign up to websites or games. Use it to communicate with your professors, and potential employers.
• Align your public social media handles with your domain name, or create new social media accounts that share your "brand name". This helps push all the things you want people to see about you to the top of the pile, and helps bury connections you'd rather people not make.
• Use your blog to practice your writing and communication and to collect your ideas and experiences. Take your time to make clear blog posts that you can come back to in order to re-learn things.
• Keep track of every major project that you do while you are in college and university. When it happened, how many people were involved, what did you do on the project, and what did you deliver. All of this can be added to your Resume/CV, so keep your records of this stuff up-to-date.
• Learn to check-in with people. Get people's contact information and check in on them once or twice a month if you don't see each other. Maintain the relationships you make at college and university and they will serve you well.

Security

This is a pretty important transition. You're an adult now, and as an adult you have things you want to protect.

1. You need to have a couple email addresses.
   • Your professional one is for emailing professors and doing business-related things. This is primarily because as your main account you want to avoid it being filled up by spam or notifications.
   • You will also need a management address, which you will use for managing your devices and your domain name, web, and mail hosting services. It is a good collection point for reciepts and such.
   • Your final address (one you likely already have) is for everything else. Signing up to games, websites, and the like.
2. Get a password manager, a security key, and a good two-factor-authentication application. Change all of your current passwords so that none of them are the same or versions of another (your password manager will help with this). This is the biggest thing you can do for personal account security.
   • Last I checked, keepass is still the go-to for free software-based password managers. But there are many good ones available, so do some research and figure out what ones meet your needs. I personally use a paid license for Dashlane as a backup for my hardware-based Mooltipass. My Mooltipass was invaluable for college since I was constantly moving from public computer to public computer, but your mileage may vary.
   • A security key is a hardware-based Two-Factor Authentication (2FA) token that will drastically reduce the likelihood of losing control of your accounts. There are a number of good security keys available. The updated Yubico Security Key is cheap and reliable. I suggest buying two of them, and keeping one in a safe space for backup.
   • Google Authenticator, Authy, and Duo are all solid 2FA applications. I use Authy because it keeps a backup of all my 2FA tokens because I have changed or reflashed my phone multiple times.
3. Enable 2FA on all accounts possible. Even if someone gets a password for one of your accounts, 2FA will go a long ways to prevent people from getting access.
   • SMS (text message) based 2FA can be compromised relatively easily, or phished fairly easily. If you can enable other forms of 2FA then do so and try to avoid letting SMS be a "backup" 2FA method.
   • If you use an app authenticator (Google Authenticaor, Authy, etc.) you will often recieve a list of back-up codes. Add these to your password manager's entry for those accounts in case you lose access to your authenticator.
4. If you use Chrome, run the uBlock Origin, Privacy Badger, and HTTPS Everywhere extensions. Or run Brave. There are similar extensions in Firefox. Ads and tracking are a security problem. Just do it and don't feel bad about lost ad revenue, try to support the people who make content you enjoy directly whenever possible.

College/University Life

This is where I am going to give some probably shady advice, but I understand the struggle so listen closely. Most of academia is an awful money-grab and it's all bullshit. Here's some advice that I give as someone who has been through it.

Textbooks

College textbooks are absolute bullshit. They are not expensive because they are difficult to produce or because they cost a lot to put together, they are expensive so that they can get a lot of money out of students. What do you do about it?

1. Find out what your book list is as soon as possible. Your college/university bookstore will have this information when it is available.
2. Separate that list into two categories:
   • Plain old textbooks (the more the better)
   • Textbooks with special access codes for online material
3. Check with the bookstore if they can or do order just the codes for those textbooks that come with online codes. The codes are often cheaper alone than with the book.
4. Get together with a bunch of your classmates and pool your money for one of each plain text book. Search them out and talk to them if you can. If you can't, don't buy your books until after the first week. You can take the text book to a print shop, have them shear off the binding and scan the entire book into PDF form for a fee. This will save the whole class hundreds of dollars if they are up for using a PDF.
5. If not enough people want to do it, look for PDFs of your textbooks. Just be a bit wary. Not paranoid, but exercise caution.

Keeping Notes

In my personal experience, making notes during lectures helps me far less than listening and engaging in class discussion (if that's a thing your professors enjoy). If making notes during class does help you, then go for it. But whatever you do, write all your notes in your own words. Do not copy explanations from wikipedia or from your professors' slides. Writing things in your own words makes them stick better, and makes it easier for reference later on.

Keep an engineering journal, preferably in a nice paper notebook, but an online copy is good too. What you do is whenever you encounter a difficult problem you track the symptoms/circumstances, what your troubleshooting steps are, and how you solved things. Then you record the symptoms and solutions in your engineering journal. Also when you have a large concept that you know is important, do some experiments and exploration of those concepts and explain in your own words that concept in your engineering journal. Write all of these so that if you handed your engineering journal to someone with very little background knowledge can use it to solve a problem or understand a concept because you will forget those important details.

Between your blog and engineering journal you will amass a significant knowledge base that you can tap into. Your goal with both of these is to ensure that you never have to come up with a solution to the same problem twice. Reference your previous solution first, and see if that works.

Taking Tests

I have a good history with tests. I have a bad record with history tests. There's a reason for that. For the most part, taking tests is more about knowing how to take a test than it is about learning the material. Here's my advice on studying for tests and how to work through tests.

Try not to study or cram the night before or just before the test. Study a couple days before the test and get a good restful sleep before the test. This pushes all the stuff you've studied and researched into long-term memory which is less likely to bail on you when you get handed the exam.

I have friends who have some other tricks for tests. They do memory association stuff like eating an apple while they study for a test and then eating an apple right before the test. I am not sure how much it helps, but you might find those tricks work for you.

The test is in front of you. You have limited time and the test is long. Remember what the goal of an exam is: To accumulate as many points as you can in the time you have. There is a process here that you will want to learn and follow on the tests you know you'll struggle on. On all of my college tests, the questions and tests were all marked with how much they were worth.

1. Look at how much the test is worth, and quickly figure out what a passing grade is. Exam out of 75, a 55% is a passing grade, you need 38 marks to pass. You want to get to 38 marks as fast as possible
2. Read through the test and look at each category (multiple choice, true/false, short answers, long answer, essay, case study). You are looking for the easiest questions first.
3. Do your first pass on the test. If you hesitate on a question move on immediately. If you read the question and are immediately certain of an answer then write it down, it's probably correct. You shouldn't be spending any time thinking on your first pass. Make a small note of how many questions you answered/marks you got, we will assume you got them all correct.
4. Check how many more marks you need in order to pass and how long you have left. You should still have most of your exam period left, and you will see there isn't much left before you pass the test or that you've already passed. If you feel it's still going to struggle, don't worry.
5. If you have thought of the answer to a question you didn't do on your first pass then go back and do it now. This was the point of taking the moment to assess how much farther you needed to go. It gave you a moment to relax and think.
6. Add the new marks to your total and see how close you are to passing. You have done all the easy questions or problems at this point. Take another read through the test and assess how much work answering each question you know you can answer is going to be.
7. Work your way through the easiest/fastest questions first and the harder/slower ones last. If you get stuck, move onto another question. It's better to have half an answer for a question than no answer since you can get partial marks most of the time. Add finished questions to your running total. By the time you have reached this stage, chances are you will have passed the test with lots of time to spare, so you can take your time and relax.

General life advice

I recognize that many people have different circumstances, and some of this advice will be more difficult for some people than others.

• Get a solid rest each night. Eat three meals a day. Drink lots of water throughout the day. Try to get some exercise in. It's simple shit but it will help you more than anything else will.
• Budget your time. You will have to make compromises, but you need to balance downtime with getting your work done, and if something has to give never let it be your rest.
• Associate with good people, not smart people. Often those two groups will overlap, but don't associate with people who are toxic or talk down about their peers. Those people are more trouble than they're worth. College and university get easier when you have good friends.
• Be a good person. Not helpful, right? What I mean by "be a good person" is that if you see someone struggling, help them out. If you have privilege, use it to help others by telling your fellow privileged people when their behavior is unacceptable. Don't let people speak over voices that don't get heard enough (read: tell white people to be quiet).
  • When someone tells you that you've made a mistake, listen to them. You don't get to decide that you didn't hurt someone. Most minority groups bear the weight of being discriminated against and having to teach others how not to, so don't make them take that. Do your own research on it and learn to be better. It's work, but also worth it.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Hello! I recently moved my blog (you're reading it!) from the official Ghost hosting to DigitalOcean. I can spend a moment talking about why I made the transition: It's hella fucking cheaper and way easier now. That was quick, so I guess now is where I talk about how I actually did it.

Strangely, it went quite according to the documentation. I signed up for a DigitalOcean account, and added my domain name first before I did anything. There was an option to simply add all the gmail mail servers, so I did that as well.

I had to boot up my chromebook's chroot and install the Windows Subsystem for Linux (WSL) on my desktop because I needed to make SSH keys to add to DigitalOcean.

Making the keys was easy, done and done. I added a password to the keys as well because it was important to do, right? After I made them I uploaded them through the dashboard.

ssh-keygen -t rsa -b 4096 ~/.ssh/id_rsa

Now I have some keys I can attach droplets at creation so that the root account never has to have a password. Now I just need the droplets. So that goes by pretty quick too. The creation-process is very quick. In the image below, I selected 'One-click apps' and selected Ghost. I also selected the cheapest droplet ($5/mo) because I know that my blog doesn't take much to run.

I had to wait for my droplet to spin up and get assigned an IP address. At this point is where this blog post becomes a work of fiction because I ran into a host of tiny issues (ones partially of my creation and partially the universe's). I'll go over them because they are all really interesting little things.

I wanted to make SSH config files so I would be able to just jump straight into my droplet. You can do it with whatever editor you want, but this is what mine ended up being.

Host ghost
    Hostname <ip>
    User root
    Port 22
    Identityfile ~/.ssh/id_rsa

I ran into issues with my config file on my Windows computer. This is what happened when I tried to ssh ghost:

malachite@localhost:~$ ssh ghost
Bad owner or permissions on /home/malachite/.ssh/config
malachite@localhost:~$ ls -alh .ssh
total 8.0K
drwx------ 1 malachite malachite  512 Jul 31 18:40 .
drwxr-xr-x 1 malachite malachite  512 Jul 31 18:40 ..
-rw-rw-rw- 1 malachite malachite   99 Jul 31 18:40 config
-rw------- 1 malachite malachite  222 Jul 31 16:42 known_hosts
...

Do you see the problem? Because I sure as hell didn't until I googled "bad owner or permissions" and came across this gist. As it turns out, ssh wants your config file to have very specific permissions. As a quick primer for how permissions in unix work, here's a comic from Julia Evans that does a much better job that I can in the space I have.

According to the gist, linked previously, the proper permissions for config should be 644, which translates to -rw-r--r--. The owner can read/write, and everyone else can just read it. This is quickly solved with a chmod 644 .ssh/config. With the permission fixed I can just pop into my droplet with a quick ssh ghost.

I followed the configuration steps from the community tutorials. It did not work. I ran into issues that no matter what I did I could not access the site with my domain name, but could on either http or https with the plain IP address. Specifically I was getting a 504 Bad Gateway error when I attempted to access with my domain name, which told me that it was something to do with the nginx configuration and not DNS. Clearly I am reaching my server, but my server can't handle the request.

When I initially loaded my droplet and signed into it, there was a message on screen that said that it was setting things up and that it would update Ghost if updates were available. I happen to recall that the DigitalOcean droplet comes installed with Ghost 1.24.9, but it updated to Ghost 1.25.3.

Semantic Versioning is a version number system that contains three positions. The first is MAJOR version, the second is MINOR version, and the third is PATCh version.

1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards-compatible manner, and
3. PATCH version when you make backwards-compatible bug fixes.

This had nothing to do with the issue, but made me consider the possibility that the update broke a dependency. So I updated stuff and it worked. Here's the steps to avoid encountering this.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get autoremove
sudo -i -u ghost-mgr
cd /var/www/ghost
ghost update
ghost restart

ghost update shouldn't do anything, but the ghost restart is really important because it reinitializes things. The upgrade and restart did the trick and my droplet was back online.

Okay, it's running, anything else?

Well part of the reason I wanted to move to self-hosting wasn't just because it was cheaper but because I could also use it to host files. I wanted to host my resume on my server so people could just click a link and get it right there in front of them.

From my Windows PC:

scp /mnt/c/Users/malachite/Desktop/resume.pdf ghost:/var/www/ghost/content/images/2018/07/devon_taylor_resume.pdf
ssh ghost
sudo chown ghost:ghost /var/www/ghost/content/images/2018/07/devon_taylor_resume.pdf
sudo chmod 644 /var/www/ghost/content/images/2018/07/devon_taylor_resume.pdf

I knew that since I was going to be copying it into the content folder as root, I needed to change it to the owner. All the files in /var/www/ghost/content is owned by ghost:ghost, so I logged into the server to give them permission of the file. I also had to change the permissions since when the file was copied in it came with some wonky permissions that made the pdf an executable.

Follow-up

While I was writing this post Ghost crashed causing it to return 404 errors, right around the time I was saying that everything was working beautifully and I solved all the problems. I was mostly right.

Edit: The 404 errors were caused by Chrome sporadically choosing to use an old DNS record that targeted my website at a previous IP address where it is no longer located. After struggling with the developer tools in Chrome a little I managed to force it to ignore its cache and do a fresh lookup. So I configured things correctly the first time, Chrome was just gaslighting me.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

This is an update to my DEF CON travel advice from 2017. I have made some major changes and separated DEF CON and InfoSec specific advice from general conference advice.

Also check out:

If you want advice from someone who is a more seasoned traveler about traveling as an introvert then check out Lesley Carhart's blog post "The Infosec Introvert Travel Blog."

General Travel Advice

Packing

• Never check bags if you can avoid it. This prevents loss, theft, or mishandling. If you check bags then keep all your valuables in your carry-on.
• Pack light; leave room for treasure. If you plan on collecting lots of treasure then pack an ultralight duffel in your carry-on. They pack small and you can check it on the trip home.
• Your personal item should be a cross-body bag or backpack. The 'personal item' is the one that goes under the seat in front of you on a plane. Keep your valuables and electronics there.
• Your carry-on item should be a frameless soft-bodied item. Pack your clothes and toiletries in this bag. Since you will be the only one handling it, you do not have to worry about it getting roughed up.
• Smaller airlines have tighter carry-on requirements and hard-shell and rolling luggage can be refused. You can squish soft-bodied luggage to fit.
• Check out other people's conference packing lists. They will give you an idea of what you should bring with you.
  • It is a good idea to pack a battery pack. I believe the largest the TSA allows is a ~20,000mAh pack. Between long distances between outlets and fake/additional cell towers in the area, your phone battery will drain faster than you expect.

Laundry

You can get away with packing less if you know how to do laundry in the sink of your hotel room. You just need a large enough sink and your shampoo. Someone I met at DEF CON was nice enough to teach me how, and I am passing that on to you.

1. Half-fill sink with water. Put a couple drops of shampoo in the water.
2. Swish clothes in soapy sink-water.
3. Rinse clothes.
4. Repeat steps 2 and 3.
5. Wring clothes out as best you can.
6. Roll up the clothes in a hotel towel then stomp the towel. This will soak up additional moisture.
7. Hang to dry.

Cotton is comfortable because it is soft and wicks moisture, but it holds moisture and becomes odorous. Cotton clothes also take forever to dry even in well-ventilated areas. Synthetic fibers don't absorb moisture as well but dry quickly and are less prone to stinking. If you intend on doing your laundry in your sink then pack a clothesline made of medical rubber that ends with hooks or carabiners. Braided rubber grips clothes without pins, and suction cups will fail you. Once you get the routine down doing your laundry takes a couple of minutes, meaning you can pack lighter.

Often overlooked fact: Odor is caused by bacteria, which thrives in moist environments. Even if you shower and scrub every day, if people are complaining about B.O. it is your clothes. Doing your laundry is important.

Air Travel

• Print your boarding passes. Unless you travel with a battery pack and can ensure your phone is always available do not rely solely on electronic boarding passes.
• In my experience, airport security has shifted focus towards getting people through as quickly as possible. Shoes on, belt on. Sometimes stuff goes through x-ray sans bin. This is not always the case.
• RFID-blocking wallets will set off the metal detector. Even if the agents say it won't. Put it in the bin.
• Customs is all electronic now. Stop at a terminal, follow the prompts, give your receipt to the person at the exit. Quick, quick.
• If you drink lots of water like I do you will have to dump your water into a garbage bin at every security checkpoint. This goes for all beverages. Dump it out before you're at the front of the line.
• If you're traveling from Canada you will likely go through TSA pre-clearance which is just US customs/border but before you take off rather than land. No special rules/precautions.

DEF CON / InfoSec Specific

• If asked about your lockpicks: refer to lockpicking as lock-sport and always have practice locks in the same bag as the lockpicks. If you bring intrusion tools pack them in checked baggage if possible.

Conferences

Planning

• Book everything as far ahead in advance as possible. Hotels charge for the first night when you make your reservation but everything else gets charged when you check-in. Booking ahead is always cheaper.
• Do not do mobile check-in if any member of your party is under the age of 21. It will cause headaches. Just check-in at the desk.
• Your trip is going to be more expensive than you expect. Bring a bunch of cash but don't be seen with it, only keep enough in your wallet.

DEF CON / InfoSec Specific

• Never, ever, do BlackHatUSA on your own dime. This advice comes from someone who can't afford it.
• If you have mobility or anxiety issues, book at Ceasers for DEF CON, and if you can manage it try to get in on the floors that have a direct elevator to the conference areas. This will make your life substantially easier.
  • If you have mobility issues this lets you skip the casino floor and the mess everyone else goes through to get to the conference area. Also take some time on Thursday to find the elevators in the conference area so you don't have to find the signs while the halls are crowded with people.
  • If you have anxiety issues, booking at the Ceasers in any tower gives you a quick and free way to get away from the conference area to your own quiet space so you can recharge.
• Most people's first year at DEF CON is spent learning how DEF CON works and where to spend their time. Don't get discouraged because you feel you didn't do enough your first year.
• The villages at DEF CON are closed on Thursday for set-up and close early on Sunday to pack up. As such the crowds are thin on these days and the lines are shorter. Take Thursday to wander around with a DEF CON map in hand and find where things will be so you don't have to do it with the crowds.

Execution

Interacting with People

• Ask for consent before taking people's picture or touching them for any reason. It makes a huge difference, even if you know the person really well.
  • Making eye contact and doing the open-arms-hug? gesture and asking, "Hug?" is easy to execute and isn't awkward. Respect their decision if they say no.
• Bring business cards with basic contact information: name/alias, social media, business email. Asking for someone else's contact information can make them uncomfortable. By offering your contact information instead you give them control of the next interaction.
• If you are better known by your online handle than your real name include your handle in your introduction. "Hi, I'm Amanda, also @AwfulyPrideful on Twitter."

General Conference Advice

• Most important tip of all: Don't push yourself. If you get anxious and can't be on the conference floor for long periods, just duck out and recharge. Under no circumstance should you force yourself to stay at the conference all day. You will exhaust yourself and ruin the experience if you do.
• Stay hydrated, bring snacks. You will have a better experience this way, I promise you. My personal choices are tap water (yuck, but free) and CLIF bars (trail food, keeps energy up sans meal).
  • Staying hydrated means drink small amounts of water every 15-20 minutes. Drinking lots of water all at once will not hydrate you since your body will only absorb a bit of it at once.
• Con Flu is real. Wash your hands often, or carry around hand sanitizer and use it often. The alternative is to not touch anything, but that won't save you.
  • Good strategy is that if you are about to touch your face, eyes, or nose, stop and use hand sanitizer or wash your hands with soap and water.
• Wear good walking shoes. You will be on your feet most of the time, either walking around or standing in lines. Get good insoles / inserts for your shoes. Your feet will thank you for it.
• Carry lots of cash for tips. Tipping your service people is really important, and people attending conferences are known not to tip well. Also thank them and be patient because it's really busy and hard for them too.
  • If you like to drink, drop a twenty as a tip for your first drink. That bartender will be your friend all night and you will get served when you go to them.
• Easy advice for working out how much to tip:

  #GoodDefconAdvice Good rule of thumb for tips: Move the decimal one place over to the left, then double that number. that is 20% tip. Add half of that number to reach 15% (e.g. 60.00 = 6.00 * 2 = 12.00 tip. 6.00 * 1.5 = 9.00)

  — SummerOfSYN (@da_667) July 28, 2018

• If you are at a bar or party, always order your own drink and keep your eyes on it. Stick with friends. Look after each other.

DEF CON / InfoSec Specific

• Never leave a good conversation to see a talk. That is, unless your employer sent you and you need to report on some talks to justify the expenses. Face-to-face networking is a huge deal, so don't pass up those opportunities.
• Hotels partnered with DEF CON stream the talks to hotel TVs. Some people host small viewing parties for talks. If you are invited to one by people you know, you should go. It is good for networking and you can ask people about content you don't understand.
• Taking pictures of the crowd is against DEF CON's privacy policy. People are there to make memories, not evidence. Don't be the asshole with the GoPro strapped to your stupid forehead.
• Don't interrogate people. If they're dodging where they work or what they do, leave them be. Remember where you are, and remember that they don't know you.
• If you're gossiping or telling stories, don't name drop unnecessarily. People just don't appreciate it. You also seem more mysterious if you have secrets or "know someone."
• LINECON is real. Start lining up at least half an hour before a talk is scheduled to start. People in line in front and behind you are there for the same reason you are and you don't have to lose networking time. This is one of the easiest ways to meet new people since you already have something to talk about.
• HALLCON is where it's at. Go to a chill area and ask if a seat's open at a table or unofficial-circle-of-people-on-the-floor. Listen to people's conversations and ask some questions. Don't intrude/eavesdrop on clearly personal conversations. You can enter a lot of conversations just by proximity.
• If you plan on getting swag from the Swag Shop at DEF CON, do it on the first day. In 2017, when I attended, the line on the first day snaked through the hallway, and through the entirety of a large conference room. Swag sells out fast, so get in line right after you get your badge.
  • A good way to make friends is to get in line, and then tell your friends that you'll pick up their swag for them. Have them meet you in line (or before you get in line) and give you cash and a list of stuff they want.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Hey there. I have been working on getting an installation of Gogs running. I want to get a git instance up that I control so that I can dig into how it works without that showing up on github. Controlling it also means that I can make as many private repositories as I want without worry. So, with all that said here was my adventure in getting this done.

Hyper-V Virtual Machine (Gen 2):

• 1 vProcessor
• 1024 MB RAM (Dynamic Memory)
• 25 GB VHDX (on an SSD)
• Ubuntu 18.04 Server

First Pass (Install + HTTPS)

I followed and spliced together a number of guides. My first attempts were to install Gogs from source, but I ran into issues with Go. Mostly that I could not for the life of me get it to recognize where Go was installed. Apparently I am not the only one who has had this problem. After a couple hours of trying to get that to work I moved on to installing Gogs from the binary.

Due to a confluence of r00k's Hack the Box walkthrough on Aragog, and several guides I was reading, I learned that git services are typically installed on a user named git and also that they typically do not have a shell or login privileges. So that's where I started on this.

After I installed Ubuntu and got to my sudo/root user notawful, I ran sudo apt-get update and sudo apt-get upgrade, followed by sudo apt-get autoremove. Once all that was done I added git as a user, then entered it to download and extract the Gogs binary.

sudo adduser --disabled-login -gecos 'Gogs' git
sudo su - git
wget https://dl.gogs.io/0.11.53/gogs_0.11.53_linux_amd64.tar.gz
tar -xzf gogs_0.11.53_linux_amd64.tar.gz

This created a new directory in git's home directory, /home/git/gogs, which contains all the files for Gogs to run. That is easy enough to do. From /home/git/gogs run ./gogs web. It runs correctly, and now I can finish configuration from the web interface. At this point my only worry is about an internet-based deployment since it is HTTP only at this point.

From a web browser on the Gogs installation screen I do the following:

• Choose SQLLite3 since the binary includes this by default and there is no additional configuration required.
• Add admin account gogsadmin with password r_3=zBd_aQH{. After enabling HTTPS I plan on adding a new admin user account and removing this one.

Now I exit/stop Gogs and exit back to notawful. Now I need to create some certificates. If I was running this in production I would create actual web certificates using Let's Encrypt, probably. Instead I will make a couple certificates using openssl. I use 192.168.229.99 as the domain when making the certificate because I don't plan on setting up a DNS for my test environment.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/gogs-self.key -out /etc/ssl/private/gogs-self.crt
sudo chown git:git /etc/ssl/private/gogs-self.key
sudo chown git:git /etc/ssl/private/gogs-self.crt

I thought I knew what I was doing when I changed the owner of the key and certificate. My linux was rusty and when I made changes to the Gogs configuration file, it couldn't fetch them. I'll get to that in a second.

I made the following changes under the [server] heading in ~/gogs/custom/conf/app.ini from user git. There are a bunch of other entries in this section, but these are the only ones that I played with.

[server]
PROTOCOL = https
DOMAIN = 192.168.229.99
ROOT_URL = https://192.168.229.99:3000
CERTFILE = /etc/ssl/private/gogs-self.crt
KEYFILE = /etc/ssl/private/gogs-self.key
LANDING_PAGE = explore

I imagine that some people already know what error I was about to run into. When I ran /home/git/gogs/gogs web it came back with an error that did not show on the on-screen output. I had to go to /home/git/gogs/log/gogs.log and check.

Yeah. So I know the files are owned by the right account, because I did that earlier. So I just needed to move them over to somewhere that git could actually see without sudo privileges. So I went back to user notawful.

sudo mv /etc/ssl/private/gogs-self.key /home/git/gogs-self.key
sudo mv /etc/ssl/private/gogs-self.crt /home/git/gogs-self.crt

Now, from git I reflect those changes in /home/git/gogs/custom/conf/app.ini and run /home/git/gogs/gogs web again.

[server]
CERTFILE = /etc/ssl/private/gogs-self.crt
KEYFILE = /etc/ssl/private/gogs-self.key

It works! Except I get an error that does not appear in gogs.log, but in the on-screen stuff. I did not read the whole error because I saw the magical phrase tls handshake and knew exactly what the error was. I confirmed in my browser that I was trying to navigate to 192.168.229.99:3000, without specifying http:// or https://. Since I set the server to only operate with https, it was denying the request because it did not look the start of a TLS handshake. I instead navigated to https://192.168.229.99:3000 and was greeted by...

Okay, that's fair and something I should have expected by making my own certificate. I was able to click pas it and get into an HTTPS instance of Gogs. From there I created a new admin user and deleted the previous one that I created over HTTP. I haven't analyzed the traffic in the HTTP session but I don't trust it, so this is the mitigation I went with.

So, it works over HTTPS, and does not work over HTTP. If I had a DNS server I would have it redircet any HTTP requests to HTTPS to this instance. I will get to play around with git now that this is set up, but I still need to get this set up so I don't have to manually start the service every time.

Performance

Now, because I plan on running this in a DigitalOcean Droplet that will also contain an instance of by blog, I am a little worried about memory usage. Checking with Hyper-V I saw that the VM had 694MB assigned to the VM, but only 582MB was in use with Gogs running and being logged into the web service.

So then, how much was Gogs actually using? I stopped Gogs and exited git's shell. Logged only into notawful, the VM's memory usage was a 485MB, which means that Gogs was using about 97MB while running with one user. On a second test, I ran /home/git/gogs/gogs web & and then logged out of both git and notawful. Memory usage in this state was at about 563MB.

I have been told that Gogs memory footprint does not increase significantly with more users or more repositories. So, I think under reasonable circumstances I am likely to be able to run Ghost and Gogs on the same droplet with only 1GB of memory.

Second Pass (the same as the last, but with less fuck-ups)

As notawful:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get autoremove
sudo adduser --disabled-login git
sudo su - git

As git:

wget https://dl.gogs.io/0.11.53/gogs_0.11.53_linux_amd64.tar.gz
tar -xzf gogs_0.11.53_linux_amd64.tar.gz
cd gogs
./ gogs web

Connect to web site, select SQLLite3, create an admin user gogsadmin. Email doesn't matter, since Mailer service is disabled (and won't be enabled).

As notawful:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /home/git/gogs-self.key -out /home/git/gogs-self.crt
sudo chown git:git /home/git/gogs-self.key
sudo chown git:git /home/git/gogs-self.crt

As git, editing /home/git/gogs/custom/conf/app.ini:

[server]
PROTOCOL = https
CERTFILE = /home/git/gogs-self.crt
KEYFILE = /home/git/gogs-self.key
LANDING_PAGE = explore

As git from /home/git/gogs, run: ./gogs web &
From browser, log in to gogsadmin, create a new user and give them admin privileges. Then log into the new admin account and delete the gogsadmin.

Next Steps?

Figure out how to make this into an auto-run when the server starts. This is not something that I have done in a long time, and I have certainly never done so for a user that shouldn't be logging in. I will have to take some time to figure this side of things out.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

I have been thinking about this for a bit now, so I thought I would share. A year ago when I set up my blog and professional email address, tools were a bit limited. There was a lot of it was do-it-yourself, which can be rather annoying and intimidating when you do not have years of experience under your belt. Some tools have been streamlined since I started, so without further ado:

Here's a quick guide for getting a professional-enough online presence for less than your monthly Netflix subscription ($15/mo).

What do I need?

This is an easy list, actually.

1. A domain name
2. A web page
3. An email address under your domain name

That's three whole things. The goal here is really to get the email address attached to a custom domain name, but in my experience with G Suite you need to put some code in your web page's header briefly to prove ownership over the domain name.

What are we going to do?

This is another easy list!

1. Create an email address for management of your online presence
2. Purchase a domain name
3. Create a blog on a virtual private server (VPS)
4. Create a professional email address
5. Host our resume/CV on our VPS

First step is creating an email address for management. You can use Gmail or Outlook for this. Since our domain name and our blog is important, because it represents us, we are not going to use this email address for anything outside of paying for our domain name, our VPS, and our email hosting service.

Purchase a domain name

Take some time to think about what you want your domain name to be. It's important because you are going to be pretty locked into it once you start using it professionally.

Everyone has a different favorite domain name provider. I won't suggest any, since they all have different prices for different Top Level Domains (TLDs, examples: .com, .ca,. org). Shop around for the domain name you want and see where you can get it cheapest. Most places will offer WHOIS protection nowadays, and I know namecheap offers that for free since GDPR.

Domain names are purchased/renewed yearly, but can be cheap — from $10-$25/year.

Create an account, buy the domain name you want, and then we're ready to get our blog up and running.

Create a web page/blog

I was going to shy away from suggesting specific products, but DigitalOcean has really kicked it out of the park since I last checked. They now have a number of one-click installation templates ready, and two are of particular use for us.

You can get a cheap VPS, pre-installed with Ghost or Wordpress, for $5/month. You will have to do some configuration, but Digitalocean has some good guides associated with their different one-click installs to help you get set up. The updated Ghost One-Click Application guide is particularly good as well.

When you sign up to DigitalOcean you should use the email address you created for managing this stuff. Find the installation guide for whichever one you choose and follow the instructions. DigitalOcean's linux guides are all top-notch for simple administration, and their guides will get you set up with secured SSH access to your VPS and getting an SSL/TLS certificate for your web page.

My personal advice is to use DigitalOcean's Ghost installation. Ghost's interface is pleasant to work with and the installation is easy to update. I wish any of the products mentioned were paying me to promote them.

Get email hosting

So, there are a bunch of email hosting options and they are all quite cheap. I use G Suite, but Microsoft's Exchange Online is also quite good.

If you don't want to be tethered to a large corporation you can also use ProtonMail. ProtonMail has suffered from DDOS attacks, costs a little more (4€ vs $5 USD) a month, and gives you less storage space. They do encrypt your mail and give you the ability to send encrypted emails to other people. If you are okay trading availability for privacy, this would be a good option.

In each case you will have to prove that you are the owner of your domain. G Suite does this by giving you something to put in your website header temporarily, and I imagine other providers do the same. Once you validate your domain name they will give you their mail servers which you will have to add to your DNS records so mail can reach you.

Host your resume/CV

First, read Women in Tech by Tarah Wheeler. Then, read it a couple more times just to be sure. The chapter on making a resume suited for applying to tech jobs is particularly good.

Now all you need to do after making a PDF of your resume is to put it where your web server hosts content (like images and so on). Get the web path to your PDF and put it in a post/page on your blog next to your contact information. Keep your resume updated and in that same spot.

Something else you will probably want to do is set up IP address logging for your resume or other files you are hosting on your web server. You can often associate IP addresses directly with businesses or organizations who are looking at your files. In case you wanted to know when government agencies are reading your draft guide to building virtual machine labs or something.

Now what?

To be perfectly honest, I am still working on the "what next" part of this whole process. Do cool projects, write posts about them, and post them to your web page. Make some business cards and put your cool new website on it. Panic realizing you are probably paying more to watch anime online than to be professional on the internet. Post some cool recipes since cooking is the new cool thing for people working in technology to talk about doing now.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

In the last post I talked about the core resolution mechanics in Shadowrun Fifth Edition. At the end of that post I talked about three questions that are useful for designing games.

1. What is your game about?
2. How is your game about that?
3. How does your game reward players for engaging the things your game is about?

When you are designing a game it helps to answer those questions in order. Each question leads into the next and together they focus your game's mechanisms to deliver on the specific experience that you want players to have. Those questions can be generalized and applied to the design of almost any system.

1. What is this system intended to do?
2. How does this system work to deliver that?
3. How does the system incentivize users to engage in the system as intended?

Any system that a person can interact with is made up of intent, mechanisms, and incentives. A system's intent can also be interpreted as the goals or results of the system. Designers sometimes misrepresent a system's intent, or more often have a one-sided view of the results of that system.

We are going to figure out what Shadowrun is about by working backwards. We are going to examine Shadowrun's rewards and the mechanisms that players have to pursue them, and we are going to see what Shadowrun is about.

RETURN REWARDS;

Rewards and incentives are perhaps the most interesting element of system design. The rewards that you offer players contextualize the mechanisms that they have been presented with. This interaction occurs in real world systems as well as game systems. Before we dig into Shadowrun I am going to talk about a couple of other game systems.

>ENTER DUNGEON

Adam Koebel's multi-award winning tabletop roleplaying game Dungeon World featuring Sage LaTorra on drums^[1] is a game about going on adventuring with people you are at least loosely associated with, exploring the world, killing monsters, and collecting treasure. All of the moves Dungeon World provides are focused on enabling player characters to do those things. Dungeon World rewards players for this at the end of a session.

When you reach the end of a session, choose one your bonds that you feel is resolved (completely explored, no longer relevant, or otherwise). Ask the player of the character you have the bond with if they agree. If they do, mark XP and write a new bond with whomever you wish.

Once bonds have been updated look at your alignment. If you fulfilled that alignment at least once this session, mark XP. Then answer these three questions as a group:

• Did we learn something new and important about the world?
• Did we overcome a notable monster or enemy?
• Did we loot a memorable treasure?

For each “yes” answer everyone marks XP.

At the end of every session of play, players have the opportunity to earn up to five experience. Each point of experience lines up with something that one of the game's designers has stated the game is about. A player also gets to mark experience when they fail a roll.

Each move will tell you what happens on a 10+ and a 7–9. Most moves won’t say what happens on a 6-, that’s up to The GM but you also always mark XP.

This point of experience lessens the blow of failure. When you make a move you either succeed, partially succeed, or fail and get experience. This encourages players take narratively exciting risks because there is always an incentive to act. When you engage in the Dungeon World's themes and mechanics your character gets better.

>USE BLADE ON DARKNESS

Blades in the Dark is a tabletop role-playing game about a crew of daring scoundrels seeking their fortunes on the haunted streets of an industrial-fantasy city. That line come directly the cover of the book and on the game's web page. Like Dungeon World, Blades in the Dark rewards players for engaging the game's themes.

Players are rewarded for making desperate (read: exciting) actions, for expressing their beliefs, heritage or background, and for struggling with their vices or traumas. The player's crew advances for contending with challenges above their station, bolstering their reputation, and expressing inner conflict or the essential nature of the crew. Each playbook and crew type has a personalized experience trigger to incentivize players towards specific things. Cutters, who are fighters and leaders, get experience for addressing problems with violence or coercion. Hawkers, crews of drug dealers and vice purveyors, get experience for procuring better product and for pulling in richer clients.

There are only twelve actions that a player can make such as Attune, Consort, Prowl, Skirmish, and Wreck. These are deliberate choices that restrict the ways that characters can approach problems. The name of each action was chosen to invoke a specific tone to shape how players think. Every problem looks like a corpse-to-be when you have four dice in Skirmish. Cutters will lean towards Skirmishing with foes over attempting to Consort with them because picking a fight will earn them experience.

RETURN KARMA;

Dungeon World and Blades in the Dark align the tools players have with rewards for engaging in the themes and mechanisms of the game.

Shadowrun rewards player characters for completing shadowruns. This is the entirety of Shadowrun Fifth Edition's reward structure: Do the job. Characters are awarded currency and Karma.

Characters are paid a base amount plus some extra for each net hit on an opposed Negotiation test. The base is then multiplied based on a number of factors involved in the run. Examples of these factors include the largest dice pool rolled against a player and how outnumbered the players were in combat. On a list of about eight possible modifiers only one is affected by decisions that the players make: Runners accomplished the task with impressive speed and/or subtlety.

Karma is Shadowrun's experience analog and is used for character progression. Karma is spent on things such as learning new spells and advancing skill ratings. Characters receive Karma at the end of a run for surviving the run, for completing some or all of the objectives, and some karma based off a calculation involving the largest dice pool rolled against a character.

Characters only receive rewards when they complete a run. In order to complete the run they must have completed some or all of the objectives of the run. Characters can only receive rewards if they are alive to collect them. The first two triggers are little more than an attendance award for the player, and players have no ability to affect the third trigger.

Since none of Shadowrun's rewards are based on choices that the players make, Shadowrun offers no incentives to the players to do anything. The reward structure is based almost entirely off of decisions that the gamemaster makes before anyone sits at the table to play. The way that players advance their characters is by showing up to play Shadowrun.

RETURN 'ETHICS IN SHADOWRUNNING';

There are a set of monetary and karmic rewards based on whether the run is about killing and oppressing people, or about helping out the little guy. This is Shadowrun's threadbare attempt at expressing ethics or alignment mechanically. They fail at this dramatically due to the lack of player agency in the reward system.

The gamemaster comes to play with everything planned out including whether or not the mission makes the runners bad people or good people. The only agency the characters have is to not take runs that oppose their own ethics. At the table this leaves players without choice due to the inherent social pressure. Nobody gets to play Shadowrun if you decline the run that the gamemaster prepared.

RETURN RESULTS;

Blades in the Dark and Dungeon World's reward structures are connected directly to choices that the players make. These games reward players for engaging in the type of fiction that those games want to create. Blades in the Dark and Dungeon World do not explicitly require that you to engage the mechanisms in order to earn rewards, but the mechanisms are such that they become a natural part of working towards those rewards.

Shadowrun's core mechanics and intricate web of systems are not incentivized. The complete disconnect between player actions and incentives leave Shadowrun feeling over-designed. Why do I have a entirely separate, well designed, vehicular combat system? Without a clear reason to engage mechanics, those mechanics end up detracting from the game. A system with no clear use is dead weight.

And that was the whole point of this post. Shadowrun tells us what Shadowrun is about by way of its reward system. Shadowrun is about showing up to play Shadowrun. There is a monetary reward for engaging notable elements of Shadowrun lore, but it is another thing that the gamemaster decides will or won't happen before a session.

NEXT INPUT

The next post will explore the gamemasting chapter of Shadowrun's core rulebook. We will see how the game intends for gamemasters to prepare for and run a game. I will continue to compare Shadowrun to other game systems as we explore the gamemaster-facing systems.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.

Shadowrun has a long history in the information security community. I cannot count the number of people with fond memories of Shadowrun and who love to share stories of their previous adventures. The game has a special place for me. I have spent a lot of time learning about tabletop roleplaying games and how they are designed, and I would like to share some of my insights.

In this series of posts I am going to analyze the systems of Shadowrun Fifth Edition. I have a lot of opinions about Shadowrun's design, and in particular the design of Fifth Edition Shadowrun. I am planning on talking about the following elements:

• Core system
• Reward System
• Gamemastering
• Other Systems

This post will be focused on Shadowrun's core system mechanics and then briefly touch on how I do system analysis. Core mechanics are the mechanisms that resolve conflict, resolve action, and determine the general flow of the game. Following posts will delve deeper into the system interactions in Shadowrun.

CORE RESOLUTION MECHANICS

Every roleplaying game has a core resolution mechanic. A core resolution mechanic is the means by which actions in the narrative are resolved.

In Dungeons & Dragons the core resolution mechanic is 1d20+[Modifier] vs Target Number. The Modifier is determined by a combination of skill bonuses and character statistics, and the Target Number might be someone else's 1d20+[Modifier] or the Difficulty Class assigned to the task. In Dungeons & Dragons, non-player characters (NPCs) use the same resolution mechanic as player characters (PCs). If you meet or exceed the Target Number then you succeed at your action, and if your result is lower then you fail. Dungeons & Dragons is a binary success-fail system.

In Apocalypse World the core resolution mechanic follows 2d6+[Modifier]. On a 10+ you succeed. On a 7-9, you succeed partially or must make a hard choice. On a 6 or lower, you miss.. If you ask a probability expert you will learn that most results of two six-sided die will fall between 7-9. In Apocalypse World most of your actions in the narrative will partially succeed and complications will arise.

Shadowrun's core resolution mechanic is a dice pool system. In Shadowrun Fifth Edition your dice pool for an action is based on a relevant skill and attribute, then you roll that many six-sided dice and for every five and six that show up on those dice you count a hit. This is called a test. The three types of tests in Shadowrun are Success tests, Opposed tests, and Extended tests. Some key values are skill, attribute, limit, threshold, and for extended tests the test interval.

A Success test is represented by Skill + Attribute [Limit](Threshold). An opposed test is similar in that the success threshold for the test is replaced by a test made by your target. An extended test is represented as Skill + Attribute [Limit](Threshold, Interval).

Skill represents your character's training in a specific task. A character's attribute represents your character's strength, agility, reaction, willpower, and so on. Most tests have a limit which represents a limitation that prevents a player from rolling too many hits on a test. You can only have hits on a test up to your limit. The threshold is the number of hits that are required to succeed on a test. Extended tests have an interval, which represents how long each test takes and you carry over your hits between tests until you meet the threshold.

Everything in Shadowrun is handled by this system from shooting to not getting shot, and anything in between. The Shadowrun core rulebook spends fifty pages explaining the lore of the world before presenting this information. This vital information is then overshadowed by character creation which spans another fifty pages.

THE RULE OF FOUR

For every group of four dice you roll, on average, at least one of those dice should result in a five or six. Unless the gamemaster says otherwise, you can 'buy' one hit for every four dice in your test's dice pool instead of rolling the test. Shadowrun never connects buying hits on tests with the narrative importance of skill rating 4. Each skill rating from 1 to 12 are described in narrative context.

A skill rating of 4 is considered 'Professional.' Therefore, a professional locksmith, with a skill rating of 4, who is suited for the task, an agility rating of 4, will always succeed at any Average (Threshold 2) locksmithing tasks. Tools add to the user's dice pool and limit for a specific test, and so Professional tools would be rating 4. Given professional tools, a professional locksmith can be reasonably assured that with a bit of effort they can complete Hard (Threshold 4) tasks often enough to make an honest enough living.

I have called this out because this is indicative of both how thoughtfully designed the mechanics of the game are and how poorly they are presented. This interaction contextualizes Shadowrun's balance choices, and enables players to make more meaningful choices during character creation.

Edit (Clarity): Buying hits is worse on average than rolling dice on a test. The purpose of this mechanic is to bypass low-stakes tests that you are likely to succeed at.

INITIATE COMBAT PROTOCOLS

The core resolution mechanic in Shadowrun is flexible. It effectively resolves any situation that one can encounter in play. This creates a smooth game flow when the players or the gamemaster know what pools are required and how the results affect the narrative. The core resolution mechanics take up six pages in the core rulebook.

Every conflict is resolved in exactly the same way, regardless of the situation. The combat rules include an action economy comprised of complex actions, simple actions, and free actions. You can always take a free action on your turn, and you may take one complex action or two simple actions on your turn.

All combat involves similar steps.

1. Attacker rolls against Defender. Example, in meat-space: Pistols+Agility [Accuracy] vs Reaction+Intuition
2. If the attacker wins, then add their net hits to their attack's Damage Value. Example, attacker 2 net hits: Ares Predator DV 8P + 2 (net hits) = DV 10
3. Defender resists the damage. Example, meat-space: Defender rolls Body+Armor. Defender rolled 4 hits. DV10-4 = DV6. Defender takes 6 damage.

This same process occurs whether you are hitting someone with a taser, throwing data spikes in cyberspace, or hurling conjured acid. The only difference is what skills and attributes are used at each step.

At the start of a combat turn, players roll initiative and when a character takes a turn they reduce its initiative by ten, and any character who has an initiative above 0 may make another turn on the next initiative pass. This continues until everyone has an initiative of 0 or lower and then everyone rolls fresh initiatives. This initiative system is difficult to describe but plays smoothly. It ends up as a simple core system that plays the same in meat space, the matrix, or in astral space.

The core combat mechanics are simple enough that I summarized everything you need to know in a few paragraphs. The combat section of the Shadowrun core rulebook spends fifty pages explaining every possible modifier one may encounter. Experienced players who have memorized all the exceptions and situational modifiers like cover, wind, visibility, and distance can flow from one action to the next as fast they can roll the dice. The problem is that combat, like many sections of the game, has a simple core system and pages upon pages of modifiers. For any possible situation in combat that might make it easier or harder to shoot there is a modifier to consider.

ANALYZING...

Shadowrun comes from an old school of game design: simulationism. I am certain that it is alive and well and that it has a group of loyal followers. Every gun has unique stats or features, and every runner has a personal favorite. There are dozens of tools and at least one of them is exactly what you need to open that door. The more careful runners will spend hours planning every detail of a run to command as many positive modifiers as possible and to minimize negative ones.

Fortunately we have learned a lot about design since the days of Dungeons & Dragons Third Edition (Revised) and Shadowrun First Edition. There is a new trend in game design that demands more than a series of generic resolution mechanics. Games can deliver specific experiences and guide players to specific types of narratives through clever use of mechanics.

INPUT QUERY

Smarter people than I have created a set of questions that guide and focus the design of games.

1. What is your game about?
2. How is your game about that?
3. How does your game reward players for engaging the things your game is about?

These questions work very well for designing games. I have found a lot of use in them for analyzing systems outside of games. If we look at those questions we are examining a couple things in particular.

1. What is this system intended to do?
2. How does this system work to deliver that?
3. How does the system incentivize users to engage in the system as intended?

With this set of questions we can examine practically anything and come to an understanding of how the mechanisms involved function and what the results of different interactions will be. It is not important to answer these questions in order when you examine an existing sytem. All that matters is that you work your way through each mechanic you can interact with until they all have context and you have sufficient answers to each question.

NEXT INPUT

There is a good core system in Shadowrun Fifth Edition. There are problems with the tacked-on systems that I am going to explore in future posts. I will examine different mechanisms and interactions through the lens of these three elements: intentions, mechanisms, and incentives. My goal is to show how to analyze systems using Shadowrun as an example.

The first system that I am going to explore in my next post will be the reward system in Shadowrun.

Support the Author

Devon Taylor (They/Them) is a Canadian network architect, security consultant, and blogger. They have experience developing secure network and active directory implementations in low-budget and low-personnel environments. Their blog offers a unique and detailed perspective on security and game design, and they tweet about technology, security, games, and social issues. You can support their work via Patreon (USD), or directly via ko-fi.