06 Mar

Fruitful Code Theme Warnings

I use the fruitful theme by fruitfulcode on my WordPress blog. After a recent change of environment, which meant that non-fatal PHP warnings would be displayed, I noticed warnings being displayed at the top of the blog

I navigated to the code highlighted in the warning (line 756 of inc/metaboxes/init.php) and found this

elseif ( is_array( $meta_box['pages'] ) && count( $meta_box['pages'] === 1 ) )

It is the second condition in this if statement that causes a problem. As we are invoking count on a boolean (the results of $meta_box['pages'] === 1) rather than on an array or countable object PHP will emit this warning. I assume that the developer actually wanted to compare the number of pages to 1 and so I moved the close parenthesis so that we are invoking count on $meta_box['pages'], which we know is an array from the first condition in the if statement.

elseif ( is_array( $meta_box['pages'] ) && count( $meta_box['pages'] ) === 1 )

After this code update the warnings disappeared. As I do not find it best practice to modify third party code I reverted my change and instead submitted it as a pull request to the fruitful theme developers.

06 Mar

WordPress Docker containers and display of PHP warnings

I use a Docker container to host a WordPress blog and I recently upgraded the Docker image to one based on an image which used PHP 7.2. I had previously been using one based on Apache with PHP 5.6.

I immediately noticed warnings being displayed at the top of the blog

The warnings where from PHP letting a developer know that their code had a non fatal error. The cause of the error is a subject for another blog post. I just wanted to turn off the display of these warning messages.

To do this you can set the display_errors configuration variable to Off in php.ini

Create a file called hide_errors.ini with the following content

display_errors = Off

I then proceeded to recreate my WordPress docker container with the same volume as the current container used, as well as mounting hide_errors.ini to /usr/local/etc/php/conf.d/hide_errors.ini inside the container.

After this the warnings were no longer displayed.

10 May

Decoding SAML responses in form post data using PowerShell

If you are ever inspecting a network trace when diagnosing SAML issues and the computer you are using is restricted enough that you cannot install useful SAML specific tools, you can always use PowerShell to help you demystify the network trace.

The techniques in this post can be used for decoding any application/x-www-form-urlencoded data however I will describe fully the procedure for decoding the SAMLResponse HTTP POST data and displaying the resultant XML in a nicely indented format.

In your network trace you will see HTTP POST data similar to the following:


SAMLResponse=PHNhbWxwOlJlc3BvbnNlIHhtbG5zOnNhbWxwPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6cHJvdG9jb2
wiIHhtbG5zOnNhbWw9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphc3NlcnRpb24iIElEPSJfOGU4ZGM1ZjY5YTk4Y2M0Y
zFmZjM0MjdlNWNlMzQ2MDZmZDY3MmY5MWU2IiBWZXJzaW9uPSIyLjAiIElzc3VlSW5zdGFudD0iMjAxNC0wNy0xN1QwMTowMTo0
OFoiIERlc3RpbmF0aW9uPSJodHRwOi8vc3AuZXhhbXBsZS5jb20vZGVtbzEvaW5kZXgucGhwP2FjcyIgSW5SZXNwb25zZVRvPSJ
PTkVMT0dJTl80ZmVlM2IwNDYzOTVjNGU3NTEwMTFlOTdmODkwMGI1MjczZDU2Njg1Ij48c2FtbDpJc3N1ZXI%2baHR0cDovL2lk
cC5leGFtcGxlLmNvbS9tZXRhZGF0YS5waHA8L3NhbWw6SXNzdWVyPjxzYW1scDpTdGF0dXM%2bPHNhbWxwOlN0YXR1c0NvZGUgV
mFsdWU9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDpzdGF0dXM6U3VjY2VzcyIgLz48L3NhbWxwOlN0YXR1cz48c2FtbDpB
c3NlcnRpb24geG1sbnM6eHNpPSJodHRwOi8vd3d3LnczLm9yZy8yMDAxL1hNTFNjaGVtYS1pbnN0YW5jZSIgeG1sbnM6eHM9Imh
0dHA6Ly93d3cudzMub3JnLzIwMDEvWE1MU2NoZW1hIiBJRD0iX2Q3MWEzYThlOWZjYzQ1YzllOWQyNDhlZjcwNDkzOTNmYzhmMD
RlNWY3NSIgVmVyc2lvbj0iMi4wIiBJc3N1ZUluc3RhbnQ9IjIwMTQtMDctMTdUMDE6MDE6NDhaIj48c2FtbDpJc3N1ZXI%2baHR
0cDovL2lkcC5leGFtcGxlLmNvbS9tZXRhZGF0YS5waHA8L3NhbWw6SXNzdWVyPjxzYW1sOlN1YmplY3Q%2bPHNhbWw6TmFtZUlE
IFNQTmFtZVF1YWxpZmllcj0iaHR0cDovL3NwLmV4YW1wbGUuY29tL2RlbW8xL21ldGFkYXRhLnBocCIgRm9ybWF0PSJ1cm46b2F
zaXM6bmFtZXM6dGM6U0FNTDoyLjA6bmFtZWlkLWZvcm1hdDp0cmFuc2llbnQiPl9jZTNkMjk0OGI0Y2YyMDE0NmRlZTBhMGIzZG
Q2ZjY5YjZjZjg2ZjYyZDc8L3NhbWw6TmFtZUlEPjxzYW1sOlN1YmplY3RDb25maXJtYXRpb24gTWV0aG9kPSJ1cm46b2FzaXM6b
mFtZXM6dGM6U0FNTDoyLjA6Y206YmVhcmVyIj48c2FtbDpTdWJqZWN0Q29uZmlybWF0aW9uRGF0YSBOb3RPbk9yQWZ0ZXI9IjIw
MjQtMDEtMThUMDY6MjE6NDhaIiBSZWNpcGllbnQ9Imh0dHA6Ly9zcC5leGFtcGxlLmNvbS9kZW1vMS9pbmRleC5waHA%2fYWNzI
iBJblJlc3BvbnNlVG89Ik9ORUxPR0lOXzRmZWUzYjA0NjM5NWM0ZTc1MTAxMWU5N2Y4OTAwYjUyNzNkNTY2ODUiIC8%2bPC9zYW
1sOlN1YmplY3RDb25maXJtYXRpb24%2bPC9zYW1sOlN1YmplY3Q%2bPHNhbWw6Q29uZGl0aW9ucyBOb3RCZWZvcmU9IjIwMTQtM
DctMTdUMDE6MDE6MThaIiBOb3RPbk9yQWZ0ZXI9IjIwMjQtMDEtMThUMDY6MjE6NDhaIj48c2FtbDpBdWRpZW5jZVJlc3RyaWN0
aW9uPjxzYW1sOkF1ZGllbmNlPmh0dHA6Ly9zcC5leGFtcGxlLmNvbS9kZW1vMS9tZXRhZGF0YS5waHA8L3NhbWw6QXVkaWVuY2U
%2bPC9zYW1sOkF1ZGllbmNlUmVzdHJpY3Rpb24%2bPC9zYW1sOkNvbmRpdGlvbnM%2bPHNhbWw6QXV0aG5TdGF0ZW1lbnQgQXV0
aG5JbnN0YW50PSIyMDE0LTA3LTE3VDAxOjAxOjQ4WiIgU2Vzc2lvbk5vdE9uT3JBZnRlcj0iMjAyNC0wNy0xN1QwOTowMTo0OFo
iIFNlc3Npb25JbmRleD0iX2JlOTk2N2FiZDkwNGRkY2FlM2MwZWI0MTg5YWRiZTNmNzFlMzI3Y2Y5MyI%2bPHNhbWw6QXV0aG5D
b250ZXh0PjxzYW1sOkF1dGhuQ29udGV4dENsYXNzUmVmPnVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphYzpjbGFzc2VzOlB
hc3N3b3JkPC9zYW1sOkF1dGhuQ29udGV4dENsYXNzUmVmPjwvc2FtbDpBdXRobkNvbnRleHQ%2bPC9zYW1sOkF1dGhuU3RhdGVt
ZW50PjxzYW1sOkF0dHJpYnV0ZVN0YXRlbWVudD48c2FtbDpBdHRyaWJ1dGUgTmFtZT0idWlkIiBOYW1lRm9ybWF0PSJ1cm46b2F
zaXM6bmFtZXM6dGM6U0FNTDoyLjA6YXR0cm5hbWUtZm9ybWF0OmJhc2ljIj48c2FtbDpBdHRyaWJ1dGVWYWx1ZSB4c2k6dHlwZT
0ieHM6c3RyaW5nIj50ZXN0PC9zYW1sOkF0dHJpYnV0ZVZhbHVlPjwvc2FtbDpBdHRyaWJ1dGU%2bPHNhbWw6QXR0cmlidXRlIE5
hbWU9Im1haWwiIE5hbWVGb3JtYXQ9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphdHRybmFtZS1mb3JtYXQ6YmFzaWMiPj
xzYW1sOkF0dHJpYnV0ZVZhbHVlIHhzaTp0eXBlPSJ4czpzdHJpbmciPnRlc3RAZXhhbXBsZS5jb208L3NhbWw6QXR0cmlidXRlV
mFsdWU%2bPC9zYW1sOkF0dHJpYnV0ZT48c2FtbDpBdHRyaWJ1dGUgTmFtZT0iZWR1UGVyc29uQWZmaWxpYXRpb24iIE5hbWVGb3
JtYXQ9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphdHRybmFtZS1mb3JtYXQ6YmFzaWMiPjxzYW1sOkF0dHJpYnV0ZVZhb
HVlIHhzaTp0eXBlPSJ4czpzdHJpbmciPnVzZXJzPC9zYW1sOkF0dHJpYnV0ZVZhbHVlPjxzYW1sOkF0dHJpYnV0ZVZhbHVlIHhz
aTp0eXBlPSJ4czpzdHJpbmciPmV4YW1wbGVyb2xlMTwvc2FtbDpBdHRyaWJ1dGVWYWx1ZT48L3NhbWw6QXR0cmlidXRlPjwvc2F
tbDpBdHRyaWJ1dGVTdGF0ZW1lbnQ%2bPC9zYW1sOkFzc2VydGlvbj48L3NhbWxwOlJlc3BvbnNlPg%3d%3d

To decode this, we need to initially interpret this data as application/x-www-form-urlencoded data and for this we will use the HttpUtility.ParseQueryString available in System.Web.dll.

First off we need to load the System.Web assembly:

[System.Reflection.Assembly]::LoadWithPartialName("System.Web")

If it is successfull we will see output similar to the following:


GAC Version Location
--- ------- --------
True v4.0.30319 C:\WINDOWS\Microsoft.Net\assembly\GAC_64\System.Web\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Web.dll

Now let’s assign our form data to a variable and parse it:

$urlencodedformdata = @"
SAMLResponse=PHNhbWxwOlJlc3BvbnNlIHhtbG5zOnNhbWxwPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6cHJvdG9jb2
wiIHhtbG5zOnNhbWw9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphc3NlcnRpb24iIElEPSJfOGU4ZGM1ZjY5YTk4Y2M0Y
zFmZjM0MjdlNWNlMzQ2MDZmZDY3MmY5MWU2IiBWZXJzaW9uPSIyLjAiIElzc3VlSW5zdGFudD0iMjAxNC0wNy0xN1QwMTowMTo0
...................................................................................................
...................................................................................................
...................................................................................................
JtYXQ9InVybjpvYXNpczpuYW1lczp0YzpTQU1MOjIuMDphdHRybmFtZS1mb3JtYXQ6YmFzaWMiPjxzYW1sOkF0dHJpYnV0ZVZhb
HVlIHhzaTp0eXBlPSJ4czpzdHJpbmciPnVzZXJzPC9zYW1sOkF0dHJpYnV0ZVZhbHVlPjxzYW1sOkF0dHJpYnV0ZVZhbHVlIHhz
aTp0eXBlPSJ4czpzdHJpbmciPmV4YW1wbGVyb2xlMTwvc2FtbDpBdHRyaWJ1dGVWYWx1ZT48L3NhbWw6QXR0cmlidXRlPjwvc2F
tbDpBdHRyaWJ1dGVTdGF0ZW1lbnQ%2bPC9zYW1sOkFzc2VydGlvbj48L3NhbWxwOlJlc3BvbnNlPg%3d%3d
"@
$parsedformdata = [System.Web.HttpUtility]::ParseQueryString($urlencodedformdata)

If everything was successful we should get pure base64 encoded data as the value of the SAMLResponse entry in $parsedformdata

$parsedformdata["SAMLResponse"]
PHNhbWxwOlJlc3BvbnNlIHhtbG5zOnNhbWxwPSJ1cm46b2FzaXM6bmFtZXM6dGM6U0FNTDoyLjA6cHJvdG9jb2
......................................................................................
......................................................................................
......................................................................................
tbDpBdHRyaWJ1dGVTdGF0ZW1lbnQ+PC9zYW1sOkFzc2VydGlvbj48L3NhbWxwOlJlc3BvbnNlPg==

At this point we can get the XML by base64 decoding the data and interpreting it as a UTF8 string:

$base64encodedxml = $parsedformdata["SAMLResponse"]
$xml = [System.Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($base64encodedxml))

At this point $xml should look like the following:

<samlp:Response xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" ID="_8e8dc5f69a98cc4c1ff3427e5ce34606fd672f91e6" Version="2.0" IssueInstant="2014-07-17T01:01:48Z" Destination="http://sp.example.com/demo1/index.php?acs" InResponseTo="ONELOGIN_4fee3b046395c4e751011e97f8900b5273d56685"><saml:Issuer>http://idp.example.com/metadata.php</saml:Issuer><samlp:Status><samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success" /></samlp:Status><saml:Assertion xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="_d71a3a8e9fcc45c9e9d248ef7049393fc8f04e5f75" Version="2.0" IssueInstant="2014-07-17T01:01:48Z"><saml:Issuer>http://idp.example.com/metadata.php</saml:Issuer><saml:Subject><saml:NameID SPNameQualifier="http://sp.example.com/demo1/metadata.php" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient">_ce3d2948b4cf20146dee0a0b3dd6f69b6cf86f62d7</saml:NameID><saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"><saml:SubjectConfirmationData NotOnOrAfter="2024-01-18T06:21:48Z" Recipient="http://sp.example.com/demo1/index.php?acs" InResponseTo="ONELOGIN_4fee3b046395c4e751011e97f8900b5273d56685" /></saml:SubjectConfirmation></saml:Subject><saml:Conditions NotBefore="2014-07-17T01:01:18Z" NotOnOrAfter="2024-01-18T06:21:48Z"><saml:AudienceRestriction><saml:Audience>http://sp.example.com/demo1/metadata.php</saml:Audience></saml:AudienceRestriction></saml:Conditions><saml:AuthnStatement AuthnInstant="2014-07-17T01:01:48Z" SessionNotOnOrAfter="2024-07-17T09:01:48Z" SessionIndex="_be9967abd904ddcae3c0eb4189adbe3f71e327cf93"><saml:AuthnContext><saml:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:Password</saml:AuthnContextClassRef></saml:AuthnContext></saml:AuthnStatement><saml:AttributeStatement><saml:Attribute Name="uid" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"><saml:AttributeValue xsi:type="xs:string">test</saml:AttributeValue></saml:Attribute><saml:Attribute Name="mail" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"><saml:AttributeValue xsi:type="xs:string">[email protected]</saml:AttributeValue></saml:Attribute><saml:Attribute Name="eduPersonAffiliation" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"><saml:AttributeValue xsi:type="xs:string">users</saml:AttributeValue><saml:AttributeValue xsi:type="xs:string">examplerole1</saml:AttributeValue></saml:Attribute></saml:AttributeStatement></saml:Assertion></samlp:Response>

Now we can use the XmlTextWriter class to indent and format the XML for us.

Jeffrey Snover has already given a sample snippet on how this can be done over at the PowerShell Team Blog. I have repeated their code here:

function Format-XML ($xml, $indent=2) 
{ 
    $StringWriter = New-Object System.IO.StringWriter 
    $XmlWriter = New-Object System.XMl.XmlTextWriter $StringWriter 
    $xmlWriter.Formatting = “indented” 
    $xmlWriter.Indentation = $Indent 
    $xml.WriteContentTo($XmlWriter) 
    $XmlWriter.Flush() 
    $StringWriter.Flush() 
    Write-Output $StringWriter.ToString() 
}

We finally use this function to format our XML data nicely on screen:

Format-Xml -xml $xml -indent 4

Our final output should look like this:

<samlp:Response xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" ID="_8e8dc5f69a98cc4c1ff3427e5ce34606fd672f91e6" Version="2.0" IssueInstant="2014-07-17T01:01:48Z" Destination="http://sp.example.com/demo1/index.php?acs" InResponseTo="ONELOGIN_4fee3b046395c4e751011e97f8900b5273d56685">
    <saml:Issuer>http://idp.example.com/metadata.php</saml:Issuer>
    <samlp:Status>
        <samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success" />
    </samlp:Status>
    <saml:Assertion xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="_d71a3a8e9fcc45c9e9d248ef7049393fc8f04e5f75" Version="2.0" IssueInstant="2014-07-17T01:01:48Z">
        <saml:Issuer>http://idp.example.com/metadata.php</saml:Issuer>
        <saml:Subject>
            <saml:NameID SPNameQualifier="http://sp.example.com/demo1/metadata.php" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient">_ce3d2948b4cf20146dee0a0b3dd6f69b6cf86f62d7</saml:NameID>
            <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                <saml:SubjectConfirmationData NotOnOrAfter="2024-01-18T06:21:48Z" Recipient="http://sp.example.com/demo1/index.php?acs" InResponseTo="ONELOGIN_4fee3b046395c4e751011e97f8900b5273d56685" />
            </saml:SubjectConfirmation>
        </saml:Subject>
        <saml:Conditions NotBefore="2014-07-17T01:01:18Z" NotOnOrAfter="2024-01-18T06:21:48Z">
            <saml:AudienceRestriction>
                <saml:Audience>http://sp.example.com/demo1/metadata.php</saml:Audience>
            </saml:AudienceRestriction>
        </saml:Conditions>
        <saml:AuthnStatement AuthnInstant="2014-07-17T01:01:48Z" SessionNotOnOrAfter="2024-07-17T09:01:48Z" SessionIndex="_be9967abd904ddcae3c0eb4189adbe3f71e327cf93">
            <saml:AuthnContext>
                <saml:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:Password</saml:AuthnContextClassRef>
            </saml:AuthnContext>
        </saml:AuthnStatement>
        <saml:AttributeStatement>
            <saml:Attribute Name="uid" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <saml:AttributeValue xsi:type="xs:string">test</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="mail" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <saml:AttributeValue xsi:type="xs:string">[email protected]</saml:AttributeValue>
            </saml:Attribute>
            <saml:Attribute Name="eduPersonAffiliation" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <saml:AttributeValue xsi:type="xs:string">users</saml:AttributeValue>
                <saml:AttributeValue xsi:type="xs:string">examplerole1</saml:AttributeValue>
            </saml:Attribute>
        </saml:AttributeStatement>
    </saml:Assertion>
</samlp:Response>

This is a lot more readable than what we had to begin with. Hopefully if you ever find yourself in this situation you will remember that PowerShell can come to your rescue.

12 Apr

Boot Arch Linux from PXE whilst directly connected via ethernet cable

I was following the PXE tutorial from ArchWiki (PXE) however I was directly connecting (via ethernet cable) to my target. This added a few complications that I’ve tackled below. I’ve repeated the instructions and made a note where (and WHY!) I have differed from the ArchWiki article.

First off we need to download the latest ISO image from the downloads page.

Mount the image in your current directory:

sudo mkdir -pv /mnt/archiso
sudo mount -o loop,ro archlinux-2017.02.01-dual.iso /mnt/archiso

Go ahead and connect your ethernet cable between you and the target. Remember you will need to use a crossover ethernet cable otherwise the host computer will be unable to communicate with the target.

Note: I didn’t have a cross over cable to hand so I used a Raspberry Pi as the host. The Raspberry Pi’s ethernet port is auto-sensing (auto MDI-X) so a normal cable or cross over cable can be used.

The ArchWiki article recommends you set a static IP address manually like so:

sudo ip link set eth0 up
sudo ip addr add 192.168.0.1/24 dev eth0

However I experienced the target dropping and resetting the ethernet connection when transitioning from the boot menu to the PXE boot option. When the ethernet connection came back up, the interface had lost it’s manually set static IP address.

For this reason I created a systemd-networkd configuration file to set a static IP address on the network interface. (Make sure you disable dhcp and/or set up a static IP address for what ever network manager you are using).

My systemd-networkd configuration file I created was stored in /etc/systemd/network/25-wired.network:

[Match]
Name=eth0

[Network]
Address=192.168.0.1/24

Next as per the ArchWiki article we need to setup a DHCP server and a TFTP server.

We install and configure dnsmasq as per the ArchWiki article’s instructions.

Install dnsmasq:

sudo pacman -S dnsmasq

Configure dnsmasq:

# /etc/dnsmasq.conf
port=0
interface=eth0
bind-interfaces
dhcp-range=192.168.0.50,192.168.0.150,12h
dhcp-boot=/arch/boot/syslinux/lpxelinux.0
dhcp-option-force=209,boot/syslinux/archiso.cfg
dhcp-option-force=210,/arch/
dhcp-option-force=66,192.168.0.1
enable-tftp
tftp-root=/mnt/archiso

Start dnsmasq:

sudo systemctl start dnsmasq.service

Next install darkhttpd as this will be used to serve the root filesystem as in the ArchWiki article:

sudo pacman -S darkttpd

Start darkttpd:

sudo darkhttpd /mnt/archiso

You are now ready to boot the target. Either modify the BIOS options or enter the boot menu so that you can choose a network boot. This is target specific so you will have to find the instructions for your particular target.

If all goes to plan, you will be presented with an Arch Linux branded syslinux menu, asking how you wish to boot. Select the HTTP option. Once the root filesystem has been copied you should end up at a terminal which has been auto logged in as root.

You have now successfully booted into Arch Linux from a PXE image.

15 Mar

A lesson in character encodings and Docker containers

I recently had to execute some commands on a MySQL docker container.

In particular the docker container was based on the image https://github.com/docker-library/mysql/blob/eeb0c33dfcad3db46a0dfb24c352d2a1601c7667/5.7/Dockerfile.

I executed a bash shell inside the docker container using docker exec -it mysql /bin/bash and then proceeded to enter a cli interface to MySQL:

mysql -uroot -p

The root password was a complex password generated by a password manager. It contained a £ symbol.

I had trouble pasting it in when mysql requested the password.

I decided to set a variable like so, PASS=complex-password so that I could enter mysql using:

mysql -uroot -p$PASS

However when I got to the £ symbol in the password, the shell inserted a # at the beginning of the line and then continued on the next line.

For example, if the complex password was super£pass then this would appear in the shell, if I tried to paste the password:

[email protected]:/# # PASS='super
[email protected]:/# pass'

I couldn’t quite figure out what was going on and I ended up posting a question on stackoverflow. Very fortunately some one answered my question and pointed me in the right direction.

I’ve repeated the explanation here as well as the remedy.

The crux of the issue is that the terminal I was connecting to the docker container from was encoding characters as UTF-8 whereas the bash shell inside the docker container had ASCII character encoding.

This meant when I typed a £ symbol in my terminal, the two bytes C2 A3 (The UTF-8 representation of the £ symbol) were sent into the docker container and interpreted as 2 characters.

Inside the docker container Bash was interpreting characters with the high bit set as characters with the Meta-key modifier set.

For more information on the meta-key see https://www.gnu.org/software/bash/manual/bashref.html#Introduction-and-Notation.

C2 is the decimal 194, which is equal to 128 + 66, which means Bash would interpret it as a capital B (ASCII value 66) with the meta-key modifier.

A3 is the decimal 163, which is equal to 128 + 35, which means Bash would interpret it as a # (ASCII value 35) with the meta-key modifier.

This means that Bash saw my key press of £ as the commands M-B, M-#.

Looking at https://www.gnu.org/software/bash/manual/bashref.html#Miscellaneous-Commands you will see that the
M-B moves the cursor back to the previous word and that M-# inserts the comment character at the start of the line and moves to a newline.

This explains the behaviour I was seeing but how do you go about fixing this so that I could type a £ symbol?

This is as simple as making sure the locale is set correctly in the docker container.

There was no LANG variable set, or no other locale information. This meant the default locale was being used which is the POSIX local. There were no proper locales installed. The only choice of locale with UTF-8 was C.UTF-8.

Setting the LC_ALL variable and launching bash (within bash) like so:

LC_ALL=C.UTF-8 bash

Allowed me to test this would work. Inside this bash shell within a bash shell I could now type the £ symbol freely without strange side affects.

I just had to make this change permanent.

I did this by appending the following to the file /root/.bashrc:

export LC_ALL=C.UTF-8
export LANG=C.UTF-8
export LANGUAGE=C.UTF-8

11 Feb

Emulating Raspbian for the Raspberry Pi on Linux using QEMU

I’ve recently been trying to emulate Raspbian for the Raspberry Pi on my desktop computer so that I can try a few things out when I don’t have my Pi to hand.

Some of the guides on the internet are a bit outdated and since I wanted to use the latest Jessie build some where not relevant.

Below you will find instructions for emulating Raspbian on a desktop PC.

Install QEMU

sudo pacman -S qemu

Download the latest Raspbian image from https://www.raspberrypi.org/downloads/. At the time of writing the latest version was Raspbian Jessie with a kernel version 4.4.

Download this QEMU compatible kernel from https://github.com/dhruvvyas90/qemu-rpi-kernel.

The image file needs to be slightly modified as the /etc/fstab file is referencing /dev/mmcblk0, which is the SD Card device. Inside the emulation environment, the disk will have device id /dev/sda.

Attach the image file on the loop device:

sudo losetup -Pf 2017-01-11-raspbian-jessie-lite.img
mkdir partition2
sudo mount /dev/loop0p2 partition2
vi partition2/etc/fstab

Inside vi change references to mmcblk0 to sda.

Now unmount and detach the image from the loop device:

sudo umount partition2
rm -rf partition2
sudo losetup -d /dev/loop0

We can finally run up the system:

qemu-system-arm -kernel kernel-4.4.34.img -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda 2017-01-11-raspbian-jessie-lite.img

Hopefully after the system has booted you will see a similar screen to this:

25 Jan

Creating an image file from a local directory on Linux

Sometimes I wish to create disk image files using a directory on my local computer. It’s an easy task with Linux, which can be accomplished with only a couple of commands.

The main scenario for when I wish to create a disk image is when creating a .img file that can be written straight to a SD card (for example Arch Linux for the Raspberry Pi comes as a tar.gz of the root filesystem rather than an image file) or used with QEMU as the hard disk.

The following steps (that I use for creating an Arch Linux SD card image file) will take you through creating an image file and writing the files you wish to it.

Create an 8GB image file:

dd if=/dev/zero of=newfile.img bs=1M count=8192

Partition the image file:

fdisk ./newfile.img

I usually create 2 partitions, a FAT partition of about 100M for the boot files and the rest of the image a ext4 for the root filesystem.

Mount the image file using the loopback device. (I used the -f switch to find the next available loop device rather than explicitly specifying one)

sudo losetup -Pf ./newfile.img

The image file should now be visible as a block device on /dev/loopX where X is 0 if you don’t have any other loop devices

Running the following should give similiar output as below:

[[email protected] blog]$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0     1G  0 loop 
├─loop0p1   259:6    0   100M  0 loop 
└─loop0p2   259:7    0   933M  0 loop

Format the partitions

sudo mkfs.vfat /dev/loop0p1
sudo mkfs.ext4 /dev/loop0p2

Mount the partitions:

mkdir p1
mkdir p2
sudo mount /dev/loop0p1 p1
sudo mount /dev/loop0p2 p2

You can now copy anything across my moving your files into directories p1 or p2.

For example, as I usually create SD card images from the Arch Linux ARM root filesystem tar.gz. I do the following:

sudo bsdtar -xpf ArchLinuxARM-rpi-latest.tar.gz -C p2
sync
sudo mv p2/boot/* p1

Clean up:
At this point we just need to clean up and unmount the image file and detach the loop back device:

sudo umount p1 p2
sudo losetup -d /dev/loop0

You should now have a nicely prepared image file. You can also mount existing image files using losetup, as we did above.

15 Dec

Create a WiFi hotspot and modify HTTP requests using the Raspberry Pi 3

Recently I was demonstrating the dangers of public WiFi, or indeed any untrusted network. Regardless of whether the WiFi has a password or not, the owner of the wireless access points can read, inspect and modify all the data you send over the connection.

To best demonstrate this I set up a rogue WiFi hotspot using a Raspberry Pi 3. The Raspberry Pi was setup to do 4 things:

  1. Act as a WiFi hotspot broadcasting a wireless network that did not have a password.
  2. Respond to DNS requests from all clients connected to it’s WiFi and had DNS server information obtained by DHCP.
  3. Return the access points IP address when a client requested the IP address of a well known website.
  4. Run a web server that returns a modified version of the well known website.

Below I take you through the installation and configuration steps necessary to achieve this.

I used Arch Linux on the Raspberry Pi 3 and so it is necessary to follow the ARMv7 installation instructions at https://archlinuxarm.org/platforms/armv8/broadcom/raspberry-pi-3 before continuing.

SSH into the Raspberry Pi using the alarm user with password alarm, change both root and alarm user passwords. (See https://wiki.archlinux.org/index.php/Users_and_groups for those unfamiliar with Arch Linux).

For convenience and security it is recommended to disable password authentication and enable public key authentication. (see https://wiki.archlinux.org/index.php/Secure_Shell#Force_public_key_authentication for those unfamiliar with Arch Linux)

Prepare the Raspberry Pi by executing the following commands as root (using su, see here for more information for those unfamiliar with su).

NOTE: Confirm any prompts that you receive

pacman -Syu
groupadd sudo
usermod -aG sudo alarm
pacman -S --needed base-devel
pacman -S git-core nodejs npm dnsmasq hostapd screen wget

At this point logout and log back in and you should be able to prefix commands that need root privileges with sudo

Create a directory called wifi

mkdir wifi
cd wifi/

create a file called hostapd.conf, in the current directory and populate it’s contents with the following:

beacon_int=100
ssid=Public-Hotspot
interface=wlan0
driver=nl80211
channel=1
ignore_broadcast_ssid=0
ap_isolate=0
hw_mode=g
logger_stdout=-1
logger_stdout_level=0

create a file called dnsmasq.conf, in the current directory and populate it’s contents with the following:

# Configuration file for dnsmasq.

port=53
domain-needed
bogus-priv
server=8.8.8.8
server=8.8.8.4
address=/example.com/192.168.30.1
interface=wlan0
dhcp-range=192.168.30.2,192.168.30.253,255.255.255.0,60m
dhcp-option=vendor:MSFT,2,1i
log-queries
log-dhcp
log-facility=/var/log/dnsmasq.log

NOTE: Replace line 8 with whatever website you wish to redirect to the Pi for modification.

Now we need to update the system to use our local DNS server

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved
sudo rm /etc/resolv.conf
sudo touch /etc/resolv.conf

modify /etc/resolv.conf so it’s contents are as follows:

nameserver 127.0.0.1
nameserver 8.8.8.8

modify hostapd.conf and dnsmasq.conf to suit your needs and then you’re almost ready.

create a file called start_ap.sh in the current directory and make it executable. Populate it’s contents with the following:

#!/bin/bash

ip link set up dev wlan0
ip addr add 192.168.30.1/24 broadcast 192.168.30.255 dev wlan0

hostapd ./hostapd.conf &

sysctl net.ipv4.ip_forward=1

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -P FORWARD ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -I INPUT -p udp --dport 67 -i wlan0 -j ACCEPT
iptables -I INPUT -p udp --dport 53 -s 192.168.30.0/24 -j ACCEPT
iptables -I INPUT -p tcp --dport 53 -s 192.168.30.0/24 -j ACCEPT

dnsmasq -C ./dnsmasq.conf

cd example.com/
node server.js &
cd ..

Optionally now download an example modified page for the example.com:

wget https://lanphier.co.uk/wp-content/uploads/2016/12/example.tar.gz
tar -xzvf example.tar.gz
cd example.com/
npm install
cd ..

You can now start the wifi hotspot using the following:

#start the WiFi
sudo ./start_ap.sh

As soon as you disconnect from the SSH then the processes launched to create the AP and node server will be terminated. To prevent this run the start wifi command inside of a screen session:

screen -S wifi
sudo ./start_ap.sh

You can detach from the screen by pressing ctrl+a and then ctrl+d. To reattach to the screen at a later date then SSH onto the Raspberry Pi and type

screen -x wifi

Protect against rogue hotspots

Any websites that use HTTPS should create a secure channel that the hotspot cannot read, however the hotspots will still be able to see that you are connecting to those secure sites though they will not be able to see the data you exchange.

The best possible protection is to use a VPN. This creates a secure tunnel to a third party and so all your traffic is secure against the hotspot owner.

10 Nov

Resize the OS Disk in Resource Manager Azure Deployments using PowerShell

Recently I’ve had to increase the OS Disk size of about 10 Ubuntu virtual machines running on Microsoft Azure. Looking to the future I can also foresee this happening quite a few times so I thought I would write a script that could do this for me. The script essentially shuts the virtual machine down (after prompting you for confirmation), checks the new size is larger than the old size (an Azure requirement), resizes the disk to the new size and then starts the virtual machine back up.

There is a small quirk in the process that you will sometimes notice if you ever use the web UI to resize the OS disk. Azure sometimes doesn’t report the size of the disk correctly and so you don’t always know what the new size should be. To get around this I previously wrote a script (see here) which can get the disk size in an indirect way. Make sure you grab the retrieve size function from the linked post.

An example usage is the following:

Set-AzureRmOSDiskSize -VirtualMachineName "APP01" -ResourceGroupName "Platform-Dev" -SizeInGB 40

Here is the code below:

function Ask-ToContinue {

    param(
        [string]$message = $(throw "specify message"),
        [string]$prompt = $(throw "specify prompt")
    )

    $choices = New-Object Collections.ObjectModel.Collection[Management.Automation.Host.ChoiceDescription]
    $choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&Yes'))
    $choices.Add((New-Object Management.Automation.Host.ChoiceDescription -ArgumentList '&No'))

    $decision = $Host.UI.PromptForChoice($message, $prompt, $choices, 1)
    
    return $decision -eq 0

}

function Set-AzureRmOSDiskSize () {

    param(
        [string]$VirtualMachineName = $(throw "Specify virtual machine name"),
        [string]$ResourceGroupName = $(throw "Specify resource group name"),
        [int]$SizeInGB = (throw "Specify Size in GB")
    )

    $currentSize = Get-AzureRmOSDiskSize -VirtualMachineName $VirtualMachineName -ResourceGroupName $ResourceGroupName

    if ($currentSize -ne $null -and $SizeInGB -le $currentSize) {

        throw "Specified Disk Size is not larger than current size"

    }

    $VM = Get-AzureRmVM -ResourceGroupName $ResourceGroupName -Name $VirtualMachineName -Status

    if ($VM -eq $null) {

        throw "Virtual Machine not found"

    }

    $VMRunning = ($VM.Statuses | ? { $_.Code -eq "PowerState/running" } | Measure-Object | Select -ExpandProperty Count) -eq 1

    if ($VMRunning) {

        Write-Host "The VM is currently running." -ForegroundColor Magenta

        $stopTheVM = Ask-ToContinue -message "The VM must be stopped" -prompt "Would you like to stop the VM now?"

        if ($stopTheVM) {

            Write-Host -ForegroundColor Yellow "Stopping the VM"
            Stop-AzureRmVM -Name $VirtualMachineName -ResourceGroupName $ResourceGroupName -Force

        } else {

            Write-Host -ForegroundColor Cyan "Not stopping the VM."
            return

        }

    }

    $VM = Get-AzureRmVM -ResourceGroupName $ResourceGroupName -Name $VirtualMachineName
    $VM.StorageProfile.OsDisk.DiskSizeGB = $SizeInGB

    $result = Update-AzureRmVM -VM $VM -ResourceGroupName $ResourceGroupName

    if ($result.IsSuccessStatusCode) {
        
        Write-Host -ForegroundColor Green "Updated VM Successfully"

    }

    $startTheVM = Ask-ToContinue -message "The VM is currently stopped" -prompt "Would you like to start the VM now?"

    if ($startTheVM) {

        Write-Host -ForegroundColor Yellow "Starting the VM"
        Start-AzureRmVM -Name $VirtualMachineName -ResourceGroupName $ResourceGroupName

    }

}
17 Oct

Retrieve the size of the OS Disk in Resource Manager Azure Deployments using PowerShell

Recently I noticed that a script I was using wasn’t quite working correctly because it was failing to get the size of a Azure VM’s disk.

Usually I would use something similar to the following:

$VM = Get-AzureRmVM -ResourceGroupName $ResourceGroupName -Name $VirtualMachineName
$size = $VM.StorageProfile.OsDisk.DiskSizeGB

However $VM.StorageProfile.OsDisk.DiskSizeGB was null. I decided that I would write a function instead which would go and look at the underlying vhd blob and find the size that way. The below is my code and usage:

function Get-AzureRmVhdSize {

    param(
        [Uri] $uri = $(throw "Please enter a URI")
    )
	
    $blobEndpoint = $uri.Scheme + "://" + $uri.Host + "/"
    $sa = Get-AzureRmStorageAccount | ? { $_.PrimaryEndpoints.Blob -eq $blobEndpoint }

    if ($sa -eq $null -or $sa.Length -ne 1) {

        Throw "Unable to locate storage account"

    }

    $containerAndBlob = $uri.AbsolutePath.Split("/", [StringSplitOptions]::RemoveEmptyEntries)

    $blob = Get-AzureStorageBlob -Blob $containerAndBlob[1] -Container $containerAndBlob[0] -Context $sa.Context

    $sizeInBytes = $blob.Length
    $sizeInGB = [int]($sizeInBytes/1073741824)

    return $sizeInGB

}

function Get-AzureRmOSDiskSize {

    param(
        [string]$VirtualMachineName = $(throw "Specify virtual machine name"),
        [string]$ResourceGroupName = $(throw "Specify resource group name")
    )

    $VM = Get-AzureRmVM -ResourceGroupName $ResourceGroupName -Name $VirtualMachineName

    if ($VM -eq $null) {

        throw "Virtual Machine Not Found"

    }

    return Get-AzureRmVhdSize -uri $VM.StorageProfile.OsDisk.Vhd.Uri

}

Get-AzureRmOSDiskSize -VirtualMachineName testvm -ResourceGroupName testrg