Decoding AWS access errors

23-Mar, 2017

AWS is a really amazing set of tools but when using it properly you want to ensure that you are running it using least privileged access. When testing this you often get a series of messages with the dreaded text of:

You are not authorized to perform this operation. Encoded authorization failure message…

Which is followed by an encoded string which is next to useless in working out what has gone wrong. As always, the fix can be found in the AWS CLI, specifically the decode-authorization-message. To decode the message run the following command:

aws sts decode-authorization-message --encoded-message <big_nasty_string>

and you will get a result as follows:

"DecodedMessage": "{\"allowed\":false,\"explicitDeny\":false,\"matchedStatements\":{\"items\":[]},\"failures\":{\"items\":[]},\"context\":{\"principal\":{\"id\":\"SECRET\",\"name\":\"MyUser\",\"arn\":\"arn:aws:iam::000:user/MyUser\"},\"action\":\"iam:PassRole\",\"resource\":\"arn:aws:iam::000:role/MyRole-B4WV0JCNQVGU\",\"conditions\":{\"items\":[]}}}"

Now once again, not incredibly clear but by looking at it more closely you can see that it was denied ("allowed":false) when trying to execute the action "action":"iam:PassRole". In order to fix this problem I need to grant the User the iam:PassRole permission.


Troubleshoot SSM Domain Join

01-Dec, 2016

Now for those that don’t know, AWS have a really handy feature called SSM (Simple Systems Manager) which allows you to perform simple actions against either Windows or Linux hosts.

I am attempting to domain join a Windows 2016 instance to an AWS AD Enterprise Directory Service and am not having any joy. This document details my experiences and the (I hope) fix.

The SSM Document that I am using is pretty simple and is as follows:

    "description": "Join instances to an AWS Directory Service domain.",
    "runtimeConfig": {
          "directoryOU": "ou=Computers,ou=domain.local",
          "directoryId": "d-97673d0000", "directoryName": "domain.local"}
    "schemaVersion": "1.2"

Instance not joining the domain

I am provisioning the instance using CloudFormation and it should be joining the domain on startup. The CloudFormation stack executes OK and the instance can be logged on using the password obtained through the AWS console.

It is connected to SSM:

aws ssm describe-instance-information --instance-information-filter-list key=InstanceIds,valueSet=i-0a529828074260000

With the result:

"InstanceInformationList": [
        "IsLatestVersion": false,
        "ComputerName": "EC2AMAZ-GU425FN.WORKGROUP",
        "PingStatus": "Online",
        "InstanceId": "i-0a529828074260000",
        "IPAddress": "",
        "ResourceType": "EC2Instance",
        "AgentVersion": "1.2.371.0",
        "PlatformVersion": "10.0.14393",
        "PlatformName": "Microsoft Windows Server 2016 Datacenter",
        "PlatformType": "Windows",
        "LastPingDateTime": 1480555932.076

When I query the action association though it has failed..

aws ssm describe-association --name SSMDocumentName --instance-id i-0a529828074260000

With the result:

    "AssociationDescription": {
        "InstanceId": "i-0a529828074260000",
        "Date": 1480552707.186,
        "Name": "SSMDocumentName",
        "Parameters": {},
        "Status": {
          "Date": 1480552953.0,
          "Message": "1 out of 1 plugin processed, 0 success, 1 failed, 0 timedout",
          "Name": "Failed"

Now the SSM Logs are stored at:


Now, you need to scroll down in the log because it may not have been cleared out in the AMI that you used to provision your image, in my case the error was as follows:

2016-12-01 01:12:56 ERROR [instanceID=i-0a529828074260000] [MessageProcessor] error when calling AWS APIs. error details - GetMessages Error: AccessDeniedException: User: arn:aws:sts::491253400000:assumed-role/HostRole-181YC69WRAQ22/i-0a529828074260000 is not authorized to perform: ec2messages:GetMessages on resource: *
status code: 400, request id: 4ae0f258-b763-11e6-86ad-dffe6caddfa2

Clearly this is an IAM problem, my current host has the following access attached to it:

- ssm:DescribeAssociation
- ssm:GetDocument
- ssm:ListAssociations
- ssm:UpdateAssociationStatus
- ssm:UpdateInstanceInformation

Which I will update to the following:

- ssm:DescribeAssociation
- ssm:GetDocument
- ssm:ListAssociations
- ssm:UpdateAssociationStatus
- ssm:UpdateInstanceInformation
- ec2messages:AcknowledgeMessage
- ec2messages:DeleteMessage
- ec2messages:FailMessage
- ec2messages:GetEndpoint
- ec2messages:GetMessages
- ec2messages:SendReply
- ec2:DescribeInstanceStatus
- ds:CreateComputer
- ds:DescribeDirectories

This modification fixed my problem and the host was able to join the domain.


Why doesn't Canada have an NBN?

04-May, 2016

I have recently returned from Canada and in all my experience I have never visited a country that is so similar to Australia. On a whole range of issues we have a similar background - geography (both big and sparsely populated) and culture (former British colony, similar political systems), economic (similar currency strengths, both resource heavy economies) but for some reason our internet access just sucks whereas theirs is actually really good.

Add to this, I didn’t see any evidence in Canada that their government fixed their internet access - if anything their government seems more hopeless at things than ours…

So I thought, why did Canada get decent Internet without a ridiculously expensive NBN?

Let me put it into perspective… lets compare and contrast internet packages:

Country Provider Monthly Cost Max Upload Max Download Data
Canada Rogers CAD 149.99 50Mbs 1000Mbs Unlimited
Australia Telstra AUD 115.00 1Mbs 10Mbs 1000Gbs

Now I grant that Telstra isn’t the best internet provider in Australia but Rogers is a similar company to Telstra in that they appear to be the biggest provider (at least in Toronto), but the fact of the matter is that the costs are similar but the provided service is significantly worse in Australia… now we can complain but the big question is… why? Why do we need the NBN when Canada didn’t have to have one and ended up with a better service?

I was in Canada for 9 months and believe that I know the answer to this question. The answer is sport… in Australia we can watch most things that we need on free to air TV. The only people who get cable in Australia are those sport nuts who need to watch every AFL/EPL/Cricket/League/Union game. In Australia, the big sporting events are on free to air TV - cricket, league, union, afl.. all of it. I can be a sports nut and watch most of what I want. What else does free to air TV have? - no infrastructure - as long as you can receive the signal you can get it. That means that the rollout of highspeed network infrastructure that is required to support cable in Australia just hasn’t happened.

In Canada, they love ice hockey as much as we love AFL/EPL/Cricket/League/Union (possibly more so), and you cannot watch a game of the Toronto Maple Leafs without watching it on cable. Every house that I visited had cable in Canada but thinking it through I only know one person in Australia that has cable.

What does this mean, this means that the cable companies in Canada have the infrastructure in place to support highspeed internet whereas in Australia we don’t because our entertainment comes from “dumb” free to air towers. In short, I now blame our free access to sport as the reason that we have rubbish internet.

That said, why is the NBN so rubbish - and lets call a spade a spade and say that Labor stuffed it and so did the Liberals (and I am a Liberal supporter). The next government (whoever wins) isn’t going to make it better because fundamentally it is stuffed and to say otherwise is in my opinion to defy reality. I thought it was a dud when it started and it is still a dud and I suspect it will never complete… Why do I think that… for this reason.

If you were going to rollout the service, how would you do it? I would suggest the following:

You would roll it out initially in the place where it would have the highest penetration. You are looking for high density areas in large cities with rows of apartment blocks and people living in them that will pay for high speed internet. This will keep your per customer rollout costs low. You will then use these areas to fund rollouts to less high density areas (aka suburbia) and finally you will move out to rural areas (possibly with government support to enable equality of service where it is not economically viable). And you can see this in the way mobile systems are rolled out - the first place they are putting 4G towers is in the CBD and then pushing them out from there.

How did the NBN do it? The initial rollout was to Tasmania, then Armidale - could you choose worse places if you tried? Brilliant - and we wonder it isn’t rolling out as quickly as we think.

Lastly, it is interesting to compare something that doesn’t have an NBN, lets look at phone prices between Telstra and Rogers

Country Provider Cost per Month Talk Text Data
Canada Rogers CAD 105.00 Unlimited Unlimited 5Gb
Australia Telstra AUD 60.00 Unlimited Unlimited 10Gb

In closing, the next time you curse your rubbish internet connection - blame the Australian Cricket Team - it is their fault, then blame the government - but don’t be surprised they stuffed it up, they are the government after all!


More Posts..